When it comes to the more favorable app in the market, the true measure of a developer’s success is the app’s retention rate. An app has to have the right mix of touch, feel and design for the best user experience that represents your business value; as well as the functionality and performance true to the needs of the user and his device health.
With touch devices everywhere now, more than 3 million apps for those devices are available for download and the numbers keep on increasing. App users are increasingly demanding too. When reports have shown that Android users are far more likely to uninstall apps than iOS users for a number of reasons, it becomes easier to collate it with the information that currently Android users are more than four times the market share of iOS devices. Both have a duopoly of 99.6 percent in smartphones. At the same time, unlike iOS, Android is pretty fragmented. Which means, your app must be compatible with tons of device versions and Android versions.
Hence, more and more businesses need to churn out apps for their growing users, and developers having to churn out more and more apps faster, with qualitative inspection of each application that shouldn’t lag, crash, have bugs or glitches, drain phone memory or battery and work anywhere in the world from a smartphone. Added to that, 70 percent of apps get deleted by users within a couple of minutes of their installation.
That’s a tall order! And technology is the only way to sway the game in our favor.
If your app has to compete on the qualitative level of a 5 star rated app in the app store, it needs a thorough and quick check before you hit the launch button. In-house testing within companies can only do so much, there aren’t enough device in-house, there is a lack of right tools or the testing cycles take far too long to make it to the market before ten new apps pop up in the app store. Low ratings, mean low revenue.
How pCloudy is changing the game
Machines and intelligent technology are the newest recruits to the workforce in the world. We at pCloudy have always worked with an approach to apply technology to exponentially enhance the ability of individuals, teams, and businesses.
pCloudy’s helps you create the magical combination of your internal workforce and intelligent platform. With pCloudy, your teams get a wide spectrum of the latest Android and iOS devices in the market to maximize testing coverage, cognitive software bots to run quick, thorough checks on your App and the power of the crowd so that you are aware of what your users think about your app.
pCloudy has developed a Single Click App certification feature within its platform enabling testers to deliver on faster test cycles for users whose attention span is shorter & shorter with every bug, crash, drain or slip that an app can suffer from. Reality check is critical for every developer. So is the need for it to be quicker.
Today pCloudy is working with global companies who have often realized not just the growing potential of the market, but also the need for a crucial wing for testing with pCloudy’s platform. This is probably due to a growing desire from companies across sizes to fully understand the buyers experience consistently.
Watch out this space as we unveil the power of pCloudy’s Test Cloud over next couple of weeks.
Mobile is the key to the digital world in today’s era. The rise of Mobile Internet has ensured that we all are going to make most of our digital decisions on a Mobile device (primarily using an App). Mobile Apps are going to be everywhere.
One of the biggest challenges for App Development teams is to ensure that their Apps works on the variety of devices. They need their apps tested on multiple devices before it is released to the market, simply because of the plethora of devices available in the market.
Each mobile from each OEM (Original Equipment Manufacturer) with different specifications is a unique device where the application should be tested.
Practically it is not possible to test the application on each and every mobile device available in the market. But the app must be tested on a large set of the devices depending on the market segment of App under test.
Introduction:
pCloudy is a cloud-based remote mobile app testing platform which allows testing of Android and iOS mobile applications on remote REAL devices. It allows the user to do Manual testing and also facilitate to run test automation on multiple devices in parallel.
It provides a whole lot of statistics like session video, device logs, Screenshots, data usage, memory usage, CPU usage, battery consumption and frame rendering count. The devices are also equipped with Real Operator SIM’s to make MO/MT calls and can receive and send SMS.
Preparing for Manual Test
Before we start the test on pCloudy, we recommend uploading the test application, test data or even test suites to the cloud drive provided by pCloudy. This is the secured location for each user.
Click on the Upload button and select the file from your local system through file explorer. After selecting the file select upload and the file will be uploaded to your cloud drive.
You can also manage the cloud drive by download. These applications can be installed on the devices as explained below.
Performing Manual Test remotely
Based on test requirement, you can search for any specific device by using the filters provided at pCloudy.
(Note: Click on any image for enlarged view)
You can access any device for Instant Access. Also, you can book the device for a longer duration for your planned testing using “Book your Device” link.
After selecting the device just click on the “Connect to a device” button to connect to the device. The below device page will be displayed with all the supported features.
The device is displayed in the middle pane with some features on the right, left and top pane.
The Left pane where the device information is displayed, followed by other tabs like App/Data, Device Settings, Debug and Automation
The top pane with handy features & a timer to show the remaining time & wake up the device and release the device back to the cloud.
The right pane to view the device logs and Performance vitals for application under test.
How to Install the application on the Remote Mobile device?
Install from my App data – enables user to upload app on device.
Push to Device – This feature allows you to transfer your test data to the device.
Upload App – Upload and install an app from your local system.
Once the application is installed you can test the application on REAL Remote Device the way you test the application with the phone in hand. You can use the mouse to give touch gestures to the device. You can click the button, drag the screen or double tap the image.
While doing functional testing on the remote device, you can also monitor the device logs next to the device. Please start the logs from the right page and start monitoring the logs next to the device.
All Mobile App Testing tools at one place: One of the biggest advantages of pCloudy platform is it provides performance statistics like Memory Usage, CPU Usage, Battery usage, Data usage & frame rendering count of the application under test without any extra efforts by the test engineer.
To view this vital statistic on pCloudy platform, please select Performance tab -> Choose an APP -> Select the application under test.
Start navigating through the application to complete the function test flow.
On the right pane, you will start seeing the important vitals of the performance.
On the left pane, you can expand the device settings. All the important frequently used applications are made available to use on the device.
You can perform below actions on the remote device:
Reboot the device remotely
Toggle the Wi-Fi connection
Adjust the volume of the device
Manage the Apps
Open Developers options
Open Play store
Open Device settings
Use Set Phone Location
If your application under test is a location aware app and if it reads the geo-location from the device to provide the necessary results, “Set Phone Location” feature can be used to inject any location on the earth to the device and the device will be teleported to the selected location.
Once you select the “Set Phone Location”, the platform shows the google maps and you can select the location of your choice on this map. The same location is injected to the device.
Other key features during manual tests
You can capture high-resolution screen shots with the skin of the device
Change the orientation of the device to Landscape and portrait
Open the browser from the list of preinstalled browsers
Use the extended keyboard if you want to enter some text on the device. However, for Android ver 5.0 and above you can directly enter the text from your system keyboard.
Check the stack trace (crash logs) if the application is crashed
You can navigate the application with the buttons
Zoom in/out the device shown on the screen
Reset the zoom level to bring the device to default size
Extend the session if the device is available
Wake up the device from sleep mode or push the device to sleep mode
Release the device back to the cloud
Reporting
Under reports folder, you can see the entire device session data for all the devices you used. You can find Videos & Logs (by default platform generates), screen shots and performance data if selected by the user.
Running test automation remotely
pCloudy platform supports test automation on multiple devices in parallel. Below test frameworks are supported for Android platform.
You just need to select the Test platform, Choose the application under test, Test Suite, time to run on single device and test cycle name.
The biggest advantage of this platform is it generates performance vitals for your test automation scripts without you adding any hooks in your scripts. Otherwise, you will be using multiple mobile App testing tools to achieve the same result.
Once the above data is provided, user can select the devices on which the test automation to be run. Once the devices are selected, the user can schedule the test cycle.
The platform will book the devices on your behalf and will start the test automation as soon as the devices are available. Once the execution is over, the user will be notified with an email and user can see the test reports.
APPIUM Test Automation
You can run the Appium test automation scripts directly from Eclipse or Android Studio. The platform provides the plugin for both the IDE’s. Simply copy a small piece of code to your driver code and run the project, select the devices on which this tests to be run.
The “LIVE VIEW” feature will allow you to monitor the test automation runs through video streaming on the platform.
LIVE VIEW
Live view – To see the automation happening on devices
Automation Test Reports
All the test automation reports will be available under Reports folder.
The reports will open in another tab with video, logs and performance data.
To view the performance data for each device, please click on the device icon for each device.
Opkey Integration
pCloudy has seamless integration with OpKey, which is a zero code multi-channel automation tool. A unique combination of Mobile App Testing Tools for manual and Automation testing.
OpKey allows single click recording and generation of automated scripts for Mobile Apps.
You can run the automated scripts on multiple devices on pCloudy directly from OpKey interface.
The quality of App is extremely important for its success in the market. Right devices and right mobile App testing tools are critical ask from every Mobile Dev and Test team. pCloudy with its comprehensive set of devices and tools could be a perfect companion to every Developer and Tester.
TestNG is a testing framework inspired from JUnit and NUnit but introducing new functionalities that make it more powerful and easier to use.
It is an open source automated testing framework; where NG of TestNG means Next Generation. TestNG is similar to JUnit, especially when testing integrated classes.
TestNG eliminates most of the limitations of the older framework and gives the developer the ability to write more flexible and powerful tests with help of easy annotations, grouping, sequencing & parameterizing.
Benefits of TestNG
There are number of benefits but from Selenium perspective, major advantages of TestNG are:
It gives the ability to produce HTML Reports of execution
Annotations made testers life easy
Test cases can be Grouped & Prioritized more easily
Parallel testing is possible
Generates Logs
Data Parameterization is possible
Architecture of the testNG framework
pCloudy
pCloudy platform provides single click access to real Android and IOS devices directly from your browser. Use these real Mobile Devices to click-and-perform Manual Testing and execution of Automation Testing for your App. Integrate our platform with CI for continuous Automation Testing and test your App on multiple real Mobile Devices with every change of your App.
Now it’s very much simple to integrate your existing TestNG project with pCloudy.
Mobile apps can often crash for many reasons. Nothing is as excruciating as memory leaks. It starts with your app crashing often, or becoming slow or maybe making your entire phone slow down, taking longer time to load activities. Memory leaks are notoriously hard to detect, and thus can easily surprise you. Have we ever thought why most of the users uninstall the application after some use?
Which is why it is critical for Developers to understand the memory consumption by any mobile application. How do you detect memory leaks during Mobile Application Testing? And if so, how can you prevent them from hogging mobile resource consumption?
Memory is one of the key resources of the mobile device.And memory leaks in Android is an important thing to be kept in mind. Java is a garbage collecting language, which removes the need for developers to dig deep to manage allocated memory. This reduces the chances of any segmentation fault crashing an app or any unfreed memory allocation from eating up space of the heap area, thus creating a safer code.
However, a garbage collector is never an insurance against memory leaks. There are other ways by which memory can be leaked within Java. This means that your android mobile application is still pretty prone to wasting unnecessary memory allocation and crashing with out-of-memory (OOM) errors.
Figure 1: Image Credit- Toptal
Why are Memory Leaks Bad?
Cause: Memory leaks occur when some variable of the application still has references to some objects that are unnecessary, no longer in use or used to be with the app anymore, is bloating the allocated memory heap and the Garbage Collector is not being able to release that memory.
Effect: Android devices mostly run on phones with limited memory, so as a result of too many leaks, the app runs out of available memory. This triggers more frequent Garbage Collector events which paralyze the device (Stop-the-world GC events) by almost stopping the rendering of UI and processing of events. This leads to an Out of Memory Exception and translates to the user as the app being unresponsive.
Testing for Memory Leaks
Memory Leaks are caused by the failure to not be able to de-allocate memory that is no longer in use, which is why they cannot be identified during black box testing. The best phase to find the memory leaks is when developers are developing the application, at a code level.
It’s best to check for memory leaks during unit testing. However, testers can follow these tips:
Release Unused Memory: Memory leaks can be present because of the bad design of the application and thus might consume lot of memory when the application is in use
Test on Multiple devices:The memory consumption by the application should be monitored while doing the system & functional testing. One shouldn’t rely on a single device testing as there are plethora of devices in the market.
Detecting Memory Leaks
There are quite a few techniques and tools devised by developers to pin point memory leaks. Android’s very own Android Studio has a powerful tool to monitor not only memory usage but also for network, GPU and CPU usage as well, called the Android Monitor.
Memory Monitor in this helps track memory usage, find de-allocated objects, identify memory leaks and helps get an overall sense of how your app allocates and frees memory. It’s done in three steps:
1. During and use and debugging of app, track the memory monitor. If the memory usage graph rises for your app and doesn’t fall even if put in background, it’s a memory leak.
2. Using the Allocation tracker, you can check the percentage of memory allocated to different types of variables in your app. This gives you a sense of how much memory is consumed by which object.
3. Create a Java Heap Option to create a heap dump that keeps a snapshot of the memory at any given point of time, thus providing the data consumption data.
Despite Android’s monitor to detect memory leaks, there are a few platforms from developers that help extract memory consumption data without so many steps for faster testing cycles.
pCloudy’s very own platform for example, generates the statistics for Memory consumption while doing the normal functional Mobile Application Testing on real mobile devices.It also generates other vital data such as CPU consumption, battery consumption & data consumption for testers and developers who reiterate several test cycles across multiple physical devices.
If you are anything like me, you’d agree that the armor of present day mobile phone technology has a few chinks on it. Nothing is worse than applications draining the battery resources of your mobile device faster than you can fill it back up. The sunnier side of course, is that you can create teachable moments for your kids better by hiding their phone chargers and watch the colors drain from their faces.
Studies show that battery life is a top priority for smartphone buyers. In fact, just recently, an IDC survey showed that 56% of Android buyers, 49% of iPhone buyers and 53% of Windows Phone buyers said that battery life was a key reason when they bought their particular device.
It is rare to find mobile app testing tools that assure proper usage of resource by the mobile app as, included in the overall test plan and strategy.
It’s rather a tedious and long process for conventional mobile app testing methods and as such, Let us talk about how resource consumption as an app takes top priority, because nothing stops a user to uninstall their resource draining apps from their phone. Also we’ll talk about how developers and testers efficiently test mobile apps for Android applications.
Figure 1: IDC Survey
Why battery testing is important:
Thousands of new mobile apps are being launched every day. And these apps have gone beyond just utilities, games and shopping apps, nowadays, apps need to be integrated into self-driving cars, digital assistants, wearable devices etc. Billions of users need to install apps that are not only compatible with their varying devices, but also provide quality experience of the app so that it doesn’t prompt the user to uninstall it and move to an alternate app.
When you get worried about potential battery abuse, its fair economics to identify the markers that can cause increased consumption of energy by the mobile application. Critical are the functionality, usability, performance and security to make sure that the app does what it was meant to do, seamlessly. The growing need for perfect enterprise and consumer apps and the continuous need to upgrade the application to match it with what the consumer exactly wants is a job that creates high pressure for developers and testers alike, who are responsible to delivering them.
We would get deep into the best practices and tactics that should be kept in mind while testing for battery drain in your mobile app in a later blog post. For now, let us look at how developers can test for battery consumption during Android App Testing.
How to test battery consumption for mobile apps:
So far there hasn’t come a good automation method to test app battery drain. So testing Real Mobile Devices meets many requirements for efficiency in the process. Let us check out a few parameters to test battery drainage due to mobile apps.
Battery test flow
This will vary based on the functionalities of the applications. But overall the below approach or points should be considered while testing for battery consumption.
Check the status of the battery before the test begins
Enable the location services for the application if app supports
Start the data sync of the application if app supports
Start the streaming service of the application if app supports
Check if the application send/receive the data when in the background
Check if the application send analytics to its server
Observe the battery consumption while doing above supported features by your application.
This will give us an idea as what is the battery consumption by the application.
Testing as end user
It is very critical to test the environment needs of the app from a real user perspective. Environments like device types, operating system, background popular apps installed and running, network conditions (WiFi, 2G, 3G, 4G, Roaming) as well as diverse set of devices with different battery states, brings DevOps teams as close as they can come to end users.
Android App Testing across varying devices
Android is guilty of OS fragmentation due to its open approach to OEM vendors. Different devices have varying battery capacity and each loses its battery chemistry over time, negatively impacting performance and battery hours. This is why we must check the battery consumption for the mobile app on variety of mobile devices. The best practice is to use multiple devices from device labs/ device clouds.
Different tools available in the market.
Android’s built in Battery App
Your Android Mobile Phone has a ubiquitous battery testing resource, as it comes preinstalled on all Android mobile devices. Android’s native battery analyzer, for example, (Settings> Device> Battery) is built into Android’s OS and helps analyze each app and let you know which are the ones draining the most fuel.
Commercial Battery Saving Apps
There are a few commercial Android Battery Analyzers which are available for free in the market, like:
GSam Battery Monitor can reset testing cycles as you go, which means there is no need for draining and recharging the battery like in android’s native battery app, and monitor app power usage without having to wait to get an information reset.
Clean Master on another hand, lets you know through its notification feature as and when an app is causing the phone’s battery/CPU to drain, which gives instant information on any CPU overuse.
pCloudy platform for Testing Battery Drain
pCloudy platform allows comprehensive features for Android App Testing. Moreover, Using pCloudy’s platform, one can monitor the battery consumption by the application. additionally, the user can also monitor memory, CPU and data consumption by the application.
The user can do the complete functional app testing on the application by touching all the critical/non-critical paths in the application. The pCloudy platform provides the necessary data for the work flow.
Conclusion
Google has said that this year their focus is on the ‘vitals’ of mobile phones, battery being one of the critical four. It’s always best to use varying angles of attack to test your app’s battery usage. And the only way to do that right now is to cover as many mobile device variants in the market and test against them.
To check battery consumption during Android App Testing Click Here.
If recent past has been any indication, then it is a certainty there are growing expectations from Testers and Developers alike, to take quality head on, as a joint feature. More so in the Mobile App Testing projects where changes are required faster than ever.
As part of DevOps practices, it’s has become imperative for developers to run as many tests as possible with every code check-in. These tests could be automated functional, API or Unit Tests. Some of the popular tools to used by Developers to create their tests are Espresso, XCTest and Appium.
Following are some the Challenges faced by developers:
Developers need access of right set of devices across different versions to be able to run their tests.
Debugging capability on those devices so that they can fix issues quickly.
Access to a specific model of devices to debug production issues.
Try taking a look at a typical developer’s cubicle and you will see a series of mobile devices connected with several long USB cables running into computers. It does get frustrating to see others furiously plugging USB cables in and out of the mobile devices for App Testing on various devices.
Many of the organizations are shifting to device cloud to provide their teams access of right Mobile Devices. Device cloud are solving the need of test teams but provide limited debugging capabilities and hence not preferred by Developers.
To directly address need of Developers, pCloudy recently introduced DeviceTunnel, which fully allows developers to take complete control of the device in cloud. This unique solution provides access of cloud devices through the Android Studio or Eclipse IDE and the command line tool installed in your computer.
It works as if Device is connected directly to your computer through a USB cable. From the point of view of tools like Android Studio or Eclipse, a cloud-based device appears physically attached. In reality, the Device Tunnel communicates with pCloudy’s servers over Internet.
Access devices directly from your terminal
Once a connection is established, the developers can perform the following actions on these devices:
Issuing a range of ADB commands for debugging, shell creations, port forwarding, and viewing general information about any cloud-based Android device
Copying and pushing files to connected cloud-based devices
Installing and uninstalling applications
Debugging apps during development or testing by adding breakpoints, inspecting variables, analyzing run-time metrics to optimize your app and more
Run their tests on the device directly from their IDE
To know more on how to connect any device on pCloudy using Device Tunnel Click Here.
It is undeniable that developers and testers need quicker access to diverse devices for the brisk evaluation of app and debugging. Device Tunnel enables both sets of engineers to instantly connect to any device hosted on cloud and run faster debug sessions and thereby, maximize the quality of their build cycles.
It can be a big challenge to build a large scale Mobile Testing Lab from scratch. Look around you and you will see that with every single day, companies in adding mobile apps in their business strategy and with it the mobile app dev and testing market is becoming increasingly competitive, dynamic and fast paced. Older testing methods are becoming obsolete and the utter multiplicity of mobile platforms, devices and networks have made it important for any company to choose the right solution (Mobile Testing Lab) in order to strengthen themselves in the market.
While creating a testing program for these mobile applications can seem like a relentless chore and a massive undertaking, it doesn’t really have to be. Here are a few considerations to choose the right lab strategy for testing your mobile apps.
Key Considerations:
• Compatibility of Device OS, Screens and OEMs: The sheer number of different device variants, OS versions and screen resolutions form a large set of factors even though each of them are significant in their own rights. In 2012 there were about 4,000 Android device models on sale. 2015 saw about 24,093 distinct Android devices. Question is, can the lab I choose, cover maximum number of devices and come close to 100 percent of my end users’ device base?
• Device Control Infrastructure: After you select your target devices, it is also key to look at the other parts of a reliable architecture of your hardware. Regardless of the technology to be used in building the device lab, one needs servers to control and take care of managing devices and execute tests. Moreover, it is crucial for these servers to collect, process and store results of the tests seamlessly and without interruption.
A snapshot of Infrastructure needed to create a Mobile Device Lab
Facebook Device Lab Infrastructure
Click Here to know about pCloudy Device Cloud Infrastructure
• Wi-Fi Infrastructure: This is another very crucial area that is often ignored when creating large-scale test lab. As the number of device in a WiFi network adds up, so do problems when all these devices transferring data at the same time. Most WiFi access points are not designed for this kind of bandwidth and you are bound to see different types of timeouts on server responses.
• Importance of Automated Testing: The very obvious benefit of automation of testing of mobile devices on the cloud is that through this one can carry out tests on a wider range of OS and devices in a much shorter time and with lesser life-cycle management investment. This not only significantly reduce QA spending, it also expands coverage and speeds up the resolution of issues. You can use a single script and apply the same on different devices and operating systems.
Integration with CI/CD pipeline:
Today, almost all organizations have mobile apps and for some, the mobile app is their only way of interacting with customers. With this increasing emphasis on mobile, the pressure to routinely update mobile apps means embracing Continuous Integration (CI) and Continuous Delivery (CD) methodologies. Any Mobile Labs has to support the CI/CD process.
1. Using Real Devices: Some Devs/testers are using emulation technology for compatibility testing. However, it has been proven beyond doubt that testing on emulators is often not reliable. Real devices help you and your team to find real bugs in your App before customers do. It is only way to have a confident App release and increase the chances of success of your test lab.
2. Tackling Multiple Devices: With thousands of different devices, it can be a bit overwhelming when building a mobile testing lab that encompasses the coverage of testing in all of them. Luckily, the major mobile operating systems use logical screen sizes which are mapped to physical screen, hence, the representative devices will get the necessary coverage. The test strategy is not to test absolutely everything, but to test the crucial elements that are most represented in the popular devices in the market, and add or subtract devices as they come in and out.
3. Using a Secure Mobile Device Testing Cloud: Using a secure cloud is vital to enterprises, especially if they aren’t located under the same roof. Testing real devices for everything can become really costly and time consuming. Using a testing cloud keeps your budget in check, reduced project cost and thus helps achieve high return on investment. Be it public cloud for small businesses, or large enterprise projects that demand a private cloud infrastructure, high performance and security are essentials to have complete control over the cloud.
4. Automation Strategy: Creating Regression Automation suites once Application is ready is a passé. Agile methodology and CI/CD process demands automation creation in parallel to development. Automation strategy should be built keeping above aspect in mind.
Here is a depiction of what the automation process should look like.
As part of Automation Strategy, Mobile Testing Lab should provide the capability to allow automation run on multiple devices in parallel.
5. Increase Lab efficiency by integrating with your existing tool ecosystem: A lab is as good as how well can it be integrated within existing ecosystem. Can it integrate with your Test Management system or can it log bugs automatically after a failure? Can it integrate with your build management tool for CI process? Here is a depiction of how “Test Tools” fit in the larger ecosystem.
Conclusion: A well thought strategy for setting up of Mobile Testing Lab is a necessity for every organization undertaking Mobility projects. In current times, organizations have plethora of choice related to setting up of Mobile Testing Lab. They can choose to setup an In-house lab or use a private-hosted service or use a cost effective Public Cloud lab.
Would you like to know more about Lab options with pCloudy? Click Here
Test Your Apps function differently in different Network Environments
There might be a whole set of bugs you probably are marking as “Cannot Reproduce” simply because you are thinking “Hmmm… Funny, it seems to works here”.
The App you tested in your lab environment will not work the same way when you use it in an elevator, or in a basement, or while you are travelling on a highway. These are probably a whole set of testing scenarios you probably haven’t thought of.
You Can do now
Mimic different real network environments that impact app functionality and performance
Create different Network Profiles and test you apps on multiple Network Environments
Identify and rectify functional and performance issues
Modify various parameters like bandwidth, packet loss and latency during testing
Hence, more and more businesses need to churn out apps for their growing users, and developers having to churn out more and more apps faster, with qualitative inspection of each application that shouldn’t lag, crash, have bugs or glitches, drain phone memory or battery and work anywhere in the world from a smartphone. Added to that, 70 percent of apps get deleted by users within a couple of minutes of their installation.
