It’s time to predict
Imagine that you developed a mobile app but it does not work properly on some devices as the device fragmentation is getting more complex with time. This will be a tester's nightmare as the users will drop the ratings of the app which will lead to a decrease in the downloads.
Teams need to consider the device usage patterns across geographies to test their apps. This will lead to smooth app functionality across all digital channels. This is the need of the hour.
We can optimize the device and test coverage by using data analytics in app testing instead of relying on guesswork. You can also watch the webinar video where we have covered this topic extensively.
Before we move on, let’s understand some of the terms that will be used in this article.
Terminology
| Term | Description |
|---|---|
| Stock Android | A basic and unmodified version of OS developed by Google. |
| Android Custom ROM | Read Only Memory modified to provide better functionality than that in Stock Android. |
| Manifest file | The manifest file describes essentialinformation about your app to the Android build tools, the Android operating system, and Google Play. |
| Compile SDK version | The version of Android that the build tools uses to compile and build the application in order to release, run, or debug |
| Minimum SDK version | The oldest version that you want your app to support. |
| Target SDK version | The version of Android that your app was created to run on. |
The growing complexities of Device Fragmentation
While 70% of the Apple devices are running on the latest iOS 12, only 0.1% of the Android devices run on the latest Android 9.0. According to statista.com, Android 7.0 is still being used by 19% of the Android users, followed by Android 6.0 with 21.6% Android user share. Around 7.8% of the Android users are still using Android 4.4 which was released in 2013. In 2018, 88% of the mobile devices were running on Android operating system compared to 12% iOS devices.
It is not feasible to run a test on all the devices as the permutations will be huge. There are 16 device display categories, 20 different resolutions, 15 android versions, 400 android device manufacturers and multiple cellular networks. I am sure you will be able to relate to it. So, there are two traditional ways to create an optimum device matrix. First is based on market penetration and the second is based on device diversity. Here is a brief overview of both the ways we have been using on our convenience.
To get a better understanding of the device fragmentation based on operating system, we need to understand the basics of Android architecture. So let's get on with it.
Android Architecture
Android is an open source operating system based on the Linux kernel created for an array of devices. Android has a five-layer architecture as mentioned below:
- Linux Kernel: It is the base of the Android operating system. It gets the
work done from the hardware for e.g. camera, display, Wi-Fi etc. - Hardware Abstraction Layer: It provides standard interfaces that expose device hardware capabilities to the higher-level Java API framework. The HAL consists of multiple library modules, each of which implements an interface for a specific type of hardware components, such as the camera or Bluetooth module.
- Native C/C++ libraries: It consists of some logical instructions that are sent to the kernel to perform an action.
- Android Runtime: It compiles the applications from java code to byte code which Linux can understand.
- JAVA API Framework: The entire feature-set of the Android OS is available to you through APIs are written in the Java language. These APIs form the building blocks you need to create Android apps.
- System Apps: These are apps built in the device like calculator, clock, calendar etc.
In some cases, mobile device manufacturers make customizations in the Android architecture to enhance the user experience. That is done by using custom ROM.
Why bugs occur
One of the major reasons for bugs is the fast evolving Android platform. If you have built an application then you need to ensure that the app works on various API levels that are still used widely. Every new API level launch comes with changes in the guidelines and this raises the chances of app failure. Many brands like Xiaomi, One Plus, Huawei etc, keep releasing new phones and as an update, they customize the ROM. All their devices have diverse specifications like screen size, RAM, processor, camera etc. In the table below you will see the layers of android app stack and how they are divided into two sections the higher level and the lower level.
The OEMs make changes in the lower level system to add a device specific functionality by creating a supporting driver at the hardware abstraction layer. So if the driver has a bug than the higher level system will have a bug too as the APIs communicating with those drivers will also have the bugs. The OEMs also change the higher level system to enhance their UI and these changes also create compatibility issues.
OEMs customizations and its impact
According to Statcounter, this year's data shows that Samsung and Apple have almost 32% and 23% of the market share in mobile sales worldwide. Chinese companies like Xiaomi, Huawei, and Oppo has around 20% market share combined. There are several local OEM's in many countries adding to the diversity in mobile devices. Many OEM uses custom ROMs to provide added features but custom ROMs comes with some duplicate apps.
Most of the custom ROMs like MIUI and Oxygen OS has its own calendar app, music app, and their own browser. UI skins are a significant source of fragmentation in OEMs. This means that we need to consider custom ROMs while creating a strategy for device coverage.
How Android evolved and its impact
Android has released many versions and API levels since its inception. API level is a unique identification code which is given to every framework API revision offered by a version of Android. For e.g., Android 8.0 is API level 26, Android 8.1 is API level 27 and the next version Android 9.0 is API level 28. Developers can specify the compile SDK version, target SDK version and a minimum SDK version in the manifest file of an app as it tells you which app is supported by a particular device.
What we have seen so far
We have observed a trend in the occurrence of compatibility issues and their root causes. We can divide the compatibility issues into two groups, device specific and non-device specific. In device specific issues the first and the most common root cause is problematic driver implementation. For e.g., there was an issue in some devices related to the proximity sensor. If you bring the device near your year the touch screen should get disabled but that didn't happen in those devices. The second root cause is OS customization. Some devices brought the multi-window support feature where you can open multiple application in a single screen. That
feature created a lot of bugs in most of the applications. The third issue is peculiar hardware composition.
| Type | Root Cause | Percentage of Issues | Issue Examples |
|---|---|---|---|
| Device-Specific | Problematic Driver Implementation | 23% | Proximity Sensor issue in Samsung devices |
| OS Customization | 10% | Multi window support | |
| Peculiar hardware composition | 7% | Amazon Fire: OMX Codec issue | |
| Non-device Specific | Android Platform API evolution | 47% | API level 26: 8.0: Internationalization |
| Original Android System bugs | 13% | Android 8.1 Bluetooth issue |
The majority of compatibility issues falls under non-device specific issues. Most common of these are due to changes in the Android platform APIs. In Android 8.0 API level 26, there was an internationalization failure. That means if your app is built to work with different languages than it will show a bug. The second root cause in non-device specific issues is the original Android system bug. These issues are resolved quickly as soon as these bugs are reported to Google it releases a patch to fix it.
How to predict
Android regularly publishes release notes on its forums which talks about the behavior changes for all apps. These changes include power management, security behavior changes, Android test changes etc. They also inform about the behavior changes for new apps targeting the particular API level. This information will help you to predict and plan your test cases. You can select the devices for which you need to plan your testing and the functionalities that you need to focus on where there are maximum chances of failure.
For example, if your app uses the current location of the user, then Android 8.0 will have an impact because android has restricted the number of location update when the app uses the background service. You must have observed a lag in the map location in all the delivery service apps like Ola, Swiggy, Uber eats, etc. Another example can be the background execution limit. Apps running in the background consumes device resources and result in unpleasant user experience. That's why Android 8.0 has limited the background execution limit.
We have created a cheat code by analyzing the data that you can refer to perform testing on Android 8.0 and Android 9.0.
Click on this link to download the cheatcode
Conclusion
It's high time that we say testing is just not enough. It's time to predict. Data has a lot to say. You can utilize it to help you in various ways. When predicted accurately, it can take your testing to another level.
Related Articles:
