Automation testing forms the core of any CI/CD pipeline and enterprises are keen to practice test automation to enhance the efficiency of the development process. Test automation saves resources and reduces the cost of any project in the long run. But there are some important points to keep in mind while testing to avoid automation failure. Let’s have a look at these salient points.
Leverage Parallel Execution
Once you are done automating the test cases, the challenge will be the complex test suites taking a long time to get executed. It affects the quality of the test queue in the test automation framework or IDE. This leads to queue timeout issues and test cases being halted abruptly due to the sequential execution of the test cases.
Parallel execution in different test environments is preferred over sequential execution as it saves a lot of time. Although in automated testing, unintended code interactions could happen. This is why you need a thorough reporting mechanism to debug the causes of test failure.
Pick The Right Tools
Choosing the right tool for test automation is critical to the success of automation testing. There has to be a set of clear requirements/parameters on the basis of which the tools have to be selected. Some important points that are to be kept in mind while selecting the tool are:
The team should be clear about the test tool requirements.
The testing requirements of the application under test (AUT) should be analyzed thoroughly.
The team’s skill set should be accessed accurately.
The cost-benefit analysis should be performed to calculate the return on investment.
Tool vendor and capability should be evaluated as technical support might be required while using the tools.
One tool might not be enough to meet any organization’s automation needs. Also, test automation engineers have to be a part of the tool evaluation process so that they can help in selecting the right set of tools. For example, you can use Appium for test automation but you need pCloudy to perform automation testing on multiple real devices in parallel.
Analyze The Test Reports
Test reports provide insights into the underlying issues that are to be resolved. A detailed test report gives an idea of the efficiency of the test automation and the automation team can analyze the report to look for the scope of improvement. While selecting an automation tool you need to make sure that the tool generates test reports to be analyzed by the test automation engineer. There will always be some tests that will fail to execute and it is necessary to analyze the test report to get an understanding of the scenario.
Test Automation Metrics
Test automation metrics will help you gauge the quality of the tests performed based on some essential parameters like test duration, unit test coverage, path coverage, number of defects found, percentage of broken builds, etc. The test metrics will give you a clear picture of how well the code is tested. In an agile process, there are frequent iterations to the builds and it becomes important to track the quality of each build. With test automation metrics you can figure out what is the percentage of your tests that passed and what was the reason behind the failed tests.
Optimum Device Coverage
Test automation is effective when the tests are executed on multiple devices in parallel. Device coverage is the most prevalent challenge as we have witnessed growing device fragmentation across the world. To ensure the smooth functioning of mobile apps on all the devices, you need to perform automation testing on hundreds of device-OS combinations.
Test automation should be designed to make the app compatible with most of the popular devices. The frequent release of new versions of OS from both Android and iOS is a major factor that drives device fragmentation. The only way to overcome this challenge is by testing the app on a cloud-based testing platform. In pCloudy, you will get the benefit of testing the app on more than 5000 device browser combinations in parallel ensuring optimum device coverage.
Summing It Up
Test automation has many benefits like better test coverage, faster feedback, and accelerated results which reduces the time to market of any application. Using the practices mentioned above you can ensure coherent test automation and increased productivity. Apart from these points, there are many other things you can do like writing original code and not copying it as the code taken from other sources might not work in your test environment. But you will always find new roadblocks which you will have to deal with spontaneously.
Intelligent Digital Mesh is the entwining of people, devices, content, and services enabled by digital models, business platforms and a rich, intelligent set of services to support digital business. We have witnessed the implementation of AI in every technology to leverage the benefits of autonomous systems. Enterprises are now focusing on using AI with technologies like blockchain and immersive technology which will create new categories of apps. In this type of environment, attaining optimum device coverage will be essential to ensure quality services. Now let’s understand the fundamentals of the intelligent digital mesh.
Intelligent
In the near future, most of the mobile applications and services will use artificial intelligence or machine learning at some level. AI will be the inconspicuous force of most of the popular app categories while creating some new ones. Intelligent apps also create a new intelligent layer between people and systems as seen in enterprise advisors and virtual user assistants. Augmented analytics is also gaining ground and helping enterprises in enhancing business intelligence and data analytics using ML and NLP. Another use of AI and ML is in intelligent things like smart vacuums, drones, autonomous farming vehicles. Intelligent devices are getting smarter to serve better and reduce human dependency to a minimum.
Source: Gartner.com
Digital
When we talk about digital, we mean digital twins, cloud to the edge, conversational platforms, and Immersive Experience. A digital twin is a digital representation of real-world objects. It offers information on the state of the counterparts, improves operations and adds value to the operations by responding to the changes. In the near future, all the aspects of human life and the real world will be interconnected with their digital representation capable of advance simulation, analysis, and operation. This combined with immersive technologies like AR, VR, and MR will take extended reality to a new level.
Mesh
Mesh is the connection between devices, people, businesses, services, and content to build a digital ecosystem that yields high-quality results. Here mesh refers to technologies like Blockchain, Event-driven, and continuous adaptive risk and trust (CARTA). Enterprises are keen to find new ways to sense the new business events to get the most out of it. A business event can be a change in the status of the deal like finalizing a deal. Using new technologies like AI, it will be easier to detect a business event and analyze it in greater detail.
Security is one of the most important and ever-evolving processes in digital businesses. There is a need to think beyond infrastructure and parameter protection. Continuous adaptive risk and trust assessment is a people-centric security approach that allows for real-time risk and trust-based decision making. New methodologies like DevSecOps and adaptive honeypots should be implemented to strengthen the security of digital businesses.
Automation Testing For Intelligent Apps
Intelligent apps are at the core of the intelligent digital mesh. Nowadays most of the apps use artificial intelligence, machine learning or predictive analysis to make suggestions to the customers. The apps use real-time and historical data from user interactions and other sources to predict the needs of their users.
To ensure the quality of apps it is important to test the apps using futuristic tools. Manual testing is just enough and even automation needs to be scalable to get better results. Testing the app on a cloud-based app testing platform is the best choice as you can use as many devices as you want to test your app. Also, parallel testing increases app testing efficiency by multifold.
pCloudy’s AI-powered autonomous testing bot steals the show when it comes to testing intelligent apps. The bot tests the app on real devices with just a single click and generates a detailed report based on the test result.
Conclusion
Mobile devices, by and large, are the focal point of most of the innovations that are happening around the intelligent digital mesh. Whether it is Ai driven development, autonomous things or immersive experience, mobile apps still used as a foundation to provide the technology to the masses. But the growing complexities of intelligent apps makes it crucial to implement new methods of app testing. A cloud-based app testing platform like pCloudy is suitable to ensure quality at speed in mobile app testing. The freedom of accessing hundreds of real devices from anywhere at any time and perform manual or automation testing using futuristic features is the correct way to test intelligent apps.
Asia’s leading software testing conference, QAI STC 2019 concluded on December 6th, 2019. The theme of this year’s STC was architecting continuous quality: think, transform, and thrive in an Intelligent future. The STC is a platform for experts to display ideas, experiments, and experiences to explore challenges and suggest techniques and innovations to overcome common problems.
Avinash’s keynote focused on the future of testing which was lauded by the top minds and became the highlight of the event.
pCloudy was the title sponsor for the event and so we got a bigger platform to showcase our contribution in shaping the future of mobile app testing.
More than 500 experts from around 130 software companies gathered to learn new trends in testing, share their ideas and grow their network. There were 14 keynotes from the industry leaders and 50 professionals got a chance to take the stage and share their views about the emerging technologies.
Let’s have a look at the major learnings from this event.
Future of Testing
The STC started with a keynote by Avinash which provided insights about the future of work, applications, and testing. How “We Working” will be the primary organizational model and algorithmic management will take over the middle management to some extent. He also talked about how people will have to constantly upskill to work with the ever-changing technology and the work-life challenges will increase in the future. A digital mesh architecture will allow enterprises to build an agile, flexible, and cloud-ready ecosystem. This will enable real-time connectivity of employees, business processes, business data, and services to help address high volumes of traffic and become cloud-native and mobile-first.
Quality Engineering and Digital Transformation
There was a great emphasis on quality engineering at the STC as most of the organizations are trying to take QE to the next level. Quality engineering focuses on the end-to-end management and the basic principle is the all the teams should bear the responsibility of maintaining the quality in the process. Software QE is the assurance of high standards in the software development life cycle while implementing DevOps and Agile.
The main role is played by testers who create, implement and maintain systems used to control the quality of production processes. These people need to have a deep understanding of all the technological activities and evaluation principles.
Quality engineering methodology is even bigger in scale than the traditional QA approach and that’s the reason that QEs cannot work in silos. In quality system engineering, people in multiple roles like IT architects, designers, test engineers, project managers, business architects, etc., must cooperate to meet customer expectations. Quality engineering is driven by emerging technologies such as AI and Big Data analytics. Automation is the driving force behind turning the traditional testing into a more effective quality support model.
The quality engineering team usually partners along with the business users and the product managers for having a better understanding of the required product details to match up the problems since the starting of the product to the last stage.
Artificial intelligence, Machine learning, and IoT
The recent development in AI, ML and, IoT were the buzz creators. Experts elaborated on how augmented analytics will be utilized for creating, developing, and consuming analytics by combining these technologies. An augmented analytics engine can identify, filter, and analyze data, and then recommend what needs to be done next without the need of an IT team. These technologies will make data-driven insights accessible to a much larger set of workers.
The conference turned out to be really productive with good insights about emerging technologies and tools. It was a great opportunity to connect with software testing experts and professionals from around the globe. To say the least, it was a remarkable event where we got a great response and positive feedback from the crowd.
In our previous chapter on Android, we learned about UI Automator Viewer, Which is available on Android SDK, to get the properties of the application object. In the case of iOS, Appium itself provides an Inspector which helps users to locate those elements in the application.
First, open the simulator by clicking on the dock option.
Now in the Device/Simulators window, select the simulator. Open the Appium Desktop and keep the simulator side by side.
Once the inspector is started, select any of the objects on the screen. It will show you the complete hierarchy and properties of that object.
At the top of the window, you can see the Record button which is used to record all the actions taken and record the script.
To select any object, click on the Select Element button and then you can use Tap button to click on an object, Send Keys to enter text and clear to undo the action.
As soon as you perform an action on an object, it is recorded in the form of a script.
Once you are done with the recording you can copy the script and paste in eclipse editor.
In the next blog, we will learn how to write the first appium script for iOS.
DevOps helps enterprises to build software at a fast pace and with minimal issues. The time to market is accelerated and the bugs are fixed faster in continuous deployment with the help of automated tools. AI is much in line with DevOps as the main focus is on automating the process and with AI the system can identify patterns, anticipate issues and provide solutions. The proactive approach improves the overall efficiency of the software development life cycle. So let’s have a look at how AI is transforming DevOps.
Feedback Loop and Correlate Data
The main role of DevOps is to take continuous feedback at every stage of the process. often people use performance monitoring tools to get feedback on running applications. These tools gather much information in the form of log files, data sheets, performance matrix, and other types. The monitoring tools use machine learning to identify the issues early and make suggestions. The DevOps teams use these suggestions to make the necessary improvements to the application. Many times teams use two or more tools to monitor the health of the app and the data from all the platforms can be correlated by the help of machine learning to get a more deep understanding of the app functioning.
Software Testing
AI is changing DevOps for good by enhancing the software development process and making testing more efficient. Whether it is regression testing, user acceptance testing or functional testing, these all produce a large amount of data. AI can figure out patterns in the data collected in the form of results and identify poor coding practices which produce a lot of errors. This information can be used by the DevOps teams to increase their efficiency.
Anomaly Detection
DevSecOps is one of the essential aspects of software development as security is the key to any successful software implementation. Distribution denial of service attacks are increasing and the business needs to prepare themselves to protect their security systems from hackers. DevSecOps can be augmented using artificial intelligence to enhance security by central logging architecture to record threats and running machine learning based anomaly detection. This will help businesses proactively attenuate the attack from hackers and DDOS.
Alerts
DevOps approach might create scenarios where the team receive an overwhelming amount of alerts without any priority tag. This will create ruckus in the teams as it will be very difficult to handle all the alerts in the continuous development environment. AI can help in this scenario by tagging the alerts and prioritizing them so that the urgent ones can be worked upon immediately.
Root Cause Analysis
To fix an issue permanently, a root cause analysis is necessary. Although it might take time to do it compared to fixing the issue with a patch which will provide the instant solution. In order to find the root cause of an issue, the developers will have to spend time which will delay the release of the product. AI can speed up the process by finding patterns in the data collected and implement to fix the root cause.
The collected data can be used by implementing AI to find a pattern and speeding up the development process. The organized data is more useful and makes prediction possible. The best practice is to use machine learning to automate the tasks which are time-consuming which will ensure the smooth and effective functioning of the DevOps teams.
Mobile app automation testing has evolved as a crucial aspect of the mobile app development process to help deliver better quality solutions, under controlled time cycles and cost schedules. But for delivering bug-free app, choosing the best suitable automation testing framework for your app is very important. There are many automation testing frameworks available in the market with exceptional capacities. This blog is all about Appium vs Espresso and we will analyze which of these two most widely used Automation testing frameworks is preferable for your app testing.
Espresso was not preferred because of its flakiness and instability issues. But, from the time Google has brought Android Test Orchestrator, a Gradle test option, instability and unreliability of Android Espresso tests have vanished. This, in turn, is creating a serious problem for the most popular automation framework Appium.
Let’s find out in this blog if Espresso now comes with a power to kill Appium or Appium can hold its stand in this fiercely competitive market.
Let’s get into the details.
What is Appium?
It is an open source, cross-platform mobile app automation testing framework. Appium allows native, hybrid and web app testing and supports automation test on physical devices as well as emulators or simulators. The Appium server uses selenium web driver which permits platform independence and allows the user to use the same code for Android or iOS.
Advantages of using Appium
Facilitates test execution without server machines
Appium is developed using cross-platform runtime environment like NodeJs which enables programmers to write server-side code in javascript. It is designed as an HTTP server and you can run the test without requiring a server machine.
Does not require app code recompilation
Most of the automation testing tools require testers to alter app code. Some of the test automation frameworks require testing professionals to recompile the code according to the targeted mobile platforms. Appium enables testers to evaluate both cross-platform and native apps without recompiling and altering the code that often.
Automates various types of mobile apps
Testers can avoid using different automation tools for different types of apps as Appium can be used for web apps, hybrid, and native apps too. It facilitates the testing of hybrid and mobile web apps as a cross-platform test automation framework. At the same time, it enables testers to test native apps through web driver protocol.
Testers can use real devices, emulators, and simulators
Testers use real devices to evaluate mobile app’s usability and user experience more precisely. Although, to speed up the mobile app testing one needs to use emulators or simulators too. Appium helps testers to produce reliable test results and reduce testing time by supporting real devices, emulators and simulators.
Provides a record and playback tool
In Appium, testers can use the inspector to accelerate testing through record and playback functionality. Appium inspector can record the behavior of native apps by inspecting their document object model (DOM). Record and playback tool can produce test scripts in a number of programming languages.
Testers can automate apps without adding extra components
Testers can execute the same test across multiple mobile platforms without putting extra time and efforts or adding extra component. Appium simplifies automation by keeping complexities in Appium server.
Supports several web driver compatible languages
You can integrate Appium with many testing frameworks and WebDriver – compatible languages including PHP, Java, Ruby, Javascript, C# and Objective C. Hence, a tester has the option to write test scripts in his preferred programming language.
Disadvantages of using Appium
Common gestures
Appium lacks commonly used gestures like double-clicking in java-client libraries. It also does not support Android alert handling directly and the users cannot evaluate alert handling through native API. Testers have to put extra time and effort to test these gestures.
No script execution on multiple iOS simulators
Simulators make it easier for testers to mimic internal behavior of the underlying iOS devices. Although Appium does not allow users to run multiple test scripts on multiple simulators simultaneously.
Lacks the capability to recognize images
Appium cannot locate and recognize images automatically to evaluate games and apps precisely. The testers have to take help of screen coordinates to make Appium locate and recognize images.
Does not support older versions of android
Appium supports only Android 4.2 and later and does not supports older APIs for Android. There are still many people using devices which run on older versions of Android and developers find it difficult to test mobile apps developed targeting older Android API level.
What is Espresso?
Espresso is a tool developed by Google which is used for testing the UI of Android apps. It automatically synchronizes your test actions with the user interface of the mobile app and ensures that the activity is started before the tests run.
Although when you execute an Espresso test you will have shared state in separate tests and some flakiness. For this Google came up with a solution. Android Test Orchestrator is a Gradle test option that helps in testing and increases the reliability of our automated test suites.
If you use Gradle build tools in any version of Android Studio below 3.0 then you also have to update the dependency setup. Let’s take a look at the advantages of using Android Espresso.
Advantages of using Espresso
Integration with Gradle
The new Android Espresso now has the power of the Android Studio and Gradle that comes along with it. So now invoking your tests, running it or modifying it is just a matter of calling a Gradle command. This gives the full power of command line to the developer and makes testability much easier.
Test Orchestrator
The new Android Espresso comes with the power of Android Test Orchestrator that allows you to run each of your app’s tests within its own invocation of Instrumentor. It ensures that there is minimum shared state and crashes being isolated. It allows you to filter the tests that you want to run and also distribute tests across devices. This implies that you have finer control over how your tests run.
Less flakiness
The scalability of the test cycle in Android Espresso is high due to the synchronized method of execution. A built-in mechanism in Espresso that validates that the object is actually displayed on the screen. This saves test execution from breaking when confronted with “Objects not detected” and other errors.
It’s easy to develop Espresso test automation
Test automation is based on Java and JUnit which Android developers are familiar with. There is no setup or ramping up to implement quality in the in-cycle stage of the app SDLC.
Reliable and fast feedback
Android Espresso does not need any server to communicate with, instead, it runs side by side with the app and delivers fast results. It gives fast feedback to the code changes so that developers can move to the next bug fix.
Simple workflow
Espresso allows developers to build a test suite as a stand-alone APK that can be installed on the target mobile alongside the app under test and be executed quickly.
Disadvantages of using Espresso
It requires access to the application source code
Without the source code, you won’t be able to do anything. Also, There is a risk to get used to the in-built test synchronization and UI – then it might be hard to work with WebDriver.
Narrow focus
If UI tests are required for both Android and iOS, it will be necessary to write twice, for two different systems. If tests require to work with Android outside the application (for example, open a received notification with a text message), you’ll have to use additional tools, such as UIAutomator.
Knowledge of launching Android app on emulators required
It is desirable to have at least minimal experience of building and launching Android applications on emulators.
Conclusion
Appium and Espresso both can be used to perform UI testing on Android app but if you have to choose one of them then you need to decide on the bases of your requirements. What kind of app is it and what kind of testing you want to perform. Developers who want to perform UI testing for their native Android app should go for Android Espresso. Although, if the test needs to support iOS and Android both and you want to test at a functional level then you can use Appium.
Testing of Mobile Apps in quite cumbersome because of sheer magnitude of testing required on variety of devices. Moreover, Mobile Apps require changes faster than other kind of Applications (Web or Desktop). That’s the reason, more and more organizations have started realizing the need of using automation testing over manual testing as much as possible.
Mobile App Automation Testing can be a massive undertaking, and if unaware, one can end up complicating the process by selecting a bad tool. With a major trending shift to open-source mobile test automation tools, there have been a plethora of tools available in most open-source software markets.
So how do you know which are the best software testing tool available in the market? Which tools will give you the most efficient solution to fulfill your enterprise’s need for speed and integration? Will manual testing suffice your app testing needs?
You would need a set of criteria to fulfil when assessing your selection of the right open-source automation tool. Here are a crucial few questions to ask:
Do you have the required skilled resource for automation tasks?
Is there ease of script development to support agile processes and shorter iteration cycles?
Does the tool support cross team collaboration for seamless use by QA and Dev?
Can it match app platform with test development language?
Will it have performance capabilities gaps while testing?
Will it support both real devices and emulators?
Does the app support multiple platforms — Mobile and Web?
Does it have multi device execution capability
How easily can it integrated with external Device cloud platforms?
Best Open-Source Mobile Testing Frameworks to use:
To take the final call, testers must have a strong awareness of the tool’s strong and weak aspects, what it can do and what it cannot, and find a balance between cost and benefit.
These are top highly adopted open source test automation frameworks available in the market. Each of these frameworks are backed by different communities due to their unique offerings to the target audiences and relevant platforms. The overall benefits are that they cover a wide range of devices. However, for technical clarity it’s important to know the pros and cons of the framework based on your mobile and web testing needs:
1. Appium: Widely adopted, it is the leading open-source test framework for mobile app (Android, iOS) test automation.
Pros:
Best suited for QA teams to test the functionality of mobile web, native and mobile hybrid apps across iOS and Android.
Its reports are limited from debugging and fast feedback loop.
Supports development tools using any WebDriver compatible language including Java, C#, Ruby etc.
Cross Browser Support and cross platform capabilities
Con: It is less suitable for performing and developing unit testing.
2. Calabash: It is a Behavior-driven development (BDD) test framework based on Ruby development language.
Pros:
Has a large community support
Cross platform development support (Android and iOS)
Provides solid reports and insights to QA and Dev teams
Easy path to both develop and test features in parallel
Simple and easy-to-read test statements
Con: It is not friendly to languages other than Ruby.
3. Espresso & XCTest UI: Both are very similar tools as they were designed for the target users. Espresso for Android and XCTest for iOS are fully maintained by Google and Apple, assuring the latest features for respective platforms.
Pros:
Latest feature integrations assure lead in market curve for developers and testers
Easy to develop techniques including test recorders
Support both types of unit testing and functional UI
Con: Both are app context only, which means limited ability to test for user condition scenarios
4. Selendroid: An open source automation framework which drives off the UI of android native, hybrid and mobile web application. A powerful testing tool that can be used on emulators and real devices. And because it still reuses the existing infrastructure for web, you can write tests using the Selenium 2 client APIs.
Pros:
Can interact with multiple Android devices and simulators simultaneously
Can simulate human actions like touch, swipe, drag etc. on devices
Supports development tools using any WebDriver compatible language including Java, C#, Ruby etc.
5. Robotium: Widely adopted open source Android test Automation framework.
Pros:
Easy to write powerful test scenarios
Full support for native and hybrid Android Apps
Easy to use recorder
Handles multiple Android routines automatically
6. EarlGrey: EarlGrey is a native iOS UI automation test framework that enables you to write clear, concise tests. It integrates with Xcode’s Test Navigator so you can run tests directly from Xcode or the command line.
Pros:
Works directly from XCode
Full support for native and hybrid Android Apps
Synchronization features which automatically synchronizes with the UI and network requests.
