Nitin Kumar

Nitin Kumar

Recent Posts

Best tools for mobile test automation

Posted by Nitin Kumar on Jun 6, 2017 9:16:00 PM

In our previous blog, we covered what mobile testing is all about, and a quick checklist to make sure you test all the right things. In this blog, we take a look at how mobile testing can be made easier and faster, i.e. mobile test automation. 

Why Do We Need Mobile Test Automation?

A mobile application comprises of two parts:

  • the API section, which does the processing
  • the UI part, which does the display and takes care of the workflows

From a testing perspective, it is relatively straight forward to be testing the APIs individually, as we can directly test the endpoints either manually using REST client or Postman, or by using automation tools like JMeter, SOAPUI etc. However, the key challenge that remains is testing whether the apps that are created and the APIs, work in tandem with the UI.

So what exactly is the challenge that we are talking about here? 

Look around you and do a quick sampling. You will see at least 3 to 4 screen sizes, 2 major operating systems (if not 3), and 4-5 operating system versions. A quick calculation reveals approximately 40 possible screen-size-OS-version combinations just from a sample of 10 mobile phones. Now imagine the numbers if you were to create an application which is meant to work on everyone's phone!

The number of possible test combinations can be huge considering just the standard screen-sizes, the OS, and the major updates/releases. With a high frequency of updates to the application, it makes it very difficult for testers to test the application thoroughly in the short time spans available. And this is exactly why we need the mobile test automation, to help reduce the overhead of testing the features manually.

Tools of the Trade

There are various tools that are available to test mobile applications. Let’s take a look at some of the most prevalent mobile test automation tools, their pros and cons, and their best use cases.

The Open Source Tools

Appium: The Appium tool is built over Selenium, with a similar syntax, and has some additional capabilities which allow it to interact with the mobile. With its strong compatibility to Selenium, this is one of the preferred open source tools. 

One of the biggest challenges that we come across when automating a mobile application test is finding the correct locators for the element. Appium comes with a built-in tool- UIAutomator, which helps easily find the locators. Appium also has a recorder that helps record the steps and use them in the form of a script.

In terms of support of applications, it has support for Android and iOS applications along with native and hybrid applications.

If you want to take a closer look at the tool, do check out our webinar on “Simplifying Mobile Test Automation with Appium”.

Calabash: The Calabash tool is another well-known tool in the open source community for mobile test automation. The tool is built atop CucumberJS and supports BDD type of automation. 

The challenge of locators is handled quite well in Calabash. There is a query language that is used here, which returns all the elements that are present on the screen. This has been very well explained here.
Calabash too has good support for Android and iOS applications. The base scripts here are written in Ruby.

Comparison

When comparing the two open source tools, it becomes very important to understand when to use a particular tool. With modern day automation, the preferred practice is to have a single suite for testing both the web and the mobile applications. And that becomes a major factor when choosing which mobile test automation tool to use.

Consider a simple example where we have a web application and a corresponding mobile app. If we have done the automation of the web component using Selenium + Java, Appium becomes an automatic choice. That’s because Appium is built over Selenium. However, if we have used Ruby for the web testing automation, we would prefer Calabash. 

In terms of features and capabilities, both the tools are equally powerful and capable.

BDD is another factor that does have a role to play in the tool finalisation. With QA professionals moving towards a lot of BDD tests written by the product owners, it has become increasingly important that we perform mobile test automation with the same approach. This is where  Calabash has an edge over Appium.

Pro-Tip: Considering the fact that almost all QA professionals have basic Selenium automation experience, I feel it’s best to start off mobile test automation with Appium. With a syntax that’s similar to Selenium, it greatly reduces the learning curve for a team that’s doing mobile testing for the first time.

Licensed Tools

Perfecto Mobile: This is more of a web tool that allows a user to record a script and play the script across different devices. The users are also allowed to extend the scripts by writing specific code that they feel may be needed. 

The advantage of using Perfecto Mobile is the reduction in the cost of procuring actual devices. The devices are present in the Perfecto Mobile lab and each test is run on actual devices where the actions that we wish to perform are performed and streamed online. 

Another advantage that comes with Perfecto Mobile is that you can create a test using any of the devices and the same can be played back on any other device. This makes sure that all the tests work fine on actual devices.

And finally, we can simulate interrupts without any additional steps. There are built-in functions available to simulate interrupt operations
 

SeeTestAutomation: This tool allows you to work on a level lower than Perfecto Mobile. Here, we can actually write code and perform the functions that we want. There is a recorder available to record simulated scenarios. Additionally, there is a SeeTestCloud which allows the tests to be run on actual hosted devices.
 

Comparison

Interestingly, both these tools give more or less the same capabilities, including a mobile devices lab, recorders etc. The difference comes in the built-in features that come with the app. The tool of choice here is SeeTestAutomation, especially after looking at the default capabilities that it supports. These include:

  • Device monitoring
  • Sim card services
  • Mocking GPS location [Important feature in mobile testing]

Personally speaking, I think it’s best to choose one of the open source tools for mobile test automation. It gives you the liberty to grow and expand the tool as required. Meanwhile, you can use the paid platforms just for device support, and also get services like browserstack to access browsers and app platforms for testing.

But that’s just what I think. Rest is up to the automation testers to think and decide.

Meanwhile, you can check out our QA Automation services as well as exploratory and accessibility testing capabilities. 

Topics: QA and Testing

The basic mobile testing checklist

Posted by Nitin Kumar on May 17, 2017 9:08:00 PM

Mobile has become a buzzword for practically every industry that you can think of. Whether you wish to sell a service, or garner a huge user base, or grow a community, you need a mobile strategy. And more often than not, for enterprises and product companies, leveraging mobile inevitably means engaging your customers via an app. 

However, someone has to ensure that all these apps getting created, or websites accessed via a mobile browser, are actually functioning as intended. Are they offering customers the responsiveness and ease-of-use that you want your brand to be known for, or are they delivering buggy, extremely frustrating experiences? And that is why you need mobile testing.

What is Mobile Testing?

Mobile testing is performed to make sure that the website or app under development is delivering the features and functionalities that it is meant for. As with any QA testing, mobile testing involves the following types:

Functional Testing: This type of testing deals with ensuring that the functional behavior of the application is as expected. This includes:

  • Checking and comparing with wireframes
  • Checking if the user flows are as per requirements
  • Ensuring that the UI as per the design
  • Checking if the app works well across devices
  • Ensuring the app works well across operating systems

Performance Testing: Performance testing is very crucial when developing a mobile application. For example, let’s look at a simple ticket booking system. There are thousands of travelers who could be booking tickets for bus, train, and flights. Even if the application is for any one single mode of transport, at the peak time, there could be thousands of requests that hitting the application. This makes it very important that the app is able to handle the load.

Having said that, it is actually the API that caters to the requests. However, we need to ensure that the app does not fail when there are slow responses.

Security Testing: The most valuable thing today is data. And when users interact with a mobile website or application, there are numerous scenarios that involve data security:

  • To make sure that people use an app, it is very important that they trust their data will not be compromised. 
  • Another aspect of security is when there are payment portals integrated with the application. The app needs to make sure that the card details that are entered are not compromised.
  • Also, in terms of data privacy, the app makers need to ensure that the app/APIs are not prone to the common security threats.

Interrupts Testing: This testing is very specific to mobiles. Here we need to make sure that the application runs fine even when there are system generated interrupts. System generated interrupts can be any of the following:

  • Incoming phone call

  • SMS

  • MMS

  • Airplane mode

  • Low battery popup

  • Phone shutdown 

  • Low memory

Installation Testing: This is another key aspect unique to mobile testing. Any application goes through a series of development stages and hence we need to test for the following:

  • Installation using the .apk files on Android, and .ipa files on iOS

  • Installation from Google play store on Android and iTunes on iOS

  • Installation of updates from Google play store on Android and iTunes on iOS

 

Usability Testing: The final key area for testing on a mobile device is usability testing. The tester needs to make sure that the app is easily usable, all navigations are as per the wireframes, and meet the iOS and Android standards. Also, we need to test the application on different screen sizes and make sure that there are no issues in the rendering of the application.

Is Mobile Testing the same as Web Testing?

One very common question that we get to hear to is, what is the difference between web based testing and mobile testing? 

Indeed, testing on a mobile web browser is similar to testing on a desktop web browser. The functional, performance, and security testing is common for mobile and web testing processes. However, there are certain key differences that need to be kept in mind:

  • There are a lot of native events that occur on a mobile device that would not occur on a desktop browser. To name a few would be phone calls, messages, airplane mode etc. And hence interrupts testing is unique to mobile testing.

  • Mobile testing is not limited to browser alone, there are apps that need to be tested. The current user preference is to use an app rather than browsing on a website, even a mobile optimized website. And while testing an app, the perspective completely changes in the sense that we are no longer just looking at the features and functionality, but also having a close look at the UI. There are separate wireframes that are created for testing on Android and iOS, primarily to maintain the design standards and also to be able to utilize the native features available with the OS.

What Are the Basics of Mobile Testing?

Mobile Testing Basics checklistWhile the goal of mobile testing is the same as web testing, there are certain additional elements that get added to the basics of mobile testing. Given that the application has to run on a mobile device, we need to consider and test for certain native conditions that do not exist while operating on a web browser. Below is a quick checklist that needs to be considered when performing basic mobile testing:

  1. App installation -  Test the app installation in different OS and devices with different supported OS versions. 

  2. Interrupt conditions - When the app is loaded or is loading, and a call is received, it generates an interrupt condition. It is important to check if the app handles the interrupt conditions properly.

  3. Offline mode - Check if there are any crashes/issues seen when the app is operated in no network/airplane mode.

  4. Landscape and Portrait orientation - Check if the app supports a change in device orientation. If it does, ensure that upon screen rotation the UI remains consistent.

  5. Location based services - If the app has any location services, it should display the location correctly. It is very important to check this on data as well as wireless network.

  6. Test the app on Wifi and data network - Check if the app works correctly on data as well as wireless medium.

  7. Simulate phone restart interrupt - Check if the app works fine if the phone restarts when the application is running.

  8. Simulate message interrupt - Check if the app works fine if there is a message received when the application is running.

  9. Simulate a low battery interrupt -  When the app is running, simulate a scenario for a low battery popup and check how the application responds to the popup.

  10. Simulate a low memory interrupt - When the app is running, simulate a scenario for low memory pop-up and check how the application responds to the popup.

  11. Security - It is important to ensure that the app being used is secure. There should not be any vulnerabilities in the form of advertisements or any other medium that causes a threat to users’ data.

  12. App social network integration - With almost all applications using Facebook and Google integrations for login, it becomes very important for testers to check if the privacy policy and notification settings are considered in the application.

  13. Performance requirements - It is important to know what is the minimum configuration needed for the application to run properly. Hence we need to do a proper benchmarking of the application to ensure that the application does not hamper the performance of the device

  14. Images - In the case of images used in the application, the images should not get distorted/stretched/pixelated when uploaded.

  15. App installation from App store/iTunes - Check if the app gets correctly installed when downloaded from the App store/iTunes.

This basic checklist helps you make sure that you do not miss out on testing any key feature, scenario, or functionality while performing mobile testing. You can take a look at the most prevalent mobile test automation tools and decide which one to use, depending upon your project requirements.

You can also check out our range of webinars on QA and Testing, especially our session on mobile test automation, to know more about mobile testing and the best practices for mobile test automation.

Topics: QA and Testing

Discussion

Write to us

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms Of Service apply. By submitting this form, you agree to our Privacy Policy.

See how our uniquely collaborative work style, can help you redesign your business.

Contact us