How to test your mobile apps in the right environments

Erin Kinsburger, Technical Evangelist at Perfecto, talks testing in the world of mobile applications, and the right environments to do so.

It should be no surprise that the explosive growth of mobile devices has changed how people interact and engage with brands, each other and the outside world at large. Whether swiping right for your love match, using ApplePay at the checkout, shopping for your next weekend outfit, watching the latest Game of Thrones or simply browsing the web, the mobile app has become integral to the consumer experience.

Like all channels that came before it, succeeding on mobile is dependent on businesses creating a winning user experience. Business goals, customer satisfaction, positive app store reviews, word-of-mouth uptick and, of course, revenue are increasingly being driven through mobile. But in complex digital environments, testing just basic mobile app features under ideal conditions will fall short of delivering a great user experience.

When trying to deliver high quality mobile on a variety of operating systems and networks, it’s important to remember to not only test your apps functionalities, but also the right user environments.

Testing beyond functionality

Most businesses’ mobile app testing and quality assurance (QA) teams carefully test the “functional” aspects of their mobile apps. This means making sure that critical functions such as downloading and installing, logging in, search, payment processing, checkout, scrolling, etc. work. Whilst this is important, developers know that they can no longer create applications assuming they will be run from desktops with steady power supplies and stable broadband connections. By their nature, mobile apps will be used from different locations and under different conditions, forcing your teams to consider various factors and user conditions, such as:

  • How frequently do your users use your app?
  • Are your users always on Wi-Fi or cellular, or are they switching between the two?
  • What other apps are interrupting their usage?
  • What mobile devices are they using to access your app?
  • How does your app perform under weak network conditions?

These unique usage patterns are critical to improving mobile quality and preventing user churn. And in today’s digital age, it’s essential to test for the real user conditions that your customers experience every day.

Testing in the user’s environment

To effectively test real user conditions, you need to be able to replicate the real environments, devices and conditions under which users will be using your app. In making user conditions part of your mobile app testing, there are three practical steps businesses will need to take:

  1. Define the right users for your products

To be able to test under real user conditions, you must first identify what conditions your typical users will likely be using your app under. The first step is therefore to identify your target users – otherwise known as “personas”. While this sounds easy, accurate personas can only be created through market research and data analytics. As such, defining personas relies on consistent communication between engineering and business teams.

The next step is integrating these users and their traits into the test lab. This is usually the toughest task for engineering teams because mimicking real environment conditions is hard and requires ongoing lab maintenance.

Some of the core requirements needed to sustain a lab that tests for user conditions include:

  • Ability to simulate various real life factors into the testing lab, such as network conditions, apps running in the background, phone call interruptions, etc.
  • Support for different geographies to cover testing of relevant users in relevant countries
  • Support for any device and any operating system to sustain lab coverage as market matures and continues to develop
  1. Add user conditions to existing functional tests

The second step, once the lab is ready, is for DevTesters to add the relevant user conditions to existing functional tests in different ways, such as manual testing, data-driven testing or through automated test code. In most cases, the most efficient path will be through automation.

Businesses can implement user condition testing on top of functional test automation code by adding test code lines that define the target. This test code will automatically inherit user traits such as their name, the platforms and devices they use, their background apps, their location, their network conditions, etc.

Integrating these user traits into an existing continuous integration (CI) workflow and running many scenarios simultaneously will pay off in the form of shorter release cycles, increased test coverage and, ultimately, a better user experience.

  1. Use reports to find and fix issues

Mobile app development and testing is a continuous process. Once your team has defined the target personas and implemented the test code correctly, they’ll need a detailed and actionable rich report to continuously drive improvements. Such a report should include network logs, device and app memory and battery consumption details, and timers showing how long it took certain actions to take place.

To have actionable data, these reports must include results for every test scenario and should enable teams to take action to fix any reported issues. To be able to consistently address bug fixes, the report should also compare the specific user condition tests to a “happy path” to show the contrast between real world conditions and ideal conditions.

By understanding your users’ conditions and taking practical steps with the right tools to test in real user environments will allow brands to ensure a high quality experience for their customers – regardless of where or under which circumstances they are engaging with your app and brand.

Edited for web by Jordan Platt