Mobile Device Testing: What You Need to Know

This blog was a collaborative effort, bringing together diverse perspectives and expertise to create a comprehensive piece. We extend a special thank you to all the contributors including Jessica Ward and Jane Kelly who generously shared their knowledge and insights.

Hi fellow readers! I’d like to share an insight into my experience with mobile testing, the compatibility issues across a range of mobile devices and its benefits. I’ve conducted testing for many years and find the same topics being debated: time, cost, money, quality and compatibility. 

Inevitably, this all leads me to wondering if assisted technology can help. However, mobile device compatibility testing takes this to a whole new level because of the velocity of turnover in software versions. Such as hi-tech integrated gadgets, bolt-ons, AI, customisation, peripherals balanced with the intrinsic need for stability, performance and security. 

Using mobile devices in a simulated real-life environment where constraints such as network connectivity, brand, location, age, size, weight; all directly impacts the testing, expectations and performance. Mobile devices sit within a distributed tiered architecture system which adds more layers of complexity.

You may wonder what compatibility testing has to do with mobile testing? Well, in this context, it assesses one or more apps over a variety of devices and environments. The objectives are to evaluate the hardware/software display and verify the functional processes are working correctly. It's important to consider the content, display, size, layout, navigations and functional features, which are all paramount to the user experience. 

Ideally, we prefer writing test scripts linked to customer requirements as it helps manage expectations and demonstrates test coverage traceability. However, some of our clients have smaller budgets and less organisational maturity therefore, we adapt our approach to reach the optimum results for each of our specific client settings.

Before embarking on testing, we must understand more about the attributes (brand, age, range, number of devices, operating system and software versions). We need to check current performance levels and usability to understand what can realistically be conducted. Any differences in the characteristics or attributes of a device could have specific bugs which only affect a particular make/model. It's important to perform a cost-benefit analysis of how much of the mobile user population target market is achievable so it adds value to testing.

There are so many variables to consider. We can use emulators, simulators and device labs to help us with the testing instead: 

• Emulators are written in machine level language, mostly useful for browser testing and easier to debug. It can help evaluate physical sensors, battery state, geolocation etc. 
  
  
• Simulators looks at how a human might interact with it and copies the fundamental behaviours of a real device but on the downside, are more difficult to debug.
  
  
• Device labs help in more diverse ways, for example enabling testing of gestures, but they cost more to set up maintain and use. Device labs can be shared, or cloud based but are less commonly used because of the costs involved. They are useful for browser compatibility testing and offer a rental service to help you perform your cross-browser testing.

They do not fully and accurately evaluate real-world conditions however, as there are so many scenario permutations which cannot be accommodated. 

Subscripting to a cloud-based real-device provider can give you access to a good variety of browser types, devices, makes, models and different operating systems, at a low price which can be paid monthly; compared to buying them all. With a monthly subscription, it gives you more flexibility which is useful to have on a commercial level where costs are limited. I compared my experience of three different providers. Inevitably, there are pros and cons with which provider you choose. 

It's beneficial to choose a supplier with the widest range of application installation options compared to the other providers. You could install your application via .ipa file, .apk file, the AppStore, Google Play Store through TestFlight or through a browser on the device when it’s launched. It may be pertinent to check if you can test on a device in this way before it’s become available to mainstream market.

However, some providers may not offer a wide selection to choose from. Even if they do, they lacked in other areas that 'Provider One' excelled in such as responsiveness and customer's experience.

Rather than having a range of installation options, 'Provider One' only had one option to install via the .apk file and .ipa file which wasn't ideal for my client. However, they were only set up to allow installation via the PlayStore and TestFlight so they had to make. ipa and .apk files work. The .ipa file worked without a hitch but the .apk didn't. 

After emailing back and forth with the customer support services, they eventually figured out the reason. It was due to the application blocking screenshots from being taken which didn’t allow screen recording from the cloud-based real-device providers. The devices therefore had to be reconfigured to bypass this application blocker. This testing process was delayed until it eventually worked. 

Another downside is that they don’t have as many duplicates of the devices so when you need to test on one of the popular devices, you can be left waiting a while before it’s becoming available. Testing time inevitably gets squeezed anyway so it can increase costs significantly, if not managed closely.

If you don’t have a wide scope of devices you want to test on and need to install your devices without the app file on hand, then 'Provider One' would be the most practical choice. If you do have all the required resources, 'Provider Two' would be the best company to go with. The third provider lost the business due to the subscription barriers and responsiveness.

To get more details about the providers and our mobile app testing, contact our team on 0370 242 6235 or email via info@nfocus.co.uk. At nFocus, we are privileged to get access to a real-world data showing what our client’s consumers are using to prioritise the devices we test against and to provide our customers with a bespoke testing plan; tailored to their budget and timescale.