Thursday, 9 December 2010

What mobile device/os combination should we test on?

So, at nFocus we have been doing plenty of mobile app type testing lately and I thought it would be useful and interesting to give you an insight into one particular problem that we have been hitting : What device/os combination do we test on?

I was asked recently to estimate the testing effort as part of a bid for the development of an app that would be based on the apple iOs platform. The first thing to consider was, what are the devices and os combinations that are possible. The result is shown below, there is some great information on wikipedia that helped with the collation of this. What we found really suprised me, a test would have to be run 74 times on individual device/os combinations.

Clearly, that was never going to be possible, so the question is, how do you get smart about which ones you should attempt? The answer to this is not so clear, and depends on whether there is an existing user base, from which data can be extracted. In this specific case the answer was no, so we had to take a pragmatic approach and pick some major combinations to always test on and a more exhaustive list to execute tests on toward the end of the cycle. For our purposes, we got away with testing on 3 device/os combinations with a final pass being executed on an additional 7 combinations.

One of the most important things to consider in making these decisions is to engage the customer, they must be happy with the subset that is chosen and the reasons why.

One of my major concerns when it comes to "App" testing is the perception that because we shorten the word application down to app, that the testing of these solutions does not need to be as robust : "it s just a little app, why does it cost so much to test?" Mobile applications is big business and revenues are growing from the use of such applications, we need to recognise their importance and start testing them properly :)

1 comment:

Justin Hunter said...

You might find a tool like Hexawise (that can generate both simple pair-wise test cases and much more thorough combinatorial test cases) to be a useful when facing these kinds of problems. Combinatorial testing methods have been specifically designed to help testers quickly identify which combinations of test conditions should be tested in each test case in order to give them the best likelihood of finding bugs quickly.

More info. about Hexawise (which is free to the first 5 users in a company) is available at

More info. on combinatorial testing is available at

- Justin Hunter
(founder of Hexawise)