I was asked recently to estimate the testing effort as part of a bid for the development of an app that would be based on the apple iOs platform. The first thing to consider was, what are the devices and os combinations that are possible. The result is shown below, there is some great information on wikipedia that helped with the collation of this. What we found really suprised me, a test would have to be run 74 times on individual device/os combinations.
Clearly, that was never going to be possible, so the question is, how do you get smart about which ones you should attempt? The answer to this is not so clear, and depends on whether there is an existing user base, from which data can be extracted. In this specific case the answer was no, so we had to take a pragmatic approach and pick some major combinations to always test on and a more exhaustive list to execute tests on toward the end of the cycle. For our purposes, we got away with testing on 3 device/os combinations with a final pass being executed on an additional 7 combinations.
One of my major concerns when it comes to "App" testing is the perception that because we shorten the word application down to app, that the testing of these solutions does not need to be as robust : "it s just a little app, why does it cost so much to test?" Mobile applications is big business and revenues are growing from the use of such applications, we need to recognise their importance and start testing them properly :)