Friday, 17 December 2010

Infrastructure Testing – What’s that??? Part 2

Now that you have considered the relevant areas to your particular solution, it’s time to assess which elements pose the biggest risk to you deliverable; there is no secret to how you approach this, here at nFocus we use pretty much an industry standard “Risk Based” approach.

We convene an appropriate forum of subject matter experts and elicit their thoughts on which areas pose the greatest risk, create test scenarios that will mitigate the risks identified, and test the solution, some of the areas for consideration could be:-

  • Security vulnerabilities

  • High Availability / Disaster Tolerance / Failover

  • Connectivity and interoperability between system infrastructure

  • OS Patch Status and updates

  • Load balancing, Network routing and firewalls, name resolution

  • System logging

  • Performance

  • Virtualisation

  • Operational Procedures

Another list of items to consider, but at this stage you should have a pretty good idea of the shape the solution will take and the infrastructure and products that will support it. Some of the work identified during the risk assessment will be undertaken by engineers; some could be delivered as part of a contract via a hosting company for example, however you should assess what level of governance to apply to these activities; confidence in the end product for your stakeholders is paramount.

Related articles
Infrastructure Testing – What’s that??? Part 1

Thursday, 9 December 2010

What mobile device/os combination should we test on?

So, at nFocus we have been doing plenty of mobile app type testing lately and I thought it would be useful and interesting to give you an insight into one particular problem that we have been hitting : What device/os combination do we test on?

I was asked recently to estimate the testing effort as part of a bid for the development of an app that would be based on the apple iOs platform. The first thing to consider was, what are the devices and os combinations that are possible. The result is shown below, there is some great information on wikipedia that helped with the collation of this. What we found really suprised me, a test would have to be run 74 times on individual device/os combinations.

Clearly, that was never going to be possible, so the question is, how do you get smart about which ones you should attempt? The answer to this is not so clear, and depends on whether there is an existing user base, from which data can be extracted. In this specific case the answer was no, so we had to take a pragmatic approach and pick some major combinations to always test on and a more exhaustive list to execute tests on toward the end of the cycle. For our purposes, we got away with testing on 3 device/os combinations with a final pass being executed on an additional 7 combinations.


One of the most important things to consider in making these decisions is to engage the customer, they must be happy with the subset that is chosen and the reasons why.

One of my major concerns when it comes to "App" testing is the perception that because we shorten the word application down to app, that the testing of these solutions does not need to be as robust : "it s just a little app, why does it cost so much to test?" Mobile applications is big business and revenues are growing from the use of such applications, we need to recognise their importance and start testing them properly :)

Thursday, 2 December 2010

Infrastructure Testing – What’s That???

So you’re on a new project; a green field site and part of your remit is to test assure the infrastructure as well as the solution, where do you start? What is infrastructure testing? One definition could be “to assure that the platform the service is to be deployed on fully supports the solution in a performant and robust manner”

Considering this overview, some of the items for consideration could be:-

  • Tin – Commissioning the appliances into the data centre and powering up, what should you be looking for to assure this, and weed out the DOA (Dead on Arrival) kit

  • Network - Making sure that all of the Tin communicates with each other as designed, without any erroneous traffic

  • Tiers – Most internet facing solutions are built using the tier model to separate the internet, application and data for security purposes, should this have a penetration test applied by a specialist in that arena

  • OS – What special consideration and configuration will need to be applied to the software stack specified to support the solution? How do you assure the build prior to releasing the solution code and any supporting artefacts?

  • Monitoring – What system monitoring will be in place and how do you prove its functioning to the solutions specific requirement

  • Code – How can you utilise the functional testing to supply you with data for assuring the infrastructure solution
As you can see, loads of things to consider prior to any developed code being released from the CM process, this has been simplified above, the main point is that it can be planned and executed as a test discipline, utilising both testers and engineers, more to follow...

Related articles
Infrastructure Testing – What’s That??? Part 2