From the Inside Out - Measuring Software Test Success 

Welcome to my blog about measuring the success of software testing. This is an important topic because it directly puts cost benefit analysis under the spotlight. Cost control is at the forefront of everything we do, so spending needs to be closely managed and justified.

The sub-header, ‘From The Inside Out’, is written from inside the delivery lifecycle and from more than two decades of testing experience, observations and perspective of a Test Consultant.

It is, of course, fundamental to understand, agree and document with key stakeholders and suppliers at the start of any delivery partnership. Such as monitoring their success with KPI's, testing expectations, risk management and timelines from the business and financial sponsors.

Success means vastly different things to different people. It's important that the success is achievable, realistic and measurable to evaluate results. It's easy to become sidetracked with procurement, bureaucracy admin or planning that the measurement itself gets pushed to the background. This happens primarily because of two reasons. Firstly, the ‘proof of the pudding’ in terms of how successfully a new code release lands. Comes in the days and weeks following implementation, where live production impacts can be analysed, using real-time live operational data. This comes at the end of the delivery life cycle or deployments if it's a staggered delivery. The second reason is because without procurement, system access, team structure/process and related bureaucracy, the team cannot physically perform the testing to be measured in the first place.

Why do we measure testing?

The obvious answers are to get management information about the stages of testing so that the cost benefits can be monitored. Spending can be tracked and the outputs can be managed in terms of getting value from the spend. Early detection of defects, risks and issues can achieve significant financial savings. This way, informed decisions can be made along the critical delivery pathway.  

Testing is partially an open multiway dialogue with key stakeholders to check up on outcomes, result expectations and 'what if' type scenarios. They would be validating preceding, subsequent processing and considering the wider picture. In some cases, this can lead to a re-evaluation of solution design which may mean changes in code or business process and training are needed.

Some clients operate a staged funding release. This means certain criteria must be met. Also, evidenced to affirm that money spent has achieved the goals set for particular stages in the development. Reassurance that the money is well spent and building confidence that the next stage will follow suit. This will provide expected collaborative benefits and continued cumulative assurance. 

Testing execution is a stage where the code quality is one area reflected in the test results. It's very apparent during test measurement that many other areas are also directly and indirectly measured. For example, the level of organisational maturity can be measured easily through evidence of structure, governance, configuration control or quality of requirements etc.

Software testing success can be measured in multiple ways at multiple stages. Calibre of individuals and the professional qualifications/experience, the communication and interpersonal skills demonstrated from the outset. Quality and comprehensively of documentation output can be evidence of that but its important to note that plans, testing scripts and estimates are dependent upon quality, consistency of information received and effective change control. These outputs are based upon this information. Generally, if the building blocks can demonstrate good quality, there are likely to be successful outcomes.

These are some of the areas you can evaluate along the way to help measure testing success.

Fiscal Spending Tracked Against Milestone Outputs

A tracked spending-based approach can give metrics on success against plan when planned spend and planned outputs are compared with actuals.

Test Scripts – Coverage Against Requirements

Test conditions and test scripts are based upon client requirements. Volume is not in itself a measure of success. However for example, if we can demonstrate coverage by a traceability matrix then it can show that each requirement will have at least one test, assuming that it is possible and testable. Note that this is only preparation stage.

Test Execution: Number of Passed Tests as an Overall Percentage of Total

This can be an indication of success if a satisfactory percentage of total test results passed testing. The number of outstanding tests and type of test must be reviewed so that critical areas are not left out. We can have hundreds of tests but if only 5 are running then it would not be likely to be considered as a success. There must be sufficient time allocated to run the tests, raise defects and retest. Number of tests run, passed, failed at reporting intervals can give clues about potential success. Success of testing is only a small part of any project and its overall success.

Severity of Defects Found

This can be an indicator of successful testing. For example, if 50% of defects found are critical in severity then it's much better that these are identified early and can be resolved prior to release into live production.

Number of Defects

If there are a lot of defects, it can indicate poor quality code or alternatively an indicator of poor requirements. Also, indicates insufficient supporting information, lack of available supported technical or business resources.

A good software testing supplier will be able to help agree and set out an agreeable framework with realistic achievable check points and measures. This will help the decision-making and evaluating success at every stage from proof of concept to post implementation.

User Acceptance Testing

This phase of testing can produce testing metrics but also capture positive feedback and constructive criticism areas which can be recorded and tracked into categories - defects, enhancements, changes and concerns etc. It should not necessarily be assumed that a low number of UAT defects equates to a successful project outcome. Rather, holistically the context and other metrics, outputs and measures need to be considered as a whole and in the context of target against plan.

Post Implementation Issues

This can be used in part to measure testing and project success. If we measure how many live production issues are raised typically in a given time frame before implementation, we can then use these a benchmark for comparison after implementation.

One client did exactly that and collectively, we managed to work out the cost of Severity 1 incidents system outages in their live production environment. We also evaluated the cost savings realised by adopting a testing partner to facilitate quality assurance and improve the change delivery.

However, it's possible to have a successful testing phase and issues raised where business process or certain operational areas are not prepared for the latest changes. In that sense, a project may be viewed as a failure. It's not the remit of testing to check for live production readiness. Although, there can be questions raised in relation to readiness during testing which may help an organisation transition to the new state more smoothly.

Our business can provide your organisation with help in any of your software testing needs, including performing vital operational consultancy. For example but not limited to health check, test resourcing, test automation and any specialist testing needs. We have a wealth of experience in digital transformation, agile/DevOps across a wide range of industries. 

A Recipe for Success: How-to Guide - Test Practice Cards

Are you looking to mature the understanding of testing within your organisation, improve software quality or looking for an event for a remote team meeting?

If so, Test Practice Cards might hold the answer!

Test Practice Cards are a set of digital or physical cards. Each card contains a succinct, pithy sentence with regards to a single aspect of testing. Learn more here.