How to Get Started with Regression Testing

Regression testing is something that sounds so simple, so logical and an activity that should be obvious to what it entails. Yet time and time again, it is something that can cause problems for projects and software delivery. Usually in the time it takes to run the tests, inappropriate tests being used for regression or the very fact that there may be no regression suite to even execute.

Regression Testing should not be an activity that is only thought about towards the end of a project or a release. And it isn’t an activity that can be rushed or done as an after-thought. Regression testing is one component that makes up the matrix of quality within a software development cycle. But it is an important one, and one that often has a key bearing on the releasable state of the software and the confidence the team and management may have in releasing it.

In an ideal world regression testing should be considered at the time of User Story elaboration and analysis. As in, what parts of the story will need to be covered by regression, what data and other pre-requisites will be required to enable this. Regression Testing should not be re-executing all of the functional tests over and over again. Functional tests are likely to be too granular and take too long to execute for regression purposes. And in the world of Agile with lighter weight test charters, individual domain knowledge would be key to perform regression testing. Which then creates risk by relaying on individual people, as well as a likelihood of non-reproducible testing.

The key things we want to achieve with regression testing are:

• Tests designed to give assurance that the system is still fit for purpose and capture major changes from the baseline – most software is there to support or enforce a process. It is this logic (business process) that a regression tests needs to validate is still possible in the system. Untracked changes shouldn’t be committed to the code baseline, but regression tests should aid in highlighting these if they do occur.
• Reproducible tests – this gives consistency and a known coverage.
• Resilient tests – the tests need to be checking the “use” of the system and shouldn’t break due to some minor changes. The maintenance of the regression suite should have a minimal overhead where possible.
• Tests that are easy to update and easy to get coverage metrics from.

Regression testing is checking for fitness of purpose while ensuring what you could do in the system previously can still be done.

A regression test shouldn’t be trying to check every minute detail and every piece of functionality within the system. For most software the scope would be too vast and any regression suite attempting this will likely end up being high maintenance, brittle and take a long time to execute. And more than likely always out of date with the software.

Regression tests can be automated, manual or even a combination of both. But the earlier they are thought about and the more structured the better. As an ideal the Acceptance Criteria for a User Story should form the framework of the Regression Test for that Story. If the Acceptance Criteria can still be met, then the software is still fit for purpose. Structuring your Regression Tests this way also aids in examining coverage as well as tracking more brittle areas of the software. Where you may decide to have higher coverage or more detailed regression tests in those areas. This may entail the development/project team learning how to write good usable and deliverable Acceptance Criteria, but the longer-term benefits are tenfold. And can even form the basis of an automated Release Note format for a software release as well.

If the foundations for regression testing can be established early on, this will enable regression testing to form a more active part of the software development cycle all the way through. And help reduce re-work while promoting confidence in the solution. Automation can aid in this and allow regression testing to be conducted out of hours or as part of a continuous integration build strategy. Excessive automation may become a costly overhead and result in a more brittle suite of tests.

Areas of the system that are proven to be more fragile to change and more likely to cause problems, can be tackled by risk analysis. Additional more detailed regression test might be added for these specific areas. Streamlining of the regression suite will be beneficial. Reducing duplication of tests will reduce the time taken to execute the suite. Collating the tests into wider End-to-End flows through the system may be a way of achieving this. If every test requires you to “Log in” to the system, then there is no need to have a separate “Log in” regression test.

User stories are ranked in Business Value, this same principle can be applied to your regression suite. Meaning you test the highest business value areas first. To improve test resilience, consider making the tests as data agnostic as possible and remove as many pre-requisites as you can. If a test fails to run, you haven’t actually tested anything.