A Beginner's Guide to Regression Testing
OK, so I’ve decided to write something about regression testing, what it is, what it does, why and how do we do it. So as the song goes, “Let’s start at the very beginning”.
You will have seen over the years that there have been software calamities when institutions have decided to enhance their software to give, us the public, easier access or make more information available. Unfortunately however, when implemented, the reverse has happened. Unbeknown to the developers, the enhancement has corrupted or affected some other area of the software. This then, rather than doing what it was supposed to do, stops the whole system from working. Regression testing of the enhanced system, before it went live, may have prevented this from happening, because it checks that everything that worked before, still does. It can’t be guaranteed, because unless exhaustive testing is performed bugs will worm their way through. But if it is performed properly then bugs should be reduced.
So, the principals of regression testing are:
- It is the testing of a changed or updated computer program, to make sure the older software features still perform exactly as they did before.
- It is performed by executing a test suite made up from previously executed tests, against the modified code, to make sure that it does not break anything that worked prior to the update or applied change.
- The regression test suite is executed as part of System test, and the scope and acceptance criteria must be agreed ahead of test execution.
- It is performed in a separate test environment, which mimics live, and contains the relevant code and/or changes.
- It will prove the changed or updated computer program still performs exactly as it did before the change or update was applied.
- The results from the execution of the previously executed tests should match the previous outcomes to confirm nothing has changed.
- Where and when bugs are found, they can be captured, documented and hopefully resolved.
- There would be reduced delivery risk and improve business outcomes.
- It will provide evidence for ‘go live’ approvals and/or project decision making.
So, it is decided that regression testing of the system following an upgrade or update is a good idea. How is this planned for, and what actions need to be taken before regression testing can take place?
Firstly, test scripts are selected from previous system test phases, to include the basic functionality of the existing application and its most popular features. Remember to use not just previously passed tests but also select ones that have previously identified bugs and defects. This will then confirm whether bugs that were previously accepted by the project, have been fixed by the implementation of the new code. These are collated into a regression test suite and are then put into priority order by performing some impact analysis to decide which tests to run, should time constraints prevent all regression tests being executed. The tests once selected must be stored in a test management tool such as DevOps, VSTS, Jira, Quality Center etc.
It can be noted here that a regression suite is ripe for automation, and wherever possible it is recommended that tests should be automated. These can then be executed repeatedly, ensuring that the same tests are used, they are repeated in exactly the same way, and they generate the same results. Additionally, of course, these can be run out of office hours, which then frees up the regression tester to work on those tests that for one reason or another cannot be automated, or for the raising of defects. While manual tests are stored in a test management tool, the automated tests and their results will be stored within the automation tools own file structure.
Execution
Once the regression suite has been planned it has to be executed. However, before execution can start the following ‘Entry Criteria’ must be met:
- The test environment ideally has been smoke tested, the relevant code and all changes are deployed, and change control is enforced.
- Exit gate criteria for the prior test stage, (Unit or System test), has been satisfied.
- Relevant user accounts for the system under test are setup and issued to the test analysts.
- Any required test data has been identified, documented and peer reviewed.
Execution should consist of a minimum of 2 test cycles. Cycle 1 should plan to achieve at least 80% coverage of tests, with Cycle 2, (and any subsequent cycles), covering all outstanding and/or previously failed tests.
- Manual test execution is performed by the allocated test analysts, with results being recorded in the test management tool.
- Automated test execution ideally is performed out of hours via the appropriate test automation tool. Tests will be uploaded from the tools own storage system and filed back afterwards together with the execution results. It is therefore suggested that for subsequent re-runs the test suite is cloned and given an incremented name. This is to ensure that previous results are not overwritten and lost.
Defects should be defined in terms of business severity such as those listed below:
- Critical: impacts the availability of the service or platform.
- High: prevents completion of a business-critical process.
- Medium: severe, but an acceptable business workaround exists.
- Low: negligible impact, operationally acceptable and will be managed in accordance with the Defect Management process.
Reporting
Throughout the execution phase it is necessary to advise the project on the current status of testing. Therefore, a summary report must be produced at the end of each day defining the total number of tests executed, a list of defects raised including their current status, details of any risks and/or issues, as well as any updates or concerns the tester has.
Finally, a ‘Regression Test Stage’ exit document, will produced for inclusion in the project Test Completion Report. This will consist of:
- Test Status by functional Area (total passed, total failed, total blocked, total not run).
- Defect Status (total by status and severity).
- Defect Trend (defect status over time).
- A summary of the test outcome and recommendation.
Now I’m not saying that regression testing had not been affected on certain banking systems before they went live, but I have to wonder, if the above procedures had been followed, could the outage suffered by thousands of people have been avoided?
.png)
