Why do you test so late?

Common problems I’ve seen during my career (if that’s what you could call this collection of happy accidents), are situations where a project's timelines are placed in jeopardy because testing takes longer than expected; or the scope expands after the requirements are signed off, usually after testing starts.

Now, there may be situations where this is a genuine occurrence. Requirements change, possibly regulations change. Availability of resource may change.

However, usually it’s because something is missed. Not necessarily anyone’s fault, it just happens. But when it does happen, it can be very disruptive and potentially very expensive. So how can we prevent it?

... by testing earlier...

Let me outline the benefits of involving testing as early as possible in the project lifecycle:

Bottom Line

At a basic bottom-line level, cost of fixing defects late vs cost of fixing early - the accepted wisdom is that software defects cost more to fix later when they are found in the software development life cycle (SDLC).

According to leading Software Engineer, Barry Boehm, from papers published in 1988 defects were found 'in the field', cost 50-200 times as much to correct as those corrected earlier;  even for small non-critical systems [1], the ratio is still 5:1 [2].

In The Beginning

Although as with most things, it’s more nuanced than that. In other words, there’s more to it than cash. There’s quality benefits that are harder to quantify as well...

Let’s start at the beginning. Involving testing can help the project make better decisions at the procurement stage. For example, a previous client replaced one system with five disparate systems that didn’t naturally talk to each other. That had to use complex integrations to work. By the time it got to testing being involved, the die was cast and whatever issues were found, it would be too late to do anything fundamental with.

Now, ask anyone worth their salt who’s worked on an IT project and they will tell you the most likely point of any in a project to cause issues is/are the integrations.

Sometimes they are unavoidable. Sometimes they are the best option. However, a view from testing of what would be required to prove the integrations work and what might happen if they don’t. It may give a clearer picture and enable a better informed decision.

Scope, Timelines & Budget

Moving through the project lifecycle, it sounds obvious but the earlier the testing function gets involved; the sooner a project can identify it’s scope, timelines and budget requirements.

Testing can help to highlight early flaws in design. By stepping through requirements and process documentation, issues can be identified before sign-off and prior to development starting.

Early walkthroughs or demos that involve testing can help to bottom out scope and effort. It can bring other key people into the process which in turn will help to identify priorities and the critical path. It will also aid training by creating experts within test.

Another example, a previous client in the utilities industry was assessing a potential second phase of a regulatory project. Initial thoughts were that the client was going to just shove it in with minimal testing as it was ‘not a real project’, simply an enhancement.

Testing assessed the ask and involved key users to outline the full scope as we saw it. More importantly, the potential risk of treating it as an enhancement and only executing a minimal regression test of the original project.

The project team took the evidence on-board and signed off a full project. On implementation, the client had the lowest rate of fines of any company in the sector that was affected by this legislation.

Another key benefit at this stage could be identifying parts of the solution that could benefit from automation of the testing effort, saving time and aiding the repeatability of regression testing future phases.

Scope Creep

Scope creep is one of the biggest headache of any project. Just when you think you are over the hump, wham! - you get ‘oh, well we also need it to do this’...

A project can avoid scope creep and late ‘new’ requirements by checking the initial requirements for testability, the design for coverage and relevance. This will also help identify unworkable solutions,  ‘edge’ scenarios and their likelihood/validity.

Another advantage of involving testing at this stage is that it can help identify possible data cleansing requirements. This can not only support a successful implementation but also potentially reduce testing time and the possibility of nasty bugs such as simply improving or rationalising the data collection process (if possible) or ‘cleaning’ existing data can save a project a lot of hassle.

Acceptance & Fitting In

How the new solution will operate within the company’s current and ongoing infrastructure is also something that can get missed in the rush to kick a project off?

Involving testing as early as possible will help to identify Operational Acceptance Testing criteria and flush out any ongoing system support requirements.

Another example, a current client has a 2 level exception process. Records in an API can fail at 2 stages but only 1 generates notifications. Testing only made aware very late in the process. Had they been involved in design/requirements stage, they would have flagged that as a potential error/risk and ensured that a process was in place to so that no errors went undetected?

In a similar vein, it will generate early thinking around Non-Functional Requirements for the solution and also Disaster Recovery. How the solution will cope if record processing fails or access issues arise?

Users

On a more human level, the earlier the testing function is involved, the earlier users can be involved. Once again, if I’ve learnt anything over the years, it’s that people don’t want things imposed on them ‘by people who don’t know how we do things here’.

Bringing key users in, getting them involved and engaged will ease this process and get more acceptance for a new solution ahead of launch. As we’ve discussed earlier, you usually only get one chance with users in terms of whether a new system is any good. The earlier you get them on board, the more likely that first impression will be a good one.

On the flip side of that argument, early involvement of users with testing can prevent the project from baking in the failings of the previous system by working through the calls to just re-platform what’s currently in place.

In summary, the earlier you get involved, the more chance the quality and usability of your solution will be better.

Footnote