Where is the Value?

As we know, we cannot test everything. This is especially true for End to End testing. This activity is often carried out at the end of the Implementation phase after System testing has completed. There will be many paths through the software and to test them all will be a large task. So only critical paths are tested through the system that provides an acceptable level of test coverage. Add automation into the mix, and this can become extremely challenging at the end of the delivery cycle. Teams often accept that the effort required for automating the End to End tests is too much of a risk to a timely delivery. Often, it is cut down to small manual End to End tests as an afterthought that adds little value to the overall quality or post-release teams. Ironically, this is where it adds the most value!

Not at the End testing
Thinking and planning for End to End testing is recommended during the Design of software. Producing wireframes and understanding the high-level workflows through the system and its interfaces will help to frame the whole delivery. It will also help to identify which parts of the system and subsystems can feasibly be automated, and which cannot.

There are 2 main approaches to delivering software, and this will influence how and when End to End testing is carried out:

• Vertical slices – Developing all requirements under each Feature before starting the next Feature. This does not lend well to a prioritised backlog and does not produce releasable software very quickly as requires more of a big bang release. The end user will likely not be able to carry out their main functions in full until all features have been delivered.
• Horizontal thin slices – Developing software based on a MSCW of each Feature and each Requirement. Using the rule of 60% M / 40% S and C, the Must haves will create small logical workflows to produce an MVP (Minimum viable product) and provides the critical functionality to allow the customer to carry out their major tasks. The MVP can be released to the customer in the first iteration, while the remaining Should and Could haves are re-evaluated to be released in future iterations as another slice or to target common pain points.

It sounds like we have gone off topic a little, but this fundamental approach to delivery will dictate how (or if!) automated End to End testing is carried out. If vertical slices through the software are used to deliver, then creating End to End tests could be a long and arduous task at the end of the Implementation phase if the requirements are not delivered in logical order. If automation is considered, then trying to create any automation framework on fully developed software is really too late for it to be successful. Using horizontal slices, or an MVP as the delivery method, the framework for End to End test flows can be planned quite quickly because only the critical paths are being delivered in that iteration and the team will know where the journeys will start and end from the work produced during Design.

Keep it Modular
As User stories are developed, System testing should combine a mixture of manual creative testing and functional UI/API checks that can be automated. For example, assuming the first story will deliver part A of a form, the first check will automate data input and validation. When the second story is delivered, it is likely that the tester will need complete part A and click next before being able to test the second story. To avoid writing the same tests from the beginning each time (this will become mundane after the 15th story), the ‘Part A’ functionality should be wrapped up into a method that can be reused. By using Page Object Model (POM) approach, a modular framework can be built, which is maintainable and keeps automated system tests in their own classes such as ‘loginPage’, ‘homePage’, ‘partA’ and ‘partB’.

Challenges
You might think that building End to End tests will now be very simple, as functional tests can be chained together to form paths through the system. It is not as easy as that though. Although End to End testing does cover process flows, it should not just be the front end that is verified. All application layers and data flow across all interfaces should be covered during End to End scenarios mimicking how a user will use the application. So, although chaining tests together using existing methods is a good start, the way that the system interacts with external interfaces can be difficult and will make automation very complex. For example, if a payment is made through the UI, the user may be redirected to an external bank, so the automated check will need to be able to find an external page, log in to the bank, verify the user, check that the API is sending and receiving the correct data, return back to the application, and verify that data has persisted correctly. As well as this, the application needs to be set up in a like for like environment that allows simulated communication with databases, Services, networks and other interfaces, so performance can also be monitored. Timing issues, operating systems, supported browsers and mobile devices are also an extra overhead to maintain.

Is it worth it?
To create an all singing all dancing End to End automation suite will take effort. What value there is in producing it should be considered early on, and questions should be asked if this should be a sanity check, chained system tests, or tests across all application layers. If the release is at the start of a long-term development project, with many years of maintenance and future development work, then putting in the groundwork will pay dividends when the project is handed over to different teams, or considerations for succession planning. In a smaller project that is enhancing a COTS product, creating a lightweight automated sanity check to be run during each developer check-in may be acceptable, so that critical functionality can be regression tested on each build. It is a form of End to End testing that will require an on-going task to maintain and update as the software is enhanced but is much more manageable and achievable and will still give a good level of confidence.

End to End testing is a very important part of delivery, but whether tests are run manually or automatically, the skill is to be able to create multiple End to End business scenarios that touch the critical parts of the software for different users.The aim is to provide confidence that quality software is being produced – that does what the customer wants using easy to use and intuitive workflows. These business processes can only be thought about at the beginning of the project while there is time to influence those decisions, as well as deciding whether to run End to End scenarios manually or programmatically.