Selecting the Right Performance Testing Technique

There are many different types of performance test – sometimes referred to as performance testing techniques. It’s not always easy to know which you need so this article aims to give some guidance on the performance testing technique you might want to consider for your system.

Load Testing

If you need to confirm the system provides fast response times and can complete concurrent transactions within a certain period of time, load testing is for you. This is the type of testing most people think of when they consider performance testing.

This type of test needs to identify the most important transactions from a business risk perspective i.e. it would be bad for the business if they are slow; those with the highest volume of transactions i.e. Amazon, has thousands more searches per second than it has requests for invoices and those which are technically complex. For a system that has a user interface, the test will need to include the user actions such as search, transaction creation – for example the checkout of an e-commerce product, data updates, field tabbing, changing page, loading of images and data. It will need to simulate the peak number of concurrent users, all doing the typical things they would do in a real-world scenario. For a back-end, transactional system such as a bank clearing, this can be thousands of transactions per second, being triggered by whatever mechanism or interface does this.

• Logins
• Account creations
• Searches
• Payment details updates
• Checkouts

The objective is to identify if any actions and transactions perform slowly and help pinpoint the bottleneck and its root cause. It might be a resource constraint, such as CPU, but it could equally be a badly formed query, poorly indexed database, poor memory allocation in the business logic, a loop in the code or any other myriad of possibilities.

Stress Testing

If you want to find the breaking point of your system, then you’ll need a stress test. It typically uses the same scripts and transactional mix as the load test, but the number of virtual users is gradually increased (ramped up) until the system breaks. The breaking point is defined by the non-functional requirements. It can be something like when the CPU utilisation of the server reaches an upper limit, when the response time drops below an acceptable level or the transactional throughput drops below a certain threshold.

Often, for new systems, how the product scales under load, it’s capacity (and therefore longevity as uptake and data grows over time) will be unknown. A stress test can therefore provide important information that can be used to plan and future-proof a system.

Scalability Testing

Continuing the theme of trying to understand a systems ability to scale, scalability testing takes this a step further. If you want to figure out the most cost-effective way to add capacity to the system and determine whether horizontal or vertical scaling will give you better performance gains, then this is the test you need to run. (Horizontal scaling means adding more servers/machines whereas vertical scaling means adding more CPU and/or RAM to an existing server/machine. I remember it by thinking of an old server room before we had blades and racks and needing to add a new server by putting it next to the existing one. But then, I’m old enough to remember old server rooms).

Once you’ve reached capacity on the infrastructure and it’s creating a bottleneck, you can try one or both methods of scaling to remove the bottleneck and improve performance. Scalability testing is necessary to determine the most cost-effective way to set up your infrastructure, even in a cloud environment as it can dramatically reduce hosting costs and maintenance overheads if you get it right.

Soak Testing (sometimes called Endurance Testing)

Soak testing involves testing a system with a typical production load, much like the load test but the test is left running for a much longer period of time. It should be done once the load test is complete and all the non-functional requirements it tests have been met.

The soak test will typically use the same transactional mix as the load test. The number of concurrent users and number of transactions might be dropped down from the peak (but not necessarily) and the test left running. Ideally, the load would emulate a typical business day and would undulate, however, it’s not always easy to do (often because the typical load of a new system is unknown). For how long the test is left running depends on how long you’ve got but if there is degradation in performance, and the load is high enough 24 – 48 hours should be enough.

• Memory use - amount of physical memory available
• Private bytes - number of bytes a process has allocated that can't be shared amongst other processes
• Committed memory - amount of virtual memory used
• Garbage collection – amount of unused memory returned back to the system

Spike Testing

If your system is prone to (or at risk of) large increases in load in a very short period of time, i.e. the load profile has a spike in it, then you need to spike test. The load profile might not necessarily look like a spike as it could rapidly increase and not come down for a while. An example of this would be National Rail enquiries during a storm at 6pm when train cancelations and delays mean the number of people checking train status and planning their journeys massively increases.

The load testing tool can be configured to generate this spike in load. The transaction types and scenarios obviously need to reflect those that will cause the spike in production. If your cloud servers have elastic scaling capability, the spike test will determine whether it reacts quickly enough and whether the bottleneck is mitigated by the infrastructure that scales or is somewhere else. An example of this would be where the servers in the rack are set to scale but the network interface into the rack maxes out. It’s this kind of thing that doesn’t always get considered and will only get flushed out by a performance test.

Conclusion

It’s important to decide on your test objectives before deciding what performance test techniques to use. Typically, a combination will be needed and you might need to prioritise which tests you do based on the perceived risks the systems face. There’s a good argument to say that all, or at lease most, of the performance testing types should be carried out but unfortunately budget and time won’t always be available to you.