Defining Clear Objectives and Goals for Load Testing

A question I get asked a lot is ‘What are the entry criteria for load testing?’ Before I go on, I’d like to clarify what I mean by load testing. Sometimes I see the term confused with; interchanged with or lumped in with performance testing or performance testing tool. When I refer to types of performance testing, I mean end to end response time, including rendering, of a single user. By load testing and stress testing I mean, how does the system including network, servers, machines and database cope when multiple users are using it at the same time?

The following is a list of entry criteria that you might find useful:

1. Non-Functional Requirements

The rule book says that NFRs should be signed off before non-functional (including load) testing can start. If they exist, then this is all that needs to be said and you can merrily plan your test case and execute your tests to verify they can be met by the system under test.

In reality, it's a rare and happy surprise if there is a set of non-functional requirements that can be referenced for load tests and can be used to determine if the tests have passed or failed. Before scripting starts, you need to know what you are trying to prove as this affects your transactions and user experience.

Before execution starts, you need to have determined your load pattern. The NFRs will explicitly point to what these should be so without them as a tester, you need to be sure that what you are testing is relevant and appropriate. Without NFRs, the minimum entry criteria into load testing should therefore be that an appropriate stakeholder has agreed in writing that what you are testing and trying to prove (throughput response time, number of users and concurrent users etc.) is the right thing.

2. Data

Data is fundamental to a successful load test. The database of the system under test needs to have enough data in it to ensure queries executed during the tests force the database query optimiser to use the same explain plan as it would in production. Failing to do this might mean that a query in test uses an index whereas in production a full table scan is executed – resulting in a big slow down.

The opposite can also be true i.e. because there is not enough data in test, a query quickly pulls back the majority of data in a table without doing an index scan. This can sometimes be quicker than traversing an index and then hitting a table and can give misleading test results.

2.1. Seed Data

The data that is needed for an application to properly function must be loaded. Hopefully, the functional tests will have already identified ALL of the seed data that is required but often this won’t be the case. This should include master data, reference data (the data that defines a set of permissible values used by other data fields such as product codes or Master data) and metadata (descriptive, structural and administrative).

Each system will have its own seed data requirements but obvious examples are account data, product data, product classifications and postcodes.

2.2. Transactional (or Dynamic) Data

The functional tests probably won’t have a full complement of transactional data. They will create and affect the transactional data relevant to specific tests. However, it will typically be low volume. In order to have high volumes of transactional data, the load tests will need to either import Production data if it is available or create enough data to emulate Production through scripts. This could be a task for the DBA or the Load Tester. If the tests need to prove adequate system performance under a future prediction of load (for example after 1, 2 or 5 years of growth), then the transactional data will need to be artificially created. My preference is to always create transactional data through the front end of the application performs under test as it is generally easier to maintain referential integrity.

2.3. Test Data

In order to execute the required number of transactions, the tests will need input data. Ideally, the data can be created by the test scripts as they execute but sometimes the test data will need to be pre-prepared. If so, this should be an entry criterion.

2.4. Gather Database Statistics

After any data load, it is important to ensure that the database statistics are updated before testing begins. This will ensure the database query optimiser uses the most efficient execution plan.

It is good practice to drop indexes before a large data load and then rebuild them. This will automatically collect statistics. With an Oracle database, if you don’t drop the indexes, you can force the gathering of statistics, or wait for the overnight scheduler to do if for you. The following link explains more. 

Statistics gathering can be forced with the following syntax for SQL Server or SQL Azure by following this link. 

3. Stable Code Base

Traditionally, load testing has been conducted after systems testing (or functional acceptance testing) is complete and depending on the tests in scope, after systems integration testing (SIT) and before user acceptance testing (UAT).

3.1. Application Load & Code

The fragility of network level load testing scripts from tools such as LoadRunner, Neoload and JMeter has created the dependency of the codebase of the system under test to be complete and stable (dev complete). Small changes in the code easily break load testing scripts and so load testing has traditionally been one of the final stages of product delivery.

What about Agile?

Even on agile delivery, the scripting effort and fragility of load test scripts means that load testing has never been an activity that is easily brought into a sprint.

A tool that I particularly like is TurboSelenium, which changes this dependency and means that load testing can now be brought in sprint. This means that the entry criteria are slightly different and load testing can be executed at that same time that functional automated tests can be executed. This is a massive shift left of load testing and makes load testing much more effective for agile projects.

3.2. Batch Jobs

If batch jobs create a background load or need to execute within a specific window, then these need be developed and functionally tested so that they can be scheduled to execute during the test run.

4. Test Rig

Regardless of the load testing tool selected to automate your load tests, the test rig needs to be in place and tested before execution begins. This includes:

4.1. Licenses

Licenses need to be in place with the necessary number of virtual users.

4.2. Load Injectors

The load injectors (aka load agents or load generators) need to be in place and the controller needs to be set up, communicating with the injectors.

4.3. Network Emulator

If you are using a network emulation tool such as iTrinergy, then it needs to be in place with the relevant parameters for latency, bandwidth and packet loss measured and set up in the tool.

4.4. IP Whitelisting

If using load generators outside of your corporate firewall, you will probably need to get the IP addresses of the servers whitelisted to stop requests being bounced. As far as the firewall is concerned, a load test looks pretty similar to a Denial of Service attack and is likely to stop your virtual users from seeing the system under test.

5. Test Environment

The test environment needs to be available and prepped ready for testing. Ideally, you will have a production-like test environment. Thankfully, this is easier to achieve and more common with cloud based hosting platforms such as Azure and AWS.

6. Test Scripts

Test scripts need to be complete. If you have scripted in a different environment to the one you are executing in, any refactoring needs to be complete before execution can start. This includes changing URLs and re-correlating your scripts. If using a tool like TurboSelenium, then this won’t be necessary and doesn’t need to be am entry criteria.

7. Monitors and Logging

Appropriate server resource monitors need to be in place for the system under test. Ideally, these will be hooked into the controller so that your test results can correlate transaction times with machine resources.

Similarly, ensure that production-like logging is switched on for your tests. This often gets overlooked in the non-production environments. Logging can impact results but also be very useful if debugging is required.

8. Support

To make sure there is a smooth set of test runs, it’s important that there is good support from various other teams. An entrance criterion should be that agreements are in place with these teams/people and their managers that they can spare time during test execution. Examples of people that might be needed to support the testing effort are server administrators, DBAs, hosting provider and business and functional subject matter experts (SMEs).

Ultimately, your entry criteria should tie back to your dependencies and the things that you need to have in place to ensure a smooth and uneventful test run. This list won’t cover all eventualities but it will give you most of what you need. Good luck!

White Paper

In the search for delivering high-performing applications, choosing the right load testing tool is vital. The heart of this white paper lies in identifying nine essential features that you should prioritise when selecting a load testing tool. Read more here.