Entry Criteria for Load Testing
A question I get asked a lot is ‘What are the entry criteria for load testing?’ Before I go on, I’d like to clarify what I mean by load testing. Sometimes I see the term confused with; interchanged with or lumped in with performance testing. When I refer to performance testing, I mean end to end response time, including rendering, of a single user. By load testing I mean, how does the system including network, servers, machines, database cope when multiple users are using it at the same time?
The following is a list of entry criteria that you might find useful:
1. Non-Functional Requirements
The rule book says that NFRs should be signed off before non-functional (including load) testing can start. If they exist, then this is all that needs to be said and you can merrily plan and execute your tests to verify they can be met by the system under test.
In reality, it's a rare and happy surprise if there is a set of non-functional requirements that can be referenced for load tests and can be used to determine if the tests have passed or failed. Before scripting starts, you need to know what you are trying to prove as this affects your transactions and user journeys.
Before execution starts, you need to have determined your load pattern. The NFRs will explicitly point to what these should be so without them as a tester, you need to be sure that what you are testing is relevant and appropriate. Without NFRs, the minimum entry criteria into load testing should therefore be that an appropriate stakeholder has agreed in writing that what you are testing and trying to prove (throughput response time, volume of users etc.) is the right thing.
Data is fundamental to a successful load test. The database of the system under test needs to have enough data in it to ensure queries executed during the tests force the database query optimiser to use the same explain plan as it would in production. Failing to do this might mean that a query in test uses an index whereas in production a full table scan is executed – resulting in a big slow down.
The opposite can also be true i.e. because there is not enough data in test, a query quickly pulls back the majority of data in a table without doing an index scan. This can sometimes be quicker than traversing an index and then hitting a table and can give misleading test results.
2.1. Seed Data
The data that is needed for an application to properly function must be loaded. Hopefully, the functional tests will have already identified ALL of the seed data that is required but often this won’t be the case. This should include master data, reference data (the data that defines a set of permissible values used by other data fields such as product codes or Master data) and metadata (descriptive, structural and administrative).
Each system will have its own seed data requirements but obvious examples are account data, product data, product classifications and postcodes.
2.2. Transactional (or Dynamic) Data
The functional tests probably won’t have a full complement of transactional data. They will create and affect the transactional data relevant to specific tests. However, it will typically be low volume. In order to have high volumes of transactional data, the load tests will need to either import Production data if it is available or create enough data to emulate Production through scripts. This could be a task for the DBA or the Load Tester. If the tests need to prove adequate system performance under a future prediction of load (for example after 1, 2 or 5 years of growth), then the transactional data will need to be artificially created. My preference is to always create transactional data through the front end of the application under test as it is generally easier to maintain referential integrity.
2.3. Test Data
In order to execute the required number of transactions, the tests will need input data. Ideally, the data can be created by the test scripts as they execute but sometimes the test data will need to be pre-prepared. If so, this should be an entry criterion.
2.4. Gather Database Statistics
After any data load, it is important to ensure that the database statistics are updated before testing begins. This will ensure the database query optimiser uses the most efficient execution plan.
It is good practice to drop indexes before a large data load and then rebuild them. This will automatically collect statistics. With an Oracle database, if you don’t drop the indexes, you can force the gathering of statistics, or wait for the overnight scheduler to do if for you. The following link explains more.
Statistics gathering can be forced with the following syntax for SQL Server or SQL Azure by following this link.
3. Stable Code Base
Traditionally, load testing has been conducted after systems testing (or functional acceptance testing) is complete and depending on the tests in scope, after systems integration testing (SIT) and before user acceptance testing (UAT).
3.1. Application Code
The fragility of network level load testing scripts from tools such as LoadRunner, Neoload and JMeter has created the dependency of the codebase of the system under test to be complete and stable (dev complete). Small changes in the code easily break load testing scripts and so load testing has traditionally been one of the final stages of product delivery.
What about Agile?
Even on agile delivery, the scripting effort and fragility of load test scripts means that load testing has never been an activity that is easily brought into a sprint.
A tool that I particularly like is TurboSelenium, which changes this dependency and means that load testing can now be brought in sprint. This means that the entry criteria are slightly different and load testing can be executed at that same time that functional automated tests can be executed. This is a massive shift left of load testing and makes load testing much more effective for agile projects.
3.2. Batch Jobs
If batch jobs create a background load or need to execute within a specific window, then these need be developed and functionally tested so that they can be scheduled to execute during the test run.
4. Test Rig
Regardless of the load testing tool selected to automate your load tests, the test rig needs to be in place and tested before execution begins. This includes:
Licenses need to be in place with the necessary number of virtual users.
4.2. Load Injectors
The load injectors (aka load agents or load generators) need to be in place and the controller needs to be set up, communicating with the injectors.
4.3. Network Emulator
If you are using a network emulation tool such as iTrinergy, then it needs to be in place with the relevant parameters for latency, bandwidth and packet loss measured and set up in the tool.
4.4. IP Whitelisting
If using load generators outside of your corporate firewall, you will probably need to get the IP addresses of the servers whitelisted to stop requests being bounced. As far as the firewall is concerned, a load test looks pretty similar to a Denial of Service attack and is likely to stop your virtual users from seeing the system under test.
5. Test Environment
The test environment needs to be available and prepped ready for testing. Ideally, you will have a production-like test environment. Thankfully, this is easier to achieve and more common with cloud based hosting platforms such as Azure and AWS.
6. Test Scripts
Test scripts need to be complete. If you have scripted in a different environment to the one you are executing in, any refactoring needs to be complete before execution can start. This includes changing URLs and re-correlating your scripts. If using a tool like TurboSelenium, then this won’t be necessary and doesn’t need to be am entry criteria.
7. Monitors and Logging
Appropriate server resource monitors need to be in place for the system under test. Ideally, these will be hooked into the controller so that your test results can correlate transaction times with machine resources.
Similarly, ensure that production-like logging is switched on for your tests. This often gets overlooked in the non-production environments. Logging can impact results but also be very useful if debugging is required.
To make sure there is a smooth set of test runs, it’s important that there is good support from various other teams. An entrance criterion should be that agreements are in place with these teams/people and their managers that they can spare time during test execution. Examples of people that might be needed to support the testing effort are server administrators, DBAs, hosting provider and business and functional subject matter experts (SMEs).
Ultimately, your entry criteria should tie back to your dependencies and the things that you need to have in place to ensure a smooth and uneventful test run. This list won’t cover all eventualities but it will give you most of what you need. Good luck!