What if Entry Criteria for Load and Performance Testing are not met?

A little while ago I wrote a blog called ‘What are the entry criteria for Load Testing?’.

For this blog, I’d like to focus on what to do if the entrance criterion of “Non-functional requirements (NFRs) are defined and signed off” has not been met. Also, what can be done if they have been defined but are simply not good enough to properly test in a meaningful way?

Defining Non-functional Requirements (NFRs)

In an ideal world the non-functional requirements would be signed off before load testing and other non-functional testing starts. However, in my experience, it is quite rare that they exist or are good enough to be usable.

So, what can be done about it? Waiting is not an option as the knowledge and skills required to define good and detailed non-functional requirements rarely exist outside the non-functional testing team. As such, you’ll be waiting a long time. I realise that this is a generalisation but typically BAs, Product Owners, the business nor the users will be able to think of all the scenarios that need to be considered. Neither will they go to a level of detail that is low enough, even with the best intentions, that worthwhile tests can be defined. The Operations or DevOps team might have this capability and also a practical view that can be helpful but, in my experience, there is sometimes a level of detail that is still missing and some scenarios that won’t be considered.

The important thing to remember is that, as reasonable human beings, we shouldn’t really expect someone to be able to know the business, product and technical detail well enough to define functional and non-functional requirements well. So, I don’t get upset if the NFRs are not well defined and instead see it as an opportunity to demonstrate the value that the test team can add to the overall project.

The most practical and cooperative thing that can be done is the testers get involved with the requirements definition. Let’s be clear here - I am not advocating that the test team defines the requirements, this must be avoided. However, by asking the right questions the testers can help the appropriate stakeholders formulate the right answers. These answers then become the non-functional requirements. This is essentially what a good business analyst would do when defining functional requirements. However, I think that unlike functional requirements, where a BA would possibly ask an open question and interpret the response and then define a requirement or PBI, for the non-functional requirements we need to give some guidance and options. Otherwise responses such as “the system needs to be fast” or “it needs to be no slower than the current system” will be given and the tester will be no closer to having a testable requirement. Instead, I advocate using very specific multiple-choice questions and having one for each NFR that you’d typically expect.

For example; 

1. <0.5 seconds
2. < 1 second
3. < 2 seconds
4. < 5 seconds

1. 100
2. 200
3. 500
4. 1000

1. <0.5 seconds
2. < 1 second
3. < 2 seconds
4. < 5 seconds

Note: Search NFRs might be broken down into number of results, complexity of search terms, with and without wild card characters.

As well as the categories we are interested in for the purpose of this blog:
• Performance – for example Render time, response Time, transaction throughput,
• Load and Scalability – performance under load, ability to scale and handle peak and anticipated future load

Other areas might include:
• Capacity
• Availability
• Reliability
• Recoverability
• Maintainability
• Security
• Regulatory
• Environmental
• Usability
• Interoperability/portability
• Data Migration
• Data Integrity

It is unlikely that a single person or group of people will be able to define these requirements. Even within the test team, it is likely to be difficult to find someone who is experienced enough to be able to formulate the questions to be able to drive the requirements definition in all of these areas.

It is also probable that the project will not have the desire/budget/foresight to test all of these areas. In this instance, the question needs to be asked “is there any value in gathering these requirements if there is no intention to test”. This is a difficult question and the answer depends on your standing and role within the team. If people don’t know what they don’t know, then by simply asking the right question, it is possible to raise someone’s consciousness and help them identify there is a gap.

For example;

1. <30 minute
2. < 1 hour
3. < 2 hours
4. < 24 hours

If resiliency has not even been considered by the business or product owner, then this one question will immediately bring it to their attention and a solution can be implemented. Testing has saved the business from a potentially embarrassing and dangerous situation. Yay for testers.

Obviously NFRs are just one of many performance testing entry and exit criteria. For others see 'What are the entry criteria for Load Testing?' and if you are struggling to know what to do if they haven’t been met before you’re expected to start testing then please get in touch here and we can try to work it out.

What next?

nFocus have identified a range of challenges when purchasing load and performance testing services. To learn more about these challenges, including how to solve them, please visit the website info page here.