Key Differences Between Performance Engineering and Performance Testing

High user expectations and the risk of putting poorly performing code live makes performance engineering more important than it has even been. Fortunately, incremental change makes it easier to engineer system performance and build a workable process. For those doing DevOps, having administrators and technical subject matter experts as part of the delivery team have a much better ecosystem within which to deliver effective performance engineering.

Many IT organisations I see, rely on a few performance testers to develop some scripts and run them after most of the testing has completed and the project is a few weeks away from finishing. The problem with this approach is that there is rarely enough time to diagnose, fix and retest any issues that are found. Don’t forget that a functional regression test is probably necessary if performance bugs are fixed.

This is where Performance Engineering can help.

Before we get into it, let’s clarify the difference between Performance Testing and Performance Engineering.

What is Performance Testing?

Firstly, there are many aspects of Performance Testing that make it different from Functional Testing.

Realistically simulating the behaviour and number of end users, when they do their work, where they are in the world and what they do is one of the most important factors towards successfully testing performance. Also, interpreting the results is critical and unlike functional testing, the results are rarely clear cut, often requiring interpretation. This means that simply knowing how to script using a performance testing tool doesn’t make someone a Performance Tester – and it makes it harder to break into the field – but that’s a topic for another day.

Performance Testing generally requires reporting metrics such as the number of concurrent users supported before degradation occurs, transaction response time and transaction throughput. Network, disc, server and database level metrics such as CPU, io, memory utilisation, swap space are all measured during performance test execution. These help to analyse the bottlenecks affecting system scalability and performance. Application Performance Management (APM) tools can be used by themselves or integrated with performance testing tools to measure and monitor how the system performs throughout the technology stack. This includes the application server, database server, web server, client, browser, network load balancers and even ports and firewalls.

What is Performance Engineering?

The engineering activity is the analysis of the performance metrics when a Non-Functional Requirement (NFR) is not met or a bottleneck is identified by the tests.

The performance engineering process will typically attempt to understand the metrics the performance tester reports and use a deep dive analysis on the specific layer that fails to meet the NFR. It is rare that I see a specific role of Performance Engineer. Instead the engineering process is fulfilled by a number of platform/tier/technology specific Subject Matter Experts (SMEs). SMEs such as DBA, Network Administrator and Technical Architect may be involved depending on the bottleneck reported. They can perform a detailed analysis of the performance issue and provide a tuning recommendation to correct it.

However, more importantly Performance Engineering is also the application of systemic practices, activities, and techniques during every phase of the software delivery life-cycle process (SDLC) to ensure the product meets performance requirements. By focusing on design, architecture, and good development practice, Performance Engineering aims to build performant systems from the bottom up. A Ferrari is not fast because it is tested and has performance improvements made at the end of the production line. A Ferrrari is fast because right from the design stage, every component is engineered and built to be fast.

Incorporating Performance Engineering Into your Delivery Lifecycle

Performance Engineering is a vital part of software delivery, yet many IT organisations find Performance Engineering to be an expensive and challenging thing do well. Despite major performance failures that have hit the headlines and been an embarrassing mess, Performance Engineering has failed to get the adoption and budget it deserves in many companies.

Here are things to keep in mind when incorporating the performance engineering process into your model:

1. Develop a Strategy

Developing a Performance Engineering strategy is a critical part of the process and you need to figure out how to incorporate it into your organisation and delivery model. Identify the SMEs you will need to call upon and the touchpoints they will have in your development lifecycle. You need to understand what are the quality gates and how will they be governed? Always remember that it all starts with the requirements. If your product owner knows what performance they want from the system, then it will be a lot easier to engineer the system to meet that requirement. A statement such as ‘it needs to be fast’ isn’t going to help anyone. You’ll be amazed at how often I see it though!

2. Budget

It’s going to take some money to develop high-end performance engineering practices. As you are developing your performance roadmap, you may need to go through a number of budget cycles in order to get all the tools and infrastructure necessary. Stay strong and positive. Use the long list of public failures to convince the budget holders of the importance of Performance Engineering.

3. Identify Key Business Workflows

Unless you have the right tool (for more information on helpful tools, please get in touch) and can re-use your automated functional tests for Performance Testing, creating hundreds of performance tests will not be cost effective and will take a lot of time. Instead you will need to focus on building the ones that are most critical to the business, have the highest throughput, number of transactions or volume of users.

4. Find the Baseline and Test Regularly

The next step is to benchmark the performance baseline with a set of performance tests that will be used time and time again. Put together a history of your runs to check for trends in system performance. Ideally this will be done with every release and every integration. If the trend analysis can be automated as part of a CI/CD process, then all the better.

5. Tools and Hardware

You will need the proper APM, diagnostic and load testing tools for Performance Engineering. It’s important that you identify the things you’ll need to properly run tests and diagnose bottlenecks. Production-like environments are generally expensive. Ideally, you’ll have one for your Performance Testing however, if you are testing regularly with every deployment, then the trend analysis will still point to a bottleneck that the engineers can be let loose on.

6. Have a Data Strategy

As you will be testing regularly, you will need to be able to create test data quickly and efficiently. It’s important that the data you have is a similar size to the production environment - query plans will be different if you are not using a representative data set. Don’t fall foul of GDPR and also remember to backup your test data.

It will take some work to properly incorporate performance engineering into you SDLC, but the rewards are worth it for any organisation that is willing to embrace it.