Synopsis

This will be a comprehensive guide to helping you minimise the frequency of maintenance when it comes to your automated test scripts. This should also be useful for those familiar with writing automated testing code.

Change management affects how we maintain our tests. In the context of manual testing, it's updating test cases, test steps, and expected results (also test suites). However, for automated tests, this is in the form of updating code or updating data.

This repeats every cycle and varies in size; the number of changes needed therefore should be both minimised and clear. I must add that I’m only a junior SDET and that much of this knowledge is work-in-progress for me; these are my experiences and opinions.

Test Automation Strategies

How you consider the automation strategy is important. This is before you even write any automation code in the first place. In general, it's good to have a test basis with which test cases are designed against first, and that basis can be created by you when you read requirements, or discussed and agreed upon by the product owner, software documentation, or risk assessments. The reason this will help you reduce brittleness is because understanding requirements means you know what your tests should be looking for, cutting the time needed to look or ask questions with developers or BAs. Secondly, it’s important to know what the use of the automated tests is in your project. This allows you to not spend time writing unnecessary tests that are obsolete or require a lot of effort with low return value.

Test Suite

Next is your test suite, which are folders you group your test cases in. Test cases typically sit inside “spec.ts” files, and these can contain multiple test cases. It’s important how you organise these files, as test cases involving the same test data can be grouped together. This helps if data needs to be updated, as the outdated data would break all those test cases anyway, and so putting them in one file allows you to update and review all test cases together.

Test Report

One thing to consider that isn’t likely to be done before testing, but it can be good to have a visualisation of what your test report might look like. For example, if you were to run the tests daily, i.e., by pipeline, you may want to box steps together to improve readability of failed test steps and allow you to quickly recognise if it’s a defect or a change needed for the test. This can be helpful, as non-test-step methods that fail can be recognised quickly without misguiding you when it comes to the software.

The overall point is that pre-work helps long-term, if it’s organising your spec files, keeping track of your list of test cases, or your test report from within your scheduled runs.

Test Data

You don’t want failures to occur due to changing test data; therefore, you should plan before test execution what data to use, when and where it’s updated, and by whom it is made. Also, during the test itself, what data gets changed and needs to be cleared or reset.

Test data can be the cause of a lot of anomalies. While sometimes it’s relevant if a test fails on test data, the majority of times the failure is due to outdated test data and not a bug in the environment. There are a variety of ways to keep your test data in your tests: 

Within the Source Code

This is the most basic way, which is to write the literal data into defined properties at the start of the test and have it executed on that data. The main advantage is that it’s the quickest way to get a test running on an environment to do testing, but it can be ‘a bit hacky’ and have possibly bad consequences.

Test data kept inside source code means that anyone with access to the repository can see potentially sensitive data. This means you can’t keep data like passwords or usernames in hard code; things like anonymised data or generic data, e.g., product names, can be kept like this.

Another negative is that the amount of data can clutter tests or even (minorly) impede test execution speed with more variables stored in memory. One last negative is that it’s generally a bit harder to scope data efficiently; for example, if this data wasn’t used by just one test but by multiple files, you’d have to reorganise the data into some higher scope, like fixtures.

This overall can lead to potential mistakes or duplicate data being stored, which just leads to more time spent maintaining code. However, having it in source code does have the advantage of test clarity, as you can see the data you use clearly. The best advice I can give is to only keep data that doesn’t change and isn’t test-relevant in source code for the sake of smoother test execution, and then consider moving other data higher in scope for quicker and easier maintenance.

Data Fixtures

A good choice is to use a data fixture such as this:

This is very usable as the data is defined in one location and then can be used by all subsequent fixtures, and therefore all tests. Then, if the data needs to be changed, you only have to change it in this location.

Similarly, if the test does not require this data, you only have to remove it from its parameter in the test file, thereby separating the data setup and the test itself from each other. You can also create default data with this as a base means for your general tests, which is much less work than storing generic data outside the project.

Interfaces

If you have data fixtures and you would like to create default data, it would be wise to separate this into a different file and use an interface. This way you can create a model for larger datasets, especially data that goes together.

First, create a general interface that maps all the data points of whichever data object you want, e.g., if you are modelling a user’s profile, you could create an interface consisting of firstName, lastName, age, and job. Then you can export a property of that interface type with initialised values and set the data fixture to that property.

Separate Location

It’s more than likely you will have a mix of two: first is some data in the source code, like for defaults; second is data outside stored securely, which is mainly for sensitive data used in the test, or credentials, like passwords. These then need to be secreted in some way. There are a couple of options:

Environment Variables

The first is to create environment variables, which are applicable to all programming languages that I know of. It involves having a.env file and storing data in these variables. Now dotenv will encounter the same problem as source code if you wish to commit this to a remote repository. You can therefore choose to store it locally, but this elicits more issues, like how you would update the list if a new feature or variable needed more data to be added.

File System

You can have another file, like a.JSON file or an Excel sheet. The advantage here is that it will be easier to add more data into this file, making it a little more structured. Also, it’s separately loaded, which can be an advantage by not leaving traces of data so easily. Sometimes these files can be a little frustrating, however, as more work is needed to read and load data from them into the project.

Remote/Secret Vaults

The other alternative is using a remote location. This is normally done with secrets and PAT tokens. You can do this using Github Actions or with Azure Key Vault if you have access to Azure Dev Ops. This will improve maintenance as data can be stored in one location securely; you can then update data without having to commit or make changes to code.

The downside is that values are stored by key-value and can’t be structured or grouped like JSON objects. This means work needs to be done to prepare the data before giving it to the test. Therefore, it’s more viable to keep secrets, IPs, and passwords stored like this and to have other data stored elsewhere.

The best way to minimise changes will be to use a mix of all. Storing secrets remotely in a vault, having files separate with necessary input data, and storing defaults and other constants within data fixtures or the test itself.

Maintenance Long Term

We want to write tests for the long term so that we don’t constantly have to go back and update these tests. However, refactoring can take a very long time; finding those moments to work on it could be challenging if sprints are short or workload is hard to manage.

The best thing to do is find a clear timeslot for allowing you to maintain and update code, where it’s right after all planning and user stories have been finalised; that way, each sprint you can consistently compare changes and update when needed.

An important thing to consider here is what programming language or test tool you are using. Some tools have advantages, like record and playback tools, where you can run them again every time there’s a change and use the newly generated script after. Nevertheless. this can backfire as the constant use of the tool every sprint will grow exponentially, the more tests that need to be remade like this.

How to Maintain Record & Play Scripts

For Record & Play, running the tool once manually will provide you the script. Next is to run these tests every sprint to check if updates are needed. It’s quite difficult to deal with constant code changes using Record & Play; therefore, I recommend trying to keep it to regression testing more than anything, as regression testing will face fewer changes and will track unintended changes better.

However, if you are facing changes, these changes could involve just some minor locator changes to the script, but worse case, you might have to run it a second time, which would be fine if the number of tests is low, but if you have a larger test suite, then it can become too taxing.

Robust Locator Design

Hence, and this applies to both Record & Play and programming languages, picking good locators is the most important key to long-term test cases. The context of good locator strategies depends on your software, and so it can be very difficult to pick one strategy. There are a few standard approaches that you can use.

• Identifier tags. HTML has an ID tag, which is very easily located using selectors by the #. Normally these don’t share multiple instances, but they can if they’re duplicate elements on the screen. In this situation you can still find by ID, but you will need to incorporate either a filtering method or find a specific trait in the element like classes or attributes to combine with the ID selector.
• Data Test-ids. The one that is full-proof but is a slight cop-out. These are ones where the dev adds and Id tags for the sole purpose of letting your automated test find them easily. It can be good to make tests perform more easily, but data test- IDs can sometimes be misleading as they don’t describe the page well and are not part of the software architecture, therefore potentially obscuring the actual test. In other words, you could be testing a false object and not the actual object itself.
• Attributes and classes. These tend to be used by JS frameworks to build elements on the screen. It can be a good way to navigate through the HTML, but the locators can often be hard to understand. This invites potential time-consuming effort in the long term.

Good Code Practice

Now the following are guidelines for good coding practice that you should follow. These all help to reduce maintenance, minimise bugs in your own code, improve clarity, and build for the long term.

DRY

Don’t repeat yourself; it’s about making sure all facets of the code are not being duplicated unnecessarily. This is done by proper object modelling, inheritance, and making sure methods are passed down the tree. Properties and values should only occur once at the relative base class. This also includes moving any repeated values, methods, and literals into separate files, then using proper importing to make sure you don’t repeat similar lines of code between tests. This is beneficial as changes won’t be needed in multiple locations that you must find and change, thereby ensuring changes are consistent for all tests.

WET

Write everything twice, the antithesis of DRY. This may look counterintuitive, but it means when DRY is actually getting in the way of clarity. Although having one occurrence is great for making efficient changes, there are some contexts where the DRY approach causes the need for detours around code like getter methods, which can be ignored if you just repeat yourself instead.

POM

AKA Page object model. If you remember that having good locators is the single most important key to a good maintainable test code, then POMs are the 2nd most important for the long term. This is critical to having a successfully maintained test suite, as dev changes are very likely going to affect things page-to-page. Then having all the locators that will be affected in one location helps to take care of broken codes fast. It also helps build a clear semantic model of the application in the form of pages, which helps breakdown the intricacies of the application for you as a tester.

Cascading Fixtures

A way to setup singular test cases with individual needs and a major feature inside of Playwright. The power to give the same fixture to multiple test cases and even to other fixtures is powerful. By doing this, changes needed for dozens of test cases can all be fixed by changing one single line of code, as all test cases are linked to the same fixture. Fixtures in general are great to easily build new test cases, as so much pre- and post-work needed can be done ahead of time.

Package Updates

One last thing to remember is that your packages might incur sudden test failures due to updates. It’s important to have a consistent routine of reading important changes before updating, then upgrading the packages either automatically, i.e., using Dependabot or Github actions, or manually using npm, Nuget, or Wget.

Are they Actual Test Failures?

Don’t forget to consider that the test failing isn’t necessarily a fault of the test itself. A previous passing test that fails means something in the environment has changed. First is to assess what the reason is for failing, then judging if it’s a bug in the test or a bug in the environment. Then it needs to be checked for severity and prioritised a time to fix it or to be ignored.

Common changes that can cause tests to fail (excluding bugs) include changes to the product's base (rather than new features) such as different locators or test-ids, outdated connection strings and credentials, or refreshed user data.

Why are we Facing Brittleness in the First Place?

This issue typically gets brought up early in the development of an automation suite in a project. Often, it’s not because of any limitations that are raising these questions, but instead obstacles that are expected to show in any early project. Therefore, the important action to take is to make sure expectations are clear about what the automated tests can do and won’t do. The best use of automated tests at the start is cutting time spent by us on tests; therefore, start with those tests to save immediate time first.

Reducing automated test brittleness involves strategies that minimise maintenance and enhance your test automation framework. A solid testing strategy lets development teams create resilient automated test cases, saving time and reducing reliance on manual testing.

Choosing the right test automation tools that integrate with your CI/CD pipeline ensures efficient test execution and quick, reliable results. Combining unit tests for code-level validation with UI tests for user scenarios balances test coverage across software development.

Leveraging open-source tools helps reduce costs, while modular test automation scripts ensure adaptability to changes, making your automated tests more durable and reducing maintenance efforts.