SDETs beware, our friendly Copilot X might be eyeing our jobs – really?

To set the context, for nearly two decades I have been working for nFocus as a consultant that specialises in technical testing. A large part of that role is writing automated tests, including UI automation which can be particularly difficult (to do well). Essentially, I program a robot to perform the actions of users to produce an efficient and repeatable method of test execution. I’ve worked on everything from writing automated tests for scientific instruments, mobile test automation from its infancy through to many other systems I’m not at liberty to disclose. 😉

I’ve recently had a chance to use Microsoft’s Copilot X in anger. It now uses GPT-4, which is a significant upgrade from the GPT-3.5 that Copilot previously used. As I delved into the features this new version offers, there was one statement that really caught my attention: “Automate automated testing.” This got me thinking, could the people who program the robot be replaced by …er… a robot?

What Does it Mean?

When you dig into the statement, what they mean is automate the process of writing unit tests, which loosely translates into auto-generated unit tests from analysing the code.

Most of the tests I write typically sit above the unit testing level, such as system or system integration tests, and so require more knowledge about what’s expected out of the tests and the business processes than just understanding the code.

The Limitations of Auto-Generated Tests

So, what does this mean? Well, automatically generating unit tests has some value in a development project, having some unit tests is certainly better than having none. However, the sort of questions I would like to ask of an automated test writer are:

“Can you generate a Page Object Model from this page: https://xxx.xxx.xxx/login?” 

or,

“Can you generate a series of Page Object Models for all the pages that served from https://xxx.xxx.xxx ?”

or even,

“Can you generate all the test cases for the create order process?”

The ‘X’ in Copilot X represents a placeholder for what they (Microsoft) expect it to be capable of in the future, while there are tools out there that can help with this now, for me the ‘X’ does not mean “for automated system tests.” Most importantly for us as automated testers, it cannot find and produce robust locators for elements on a page (...phew!).

The Importance of Test Driven Development (TDD)

This does not mean that it is not currently helpful to automated testers or SDETs. It can speed up writing automated test code for you, but you must know what you want, how to precisely articulate your queries, and how to validate what it produces, as it can’t read minds, and it can get things subtly wrong. For instance, requesting assistance with classes without clarifying whether one refers to object-oriented programming classes or CSS classes can lead to confusion. Generally, a lack of specificity, misuse of terminology, or inquiries about outdated practices will likely result in misleading responses.

Manual Tests: The Foundation of Effective Automation

The primary time savings gained from using ChatGPT for coding test automation lie in its ability to assist in generating code snippets, providing explanations for various programming concepts, and offering solutions to common coding challenges. Additionally, ChatGPT can help by describing best practices, and answering questions related to the test automation frameworks and tools used on the project. It can help both novice and experienced SDETs in streamlining their automation processes or to learn new tools.

AI in Testing: Benefits and Caveats

Tools offering auto generation of unit tests have been around for well over a decade. The idea is not new, and my perspective on their utility hasn’t really changed since I first saw them. Although, it should be noted that while Copilot X, right now, will likely write better unit tests when it can see the unit under test over the ones generated using predefined heuristics, it shares most of the drawbacks of the older generation tools.

For one, it must assume that the code it’s analysing is already correct. Two, and it’s the big one for me, writing unit tests last violates the principle of TDD (Test Driven Development). The most significant advantage of embracing TDD principles, in my view, lies in the mental shift it compels us to undertake.

Security and Privacy Concerns with AI Tools

As SDETs, we’re often required to draft our own manual tests first before we can start automating them. We write these tests with the basic question always looming in our heads: ‘’How will I automate this?” That simple question makes us write a greater number of simpler, shorter, more to the point tests that are easier to automate and report on.

Conversely, expansive manual tests in the theme of ‘while I’m here I might as well check XYZ too’ may offer fleeting time savings during execution but typically demand extensive domain knowledge to execute effectively. When such a big test fails, it’s hard to report back to stakeholders on what part of the system is broken and how much of the functionality is really affected.

When developers adopt the ‘unit test first’ mentality, something very similar happens. By thinking about how to facilitate unit testing of their code first, they craft simpler smaller modules that fulfil their requirements. This leads to better-designed, more maintainable code. It helps identify issues early in the development process. This focus on testable code permeates throughout the system, often making our jobs as SDETs easier in the process.

The Future of SDETs

Using any AI assistance with a project will always have to be carefully considered. There might be reasons why using it would be risky or outright prohibited. ChatGPT retains users’ chat logs, which could potentially be leveraged for further model training. Providing project-specific information in queries may be essential for obtaining quality responses, yet doing so could land us in hot water. Local regulations or contractual constraints might forbid the storage of any project data abroad or outside designated secure storage.

Finally, there may be concerns regarding the ownership of the content generated by ChatGPT. It's imperative to understand the terms of service and licensing agreements associated with the use of the AI model to determine who holds the rights to the generated code or text. Luckily, at least that last point is less of an issue with Copilot X.

Conclusion 

As SDETs, our roles are increasingly intertwined with advanced tools like Microsoft’s Copilot X, powered by GPT-4. While the potential to automate unit test creation can streamline testing processes, we must recognise the limitations of auto-generated tests, especially for system and integration testing. Effective Test Driven Development (TDD) emphasises designing testable code from the outset, which is crucial for maintaining quality.

Copilot X can assist in generating test scripts and enhancing regression testing, but it cannot replace the deep domain knowledge and critical thinking SDETs bring to exploratory and performance testing. By effectively leveraging automation testing tools alongside AI, we can boost productivity, improve test results, and focus on delivering value to our customers. Ultimately, AI should complement our expertise, not compete with it.

In short, Copilot X cannot take our jobs, but as our “AI pair automated tester,” it can make us more productive, help us write more robust automated tests and allow us more time to add value to our customers… albite with some caveats attached.