Don't Overlook It

I’m not sure why but automated security testing is, without doubt, the poor relation to all other types of automated testing. The software testing industry has been trying super hard to automate functional testing for well over 20 years – and the results have been patchy at best. I see all sorts of attempts but it’s rarely questioned as a sensible aspiration, even in situations when the return on investment (ROI) is nowhere to be seen. We relish the thought of automating unit tests and even have whole conferences dedicated to test driven development. 

Why is it then that whenever I recommend automating security testing to my clients, it feels like I really have to sell the idea. More often than not, they choose to do it manually. And I’m always surprised when they do.

The interesting irony, is that even with manual security testing, the techniques used by the tester rely heavily on scripts and automation. An obvious example would be simulating a brute force attack. A tester (or hacker) won’t attempt to guess the username and password needed to access a system themselves. The tester will have a script that tries many combinations of characters until the correct one is found. So what’s so special about kicking off this script manually, rather than automating its initiation?

One of the reasons why automated security testing may not be considered in such high regard as manual security testing is the lack of capability to exploit vulnerabilities in many of the automation tools. A good manual penetration test will not only identify the security weaknesses of the software and infrastructure under test but the tester will also attempt to exploit them to reach the underlying assets and data. A penetration test therefore not only identifies the weaknesses but uses the identified vulnerabilities to simulate attacks that a skilled and determined attacker could carry out on the system under test once inside.

While this is thorough, I’m not entirely convinced it’s necessary. I recently had a security assessment carried out on my house by a local locksmith. He looked at the doors and windows, the quality of the locks and any potential access points. He identified the weaknesses and then we figured out how to fix them. We didn’t feel it necessary to kick in the doors and ransack my house. I know it’s not a perfect analogy but it’s the same principle.

Once a security scan has identified the security vulnerabilities of a web application and the results have been passed to the development team, they can be fixed regardless of the implication of said vulnerabilities. Besides, with the right tool, the potential risk and impact of exploitation can be determined from the specific vulnerability and therefore prioritised for fixing. It’s also worth noting that there are automation tools out there that can attempt to exploit an identified security flaw. The benefit of this is that it reduces the number of false positives, which is another drawback of many vulnerability scanning tools. Lots of false positives can bury the real vulnerabilities. The developers then wade through a lot of data and spend valuable time trying to fix things that don’t need fixing. We must therefore avoid this.

One of the disadvantages of manual penetration testing is the time it takes and the high cost of skilled testers spending this time. This limits its viability and scalability. Because of this, I see many organisations choose to take the risk of not security testing anywhere near regularly enough. We know that any time code is touched, bugs can be introduced and there’s a regression risk. This holds true for security vulnerabilities too. So, in an Agile world, there is a huge security risk when organisations choose to test just once a year. It is literally criminal.

Even if you don’t think automated security testing is as good as manual security testing, it is certainly better than nothing. If a product has several releases a year and only one manual pen test, this gap can be filled easily with a good automated security testing tool at a relatively low cost. Even if the tool is just a vulnerability scanner it’s better to run this on every release than to do just one manual test a year. A combination of a manual penetration test once a year and automated security testing with a sophisticated tool for the releases in between, can be a very powerful and cost-effective solution.