Embracing the Gray Areas of Testing

We are pleased to have one of our blog writers Renée Mineart, returning with a new blog item ‘Testing is not always Black and White’. It’s a follow-up blog from last month’s piece Black Box Testing Techniques by Jane Kelly which we would recommend reading before proceeding.

The differences between Black Box and White Box testing are not always (…please forgive the pun!) as black and white as the title might suggest. That is to say, even though they are not the same, it’s not a case of one or the other. 

When doing Black Box testing, you don’t require an in-depth understanding of the underlying code, whereas in White Box testing you do. The analogy is that you can’t see inside a black box but you can with a white box (or in other words, a clear box, open box, glass/transparent box; as it’s also sometimes called).

Black Box testing is largely concerned with inputs and desired outputs but it doesn’t have a requirement to explore all possible paths through the code, such as error handling.

One of the aims of White Box testing is to explore all possible paths through the code, even dead ends, such as error handling (although one could argue that proper error handling shouldn’t result in a dead end). With Black Box testing, you might accidently stumble upon errors, whereas you actively seek them out in White Box testing.

If you don’t know the full potential of the code, as might be the case with Black Box testing, you probably aren’t going to know all the possible variables that code tests for. In the case of White Box testing, you do - or at least should. For example, if I have a field on my form that asks for a person’s first name. In Black Box testing, the tester might enter values such as Bob, Dave, Susan, etc. But if you’re doing a White Box test, you might know that the code is supposed to check for non-standard characters, numbers, symbols and generate an error message on incorrect values, preventing the user from entering something they shouldn’t. When I’m testing a first name field, I always test it with my own first name, Renée to see how it handles the acute accent above my middle ‘e’. It is one of my pet-peeves when a website or application struggles with these letters or when I get an email response spelling my name as ‘Renée’.

If I know what the code is supposed to do, I will know to test the first name field with numbers, special characters and email addresses. I shouldn’t be able to enter [name]@gmail.com, for example without getting a warning or error message of some sort. So, this test isn’t so much testing the successful functionality of a piece of code, end-to-end but rather the built-in error handling of a specific part of the code.

One of the big advantages of Black Box testing is that you don’t need much, if any, information on the inner workings of the code, which is fantastic if you want to bring in consultants to do the testing (such as the fine testers at nFocus). They can usually get started with Black Box testing with very little time required to spin up a project.

White Box testing, in contrast, requires a more in-depth knowledge of the code, at least one person on the testing team should be directly involved with and have an intimate knowledge of the code.

One advantage White Box testing has over Black Box, is that you can start carrying out these tests before the user-friendly interface is completed. These test scripts can be easier to automate and can interrogate more paths through the code. However, the downside to this is that as the application develops and grows, the test scripts will need to progress and expand as well. Therefore, we might want to add onto the planning list, when and what to test to minimise reworking and redesigning test scripts.

Another advantage of White Box testing is that you can test a subsegment of code or the integration between units of code by the entire system or as subsystem. You can easily scale it up or down, target specific areas of the code and focus on specific issues.

That’s not as easy with Black Box testing, as you are essentially testing the user experience from end-to-end and because you can’t see inside the box, it makes it more difficult to design tests that is limited to certain aspects of the code.

Both of these methods of testing fit very nicely into an Agile approach to software development. White Box testing can be done as a part of each Sprint on that specific part of the code. Then, as the application develops further and user interfaces created, Black Box testing can be added to the process to gain that higher level, end-to-end view.

Additionally, Black Box testing should always be done before the application goes to your customer base for User Acceptance Testing (UAT). There’s nothing more annoying than to have your customers point out obvious faults, typos and bugs that you should have picked up with internal testing.

Where, I believe, the line between these two testing methodologies starts to blur, is when you discover an error with Black Box testing and then run a series of White Box tests to isolate and fix the bug. In a situation like this, the two methodologies work hand-in-hand.

There is a real advantage to having a fresh set of eyes look over and test what you’re working on before it goes to your customer. I try and do this a lot with my writing, as a good editor will instantly spot mistakes I hadn’t noticed because I’m too close to the project. This is where not knowing the code inside the box is a real bonus. It gives the tester a fresh view of the application and they will spot things the developers would otherwise never see. You then follow that up with White Box testing to find and fix the bugs.

Black Box and White Box testing are two important steps in software delivery and they each provide a different perspective of the application being developed. When they are used together, they can provide a powerful method of finding bugs and producing quality software.