Understanding the Purpose of Test Requirements

Sometimes, testers are required to think outside the box. Maybe they don’t have test requirements or don’t have sufficient data; or maybe it’s simply a case of unscripted, exploratory testing. Whatever the reason, they are expected to use their testing superpower to interrogate the system and provide some answers…

Let me illustrate the issue with a simple general knowledge quiz (answers at the end of the blog but no peeking!)

First of all, the questions: 

1: What was King George VI’s first name?
2: How long did the ‘100 year war’ last?
3 In which month, do Russians celebrate the October Revolution?
4: What is the acceptable load time for a standard webpage? 

They all look like perfectly acceptable questions and to make matters easier, let me make this a multiple choice quiz!

1: What was King George VI’s first name?
A: William
B: George
C: Albert
D: Charles

2: How long did the ‘100 year war’ last?
A: 93 Years
B: 100 years
C: 101 years
D: 116 years

3: In which month, do Russians celebrate the October Revolution?
A: May
B: September
C: October
D: November

4: What is the acceptable load time for a standard webpage?
A: 2 seconds or less
B: Under 4 seconds
C: 10 seconds if containing rich media
D: 20 seconds

Questions 1, 2 and 3 are simple and by that, I mean all you have to do is google the answer. There is a point of authority you can refer to and a consensus to the answer. But what about question 4?

If you have the answers, the decision to mark a result as a pass or a fail is easy.
However, if you just have the question and no point of reference for the answer, how can you possible mark the answer?

The first two questions, the answer seemed obvious, George and 100 years but with the benefit of a point of reference, it was a simple task to dismiss the obvious for the correct. However, with question 4, it's not so easy.

The obvious answer might be B, under 4 seconds - if you listen to John Mueller, the Senior Webmaster Trends Analyst at Google. See his statement here.

Or the obvious answer might be C under 10 seconds - if you take note of the findings of Backlinko’s analysis of 5.2 million desktop and mobile webpages! Check out the stats here. 

Or… it might be A or D… it all depends!

Simply, having a good question is not much help if you do not have a definitive answer. To be able to mark a test pass or fail requires that you know that definitive answer. Knowing the answer is much more important than knowing the question when it comes to testing.

So, what got me thinking this way?! Well, it was a requirement for some ad hoc website testing that read ‘...website links must be obvious and accessible’. That was it. No other documentation around that requirement, just a link to the website. I had to devise a test that could mark links as ‘pass’ or ‘fail’.

'This link is in blue and has an underline… Pass'

The question for each link is: “‘is this link obvious?” But what is the answer?

On the face of it, the answer seemed obvious: “...links should be underlined and in blue". Industry standard, right?!

'This link is blue but no underline… Fail'

Except that the link itself is blue and all the other text is black… so that’s obvious right? So a pass? 'This link' is in the footer section of the website, which is blue, the link is white and reads “Click Here To Chat Now” but not underlined, is that obvious?

You see the dilemma, unless it has been agreed what ‘obvious’ for each class of link will mean, then the result is open to interpretation. ‘All textual links will be underlined and coloured blue’ is great, until you decided that actually, the small footer links (privacy policy etc.) will be underlined and grey. Knowing the answer for each link is more important than knowing the question: 'is this link obvious?'.

Of course in any real project, you seldom get all the answers upfront and so asking the question is an important way of getting the answer. I am certainly not saying you should always wait until everything is nailed down before you write your tests.

However, I am suggesting that as testers, we should not get too wedded to the question and that if as a result of a test, an answer gets changed, then we should keep in mind that the answer is the important thing that drives us. Get the right answer and you can ask the right question.

=====================================================

The correct ‘pass’ criteria for the questions: 
1: C
2: D
3: D
4: There is no universally accepted pass criteria