<img alt="" src="https://secure.wauk1care.com/164394.png" style="display:none;">

Test Requirements - When the Answer is more important than the Question

Posted by Tony Simms on 12/11/2021

Test Requirements - When the Answer is more important than the Question

Sometimes, testers are required to think outside the box. Maybe they don’t have test requirements or don’t have sufficient data; or maybe it’s simply a case of unscripted, exploratory testing. Whatever the reason, they are expected to use their testing superpower to interrogate the system and provide some answers…

nFocus LinkedIn (14)

It's that question of what the right answer is, rather than what is the right question that has been exercising my brain recently.

Let me illustrate the issue with a simple general knowledge quiz (answers at the end of the blog but no peeking!)

First of all, the questions: 

1: What was King George VI’s first name?
2: How long did the ‘100 year war’ last?
3 In which month, do Russians celebrate the October Revolution?
4: What is the acceptable load time for a standard webpage? 

They all look like perfectly acceptable questions and to make matters easier, let me make this a multiple choice quiz!

1: What was King George VI’s first name?
A: William
B: George
C: Albert
D: Charles

2: How long did the ‘100 year war’ last?
A: 93 Years
B: 100 years
C: 101 years
D: 116 years

3: In which month, do Russians celebrate the October Revolution?
A: May
B: September
C: October
D: November

4: What is the acceptable load time for a standard webpage?
A: 2 seconds or less
B: Under 4 seconds
C: 10 seconds if containing rich media
D: 20 seconds

Questions 1, 2 and 3 are simple and by that, I mean all you have to do is google the answer. There is a point of authority you can refer to and a consensus to the answer. But what about question 4?

If you have the answers, the decision to mark a result as a pass or a fail is easy.
However, if you just have the question and no point of reference for the answer, how can you possible mark the answer?

The first two questions, the answer seemed obvious, George and 100 years but with the benefit of a point of reference, it was a simple task to dismiss the obvious for the correct. However, with question 4, it's not so easy.

The obvious answer might be B, under 4 seconds - if you listen to John Mueller, the Senior Webmaster Trends Analyst at Google. See his statement here.

Or the obvious answer might be C under 10 seconds - if you take note of the findings of Backlinko’s analysis of 5.2 million desktop and mobile webpages! Check out the stats here

Or… it might be A or D… it all depends!

Simply, having a good question is not much help if you do not have a definitive answer. To be able to mark a test pass or fail requires that you know that definitive answer. Knowing the answer is much more important than knowing the question when it comes to testing.

So, what got me thinking this way?! Well, it was a requirement for some ad hoc website testing that read ‘...website links must be obvious and accessible’. That was it. No other documentation around that requirement, just a link to the website. I had to devise a test that could mark links as ‘pass’ or ‘fail’.

'This link is in blue and has an underline… Pass'

The question for each link is: “‘is this link obvious?” But what is the answer?

On the face of it, the answer seemed obvious: “...links should be underlined and in blue". Industry standard, right?!

'This link is blue but no underline… Fail'

Except that the link itself is blue and all the other text is black… so that’s obvious right? So a pass? 'This link' is in the footer section of the website, which is blue, the link is white and reads “Click Here To Chat Now” but not underlined, is that obvious?

You see the dilemma, unless it has been agreed what ‘obvious’ for each class of link will mean, then the result is open to interpretation. ‘All textual links will be underlined and coloured blue’ is great, until you decided that actually, the small footer links (privacy policy etc.) will be underlined and grey. Knowing the answer for each link is more important than knowing the question: 'is this link obvious?'.

Of course in any real project, you seldom get all the answers upfront and so asking the question is an important way of getting the answer. I am certainly not saying you should always wait until everything is nailed down before you write your tests.

However, I am suggesting that as testers, we should not get too wedded to the question and that if as a result of a test, an answer gets changed, then we should keep in mind that the answer is the important thing that drives us. Get the right answer and you can ask the right question.


The correct ‘pass’ criteria for the questions: 
1: C
2: D
3: D
4: There is no universally accepted pass criteria

SSE Health Check Case Study

Topics: Software Testing, Test Requirements, Test Insights

nFocus Blog

Welcome to the nFocus software testing blog. As thought leaders and technical innovators, we created this blog to distil our thoughts, ideas and musings on improving software quality.

Fill out the form below to receive future communications from nFocus including our latest:

  • Blog articles
  • White Papers
  • and plenty more!

Follow us

Follow us on LinkedIn to see our latest content!

Subscribe Here!

Recent Posts

Posts by Topic

see all