Introduction: The Importance of Document Review
In testing, as in development in general, we rely on good documentation to help us ensure that we deliver the right product to our end users. These documents may be user stories, requirements, technical designs or any other document that you can rely on to “build It right” and “test it right”. All should all be subject to a critical review.
However, document review is not something most testers are taught and for many, the review will consist of little more than reading the document, pointing out typos and obvious mistakes. They are not aware of the two key elements they should be looking out for which are...
- A: Is the document correct in and of itself? For example, is it complete, accurate and sufficiently detailed to support the work that is required to be undertaken?
- B: Is it consistent with previous documents? For example, does it contradict previous statements or provides information that is incompatible with other documents?
(A quick test, how many of you spotted that the use of e.g. above was wrong and that it should have been i.e.?)
In order to help develop better document review skills, here are 10 techniques to employ when reviewing a document and a list of 10 checks you can use to ensure that the document is fit for purpose.
Document Review Techniques
1: Read the sentence several times, ideally out loud. Vary where you put the emphasis in the sentence to see if there is any other interpretation that could be logically made. Documents need to be un-ambiguous.
2: Look for statements that imply certainty and then challenge them to see if they hold true in every circumstance. Look for words that imply certainty, such as “always”, “never”, “every”, “all”, “must” and “must not”.
To test the validity of the statement, research as many scenarios as you can to see if the condition holds true. Coders will need to understand if there are any exceptions to the certainty statements.
3: Look for statements that imply uncertainty and challenge them to ensure that the rules that dictate the various responses are clearly documented. Look out for words that imply uncertainty, such as: “some”, “sometimes” “most”, “often”, “usually” and “mainly”.
4: Beware of vague verbs such as "handled”, “skipped”, “rejected” and “processed”. Check to see that the mechanism and consequences for such actions are clearly defined.
5: Check for vague verbs, items written in a passive voice, for example “the database is duplicated” or “the file is deleted”. Ensure that such actions have actors and ask if the “who” or “what” will undertake these actions is clearly specified.
6: Where a calculated value is specified, try working out three examples and checking that they all meet the requirement. Concentrate on edge cases, those that result in ‘0’ or some extreme number such as ‘100,000,000,000’ to see how the logic holds up.
7: Check for the use and misuse of the abbreviations i.e. (that is) and e.g. (for example). Check that where i.e., is used, then the given value is the only value possible (if not it should be e.g.). Check where e.g., is used, that the examples given are sufficient in breadth and depth to cover the requirement. The use of e.g., is particularly problematic in lists, unless there is a clearly documented list of all possible results. There is a danger that the code will not cater for all possibilities.
8: Use a ‘reading rule’, a folded piece of paper or a ruler to highlight one line at a time. This will help you to read what is written and not to scan ahead either consciously or subconsciously.
9: Highlight (if on screen) or better still, circle (if on paper) every punctuation. This will force your mind to stop at each point and break down the mental assumptions that your mind is making about the text.
(Side note: Studies have shown that proofreading a screen document is harder and less effective than proofreading on paper and others. (Read more here. See footnote.)
10: Look to see if technical terms, acronyms or abbreviations are explained somewhere so that there is no confusion as to their meaning. Be particularly aware of common shorthand that may be understood differently across departments and organisations; for example a google search of “system integration” returned these top three results, which do not all mean the same thing.
System Integration:
Mar 10, 2021 — "System integration is the process of joining software and hardware modules into one cohesive infrastructure, enabling all pieces to work as..."www.altexsoft.com/blog/system-integration/
What is system integration?Jul 12, 2018 - "System integration is the process of connecting different sub-systems (components) into a single larger system that functions as one to..."
www.gartner.com/en/information-technology/glossary/system-integration
What is system integration and why it is (really) necessary?
Mar 29, 2021 - "Systems integration plays a fundamental role within a company, as it facilitates communication between systems that do not normally communicate."
As you might imagine, where lives depend on the end product, document review is vital. So Boeing developed a checklist that I have found very useful over the years, here is a version you might find helpful:
Document Verification Checklist
The check list below is adapted from the Boeing Computer Services Requirements Checklist.
Complete
All items needed to specify the solutions to the problem have been included.
Correct
Each item is free from error.
Precise, unambiguous and clear
Each item is exact and not vague; there is a single interpretation; the meaning of each item is understood; the specification is easy to read.
Consistent
No item conflicts with another item in the specification.
Relevant
Each item is pertinent to the problem and its solution.
Testable
During program development and acceptance testing, it will be possible to determine whether the item has been satisfied.
Traceable
Each item can be traced to its origin in the problem environment.
Feasible
Each item can be implemented with the available techniques, tools, resources, personnel and within the specified cost & schedule constraints.
Free of unwarranted design detail
The requirement specifications are a statement of the requirements that must be satisfied by the problem solution and they are not obscured by proposed solutions to the problem.
Manageable
The requirements specification can be controlled in such a way that each item can be changed without excessive impact on other items.
Footnote:
Jill Barshay, Online Paper, 'Evidence increases for reading on paper instead of screens', (The Hechinger Report, 2019). https://hechingerreport.org/evidence-increases-for-reading-on-paper-instead-of-screens/.
www.altexsoft.com/blog/system-integration/
www.gartner.com/en/information-technology/glossary/system-integration
https://www.youredi.com/blog/what-is-system-integration
.png)
