Leveraging Time-Travel Techniques in Testing

I am often asked what are the biggest mistakes that companies, projects or testing teams make in execution. Well, aside from hiring me to do the testing in the first place, there is one area that does keep cropping up as being missed… thinking about and testing beyond the initial or obvious journey.

There are many labels attached to this – one of the best I’ve heard though is 3-dimensional testing. So, what do we mean by that?

Well, it’s about testing the solution past what is the obvious end point of the process or user journey. What happens next? Is this really the end of the journey or does something else need to happen? Or could something else happen?

We know that we need to look at the solution from all angles and accommodate all potential outcomes (via testing or a process) but this is a level above. It is about ensuring that the ongoing processes are also proven to operate correctly and effectively. It might worth doing user tests and run focus groups at this point. 

An example: many moons ago, I worked on a website solution for a high street bank that allowed customers to purchase personal home and car insurance online. Our initial estimate for the testing scope and environment requirements was challenged for a very simple reason. After all, the various combinations of policy had been ‘bought’ through the ‘website’ (with the various payment options selected), most people assumed that was all we need to prove to say that the solution worked. We covered the cancellation of policies at all possible stages - so what else was needed?

Well, once someone has bought an insurance policy, what happens next? They either renew it or they let it lapse. So, can the solution handle that? Does it offer all possible renewal options? Is the preferred option offered the best one for the company’s policy? If the customer opts not to renew, does the policy automatically lapse correctly? Is the renewal date affected by leap years?

But test environments exist in the now, so how can a test team provision a test environment to enable us to see how the solution handles scenarios that exist in the future? It’s not like we can all sit around for a year or so. Well, this is where we would need to time travel the test environments.

Time travel testing is how we test date and time sensitive functionality to validate business rules and logic. It can also be known as:

• Temporal Testing
• Time Shift Testing
• Forward Date Testing
• Date Simulation Testing

The kind of functionality or time-based scenarios that would be useful to be able to time travel test include:

• Leap year
• Daylight saving time
• Time-sensitive transaction triggers
• Expiry of policies
• Loan repayment
• Student enrolment

However, this throws up potential issues as it can add a significant amount of time to the testing effort and to the support requirements which in turn brings extra cost.

Which are illustrated by looking at the following options for time travel testing – all of which have significant pros and cons. Let's have a look at the key points below:

Option 1: Manually Shifting Server Time

This method involves manipulating the operating system clock via the system settings in order to time travel the server.

Pros

• No third-party tools required
• Intuitive

Cons

• System files & timestamps corrupted
• OS & application restores
• Takes a large amount of time
• Active directory & kerberos lockout
• Higher costs & resources required

Option 2: Isolate Servers

This involves purchasing additional servers, isolating them from the domain and then manually shifting the server time.

Pros

• No third-party tools required
• Intuitive
• Dedicated time travel environment

Cons

• Increased hardware and software
• Increased deployment, maintenance,  human costs and resources
• Test environment differs from production
• Same challenges as “Manual Time Shift”

Option 3: Hardcoding

Hardcoding means utilising the organisation’s Software Engineers to hand-code a time travel test harness into the application code.

Pros

• No third-party tools required
• You own the time travel test harness
• Built-in automation

Cons

• Increased lines of code
• Development & maintenance costs + risk of bugs
• Time & resource drain for Developers
• Unable to time travel third-party apps
• Potential loss of customers, revenue & shareholder confidence

Option 4: Test Automation Tools

The easiest and best way to perform temporal testing is through a third-party software tool.

Pros

• Instant time travel
• Find and fix all date bugs
• Expedite test cycle
• Save time, money & resources
• Reduce hardware & software costs
• Free up organisational resources
• Enable time travel in locked environments

Cons

• Cost
  Source

So, as we can see, there are significant time and cost implications for each of these options. But, what happens if we don’t prove those ‘follow on’ scenarios in this situation?

Well, the solution isn’t complete. Key functionality has not been tested and the potential pitfalls of that for any project could be catastrophic down the road.

When the damage is large enough, it can become newsworthy. Here are some examples that have made the news over the years.

1. National Australia Bank’s health payment system crashes by leap-year bug, blocking 150,000 customers from conducting medical transactions & processing payments.
   
   
2. Microsoft’s Cloud Platform Azure offline for over 12 hours due to leap-year bug.
   
   
3. VMware ESX update introduces date-bug which blocks all virtual servers from starting up, disabling customer’s infrastructure.
   
   
4. Thousands of Bank of Queensland Point of Sale Terminals crash from rollover date-bug, rejecting customer debit cards.

Source

In other words, it's a very important element in considering the testing effort for a project and not something that can be ignored. 

Which is why thinking beyond the obvious is a key attribute for effective Test Consultant. One that will hold you in good stead regardless of the client you are working for or the solution you are working on. 

Here at nFocus Testing, we are very well placed to help you with managing your test strategy. Contact our team by using our phone number 0370 242 6235 or email us at info@nfocus.co.uk.