Friday, 27 July 2012

Exploratory testing in Microsoft Visual Studio 2012

What is exploratory testing?
Some people think exploratory testing is completed in an uncontrolled manner. The tester will set out to determine if the software actually works without giving it a great deal of thought or consideration of his/her goals.

For exploratory testing to be considered a success the tester must have a certain level of knowledge about the AUT as well as being creative in approach. Unlike scripted testing an exploratory testing session may not be linked to any form of requirement, user story or back log item and so it can become an exercise which fails to deliver an audit trail.

Given the lack of test steps or testing context which you have with requirements, user stories and back log items exploratory testing can result in bugs being logged that are difficult for developers to reproduce, without test steps how is the bug reproduced? Unless the tester is actually capturing the exploratory test steps as they go, which would become a burden as well as a time overhead, then how do they demonstrate what has been done?

As testing is an activity which produces information about the AUT then in my opinion exploratory testing must be given the same level of control as any other testing approach. Exploratory testing sessions should be focused on product backlog items and each should have a direction and certain goals. It is also useful when performing an exploratory testing session to capture information about that steps are being taken and which data is being used, this information can be helpful when a bug is uncovered to allow the developer to reproduce the bug. Capturing this level of information along the way provides an opportunity to quickly convert an exploratory test in to a scripted test in the event of a bug being found, this then enables a bug to be retested when fixed.

Introducing MS VS2012
Microsoft Visual Studio 2012 has now reached Release Candidate and in this release we get some exciting new features with its quality tools. One of the more notable features is an exploratory testing component in Microsoft Test Manager (MTM).

With the Exploratory testing component we are able to be more focused with our exploratory testing sessions which also deliver full traceability. We have the option to complete our exploratory testing session in a completely maverick manner or from within the exploratory testing component we are presented with a full list of pre-defined Work Items.  If our Work items have been created correctly they will have Acceptance criteria which will provide the exploratory testing session with some focus.

The user can select a Work item and begin the exploratory testing session. This will ensure that every action carried out with this test will be associated back to the work item providing full traceability. The session will be in context with the focus on testing and the information gathered during the exploratory testing session is automatically linked to the Work Item for example; Bugs and Test steps are generated without any extra effort from the tester.

As with MTM in VS2010 when executing manual test scripts you are presented with a "Run options" dialog, a similar dialog "Explore options" is now presented when executing an exploratory test allowing the tester to choose which data and diagnostics to capture during the exploratory testing session.

During execution, by default,  the user is presented with a window similar to the test runner docked on the left which allows the application under test to be surfaced in the remaining desktop real estate. The exploratory test execution window presents many opportunities for the user to capture information as they are completing the exploratory test again presenting a huge time efficiency gain as well as an opportunity to maintain auditability.

When a bug has been identified, it is extremely simple to create a bug from the test execution window and capture all of the standard bug information. One of the most powerful aspects of this new component is that as a tester is completing the exploratory test it is recording the actual test steps being executed, so when a bug is being raised, all of the steps to reproduce are automatically populated along with other information such as; screenshots and user notes.

MTM Exploratory testing feature can re-use the information which has been captured during the exploratory test to quickly generate a test case with steps from the testers actions that where recorded by MTM. Both the new scripted test case and the newly identified bug are associated back to the work item thus providing a closed loop ensuring traceability and auditability of the exploratory testing process.

Another piece of information captured which is extremely valuable to the resolution of any new bug is the IntelliTrace feature which helps pin-point the actual line of code where the bug has impacted. This coupled with test steps and screenshots reduces the negotiation time between a tester and developer for all bugs - no more bug "ping pong"!

Microsoft Visual Studio – Microsoft Test Manager…… Don’t do exploratory testing without it!

No comments: