Friday, 17 December 2010

Infrastructure Testing – What’s that??? Part 2

Now that you have considered the relevant areas to your particular solution, it’s time to assess which elements pose the biggest risk to you deliverable; there is no secret to how you approach this, here at nFocus we use pretty much an industry standard “Risk Based” approach.

We convene an appropriate forum of subject matter experts and elicit their thoughts on which areas pose the greatest risk, create test scenarios that will mitigate the risks identified, and test the solution, some of the areas for consideration could be:-

  • Security vulnerabilities

  • High Availability / Disaster Tolerance / Failover

  • Connectivity and interoperability between system infrastructure

  • OS Patch Status and updates

  • Load balancing, Network routing and firewalls, name resolution

  • System logging

  • Performance

  • Virtualisation

  • Operational Procedures

Another list of items to consider, but at this stage you should have a pretty good idea of the shape the solution will take and the infrastructure and products that will support it. Some of the work identified during the risk assessment will be undertaken by engineers; some could be delivered as part of a contract via a hosting company for example, however you should assess what level of governance to apply to these activities; confidence in the end product for your stakeholders is paramount.

Related articles
Infrastructure Testing – What’s that??? Part 1

Thursday, 9 December 2010

What mobile device/os combination should we test on?

So, at nFocus we have been doing plenty of mobile app type testing lately and I thought it would be useful and interesting to give you an insight into one particular problem that we have been hitting : What device/os combination do we test on?

I was asked recently to estimate the testing effort as part of a bid for the development of an app that would be based on the apple iOs platform. The first thing to consider was, what are the devices and os combinations that are possible. The result is shown below, there is some great information on wikipedia that helped with the collation of this. What we found really suprised me, a test would have to be run 74 times on individual device/os combinations.

Clearly, that was never going to be possible, so the question is, how do you get smart about which ones you should attempt? The answer to this is not so clear, and depends on whether there is an existing user base, from which data can be extracted. In this specific case the answer was no, so we had to take a pragmatic approach and pick some major combinations to always test on and a more exhaustive list to execute tests on toward the end of the cycle. For our purposes, we got away with testing on 3 device/os combinations with a final pass being executed on an additional 7 combinations.

One of the most important things to consider in making these decisions is to engage the customer, they must be happy with the subset that is chosen and the reasons why.

One of my major concerns when it comes to "App" testing is the perception that because we shorten the word application down to app, that the testing of these solutions does not need to be as robust : "it s just a little app, why does it cost so much to test?" Mobile applications is big business and revenues are growing from the use of such applications, we need to recognise their importance and start testing them properly :)

Thursday, 2 December 2010

Infrastructure Testing – What’s That???

So you’re on a new project; a green field site and part of your remit is to test assure the infrastructure as well as the solution, where do you start? What is infrastructure testing? One definition could be “to assure that the platform the service is to be deployed on fully supports the solution in a performant and robust manner”

Considering this overview, some of the items for consideration could be:-

  • Tin – Commissioning the appliances into the data centre and powering up, what should you be looking for to assure this, and weed out the DOA (Dead on Arrival) kit

  • Network - Making sure that all of the Tin communicates with each other as designed, without any erroneous traffic

  • Tiers – Most internet facing solutions are built using the tier model to separate the internet, application and data for security purposes, should this have a penetration test applied by a specialist in that arena

  • OS – What special consideration and configuration will need to be applied to the software stack specified to support the solution? How do you assure the build prior to releasing the solution code and any supporting artefacts?

  • Monitoring – What system monitoring will be in place and how do you prove its functioning to the solutions specific requirement

  • Code – How can you utilise the functional testing to supply you with data for assuring the infrastructure solution
As you can see, loads of things to consider prior to any developed code being released from the CM process, this has been simplified above, the main point is that it can be planned and executed as a test discipline, utilising both testers and engineers, more to follow...

Related articles
Infrastructure Testing – What’s That??? Part 2

Wednesday, 20 October 2010

Software Testing Event - Last days to register!

"Getting the most out of Test within Visual Studio 2010, in the Azure Cloud!"

With projects moving to more iterative approaches like agile, why should I tightly integrate testing into the application lifecycle, and how can I use Visual Studio 2010 to help me?

There is only a few days left to register for this event, for more information, and to register your place, please visit:

Friday, 8 October 2010

Getting the most out of Test within Visual Studio 2010, in the Azure Cloud

A Microsoft and nFocus Software Testing Event - 26 October 2010

With projects moving to more iterative approaches like agile, why should I tightly integrate testing into the application lifecycle, and how can I use Visual Studio 2010 to help me?

We've come a long way from the days of testing coming last, after report generation, within the development lifecycle. Agile development practices and tooling have reaped many benefits but how do we ensure that testing is integrated end-to-end throughout the lifecycle? How can testing benefit from the same high class tooling and become a first class citizen?

Session 1 - “We want to be together”, why is it important that test is integrated into the application lifecycle?

Session 2 - Demo… A day in the life of a tester and a developer in the world of Visual Studio 2010, in the Azure Cloud.

To find out more, and to book your place, please click here.

Tuesday, 28 September 2010

Using Expression Encoder with Microsoft Test Manager

I thought this was worth sharing, Giles from the Microsoft UK Developer Tools Team, just blogged about the change from Microsoft Media Encoder to Microsoft Expression Encoder in the Microsoft Test Manager. Now we can have tighter control over what gets recorded, please see his post here...

Friday, 17 September 2010

Okay Azure, here I come!

Hello out there in the testing world, I've been busy developing new relationships of late and so I have not had time to get my hands dirty with any testing or test tool work. But that is all going to change now. I've got a couple of presentations coming up and in an attempt not to regurgitate the same old testing drivel, I have opted for a more practical demo- type presentation around cloud application testing. I'm going to be developing it over the next couple of weeks using Azure, so if I come across anything useful, I'll keep you posted.

Monday, 2 August 2010

September Software Testing Strategy Workshop date announced!

nFocus have announced a September date for the next Software Testing Strategy Workshop to be held at Intellect (the Technology Trade Association), Russell Square, London.

We only have 10 places available for this course, and we would encourage anyone interested in attending to book early to avoid disappointment.

For more information including course details, discounted rates, course testimonials, and more, please click here.

Related articles:
The last Test Strategy Workshop write up

Wednesday, 21 July 2010

nFocus at WPC10

nFocus was lucky enough to attend the Microsoft Worldwide Partner Conference in Washington DC this year and there was loads of great keynotes, sessions and networking. We were even treated to a keynote by President Bill Clinton at the Verizon Center

Brad Brooks (Corporate Vice President, Windows Consumer Marketing) introduced some of the Personal cloud stuff and I have to say it sounds great. All your consumer devices interconnected with your own personal cloud, syncing seamlessly, it’s like live mesh on steroids! They demo’d Kinect™ for Xbox 360 which went down a storm with the audience but I’m glad I have mine on pre-order, can’t wait for November, this device really rocks!!


Brad Books – Personal Cloud

Well, we managed to find one chat on Software testing by the one and only Brian Keller (Technical Evangelist for Microsoft Visual Studio), he demo’d Visual Studio 2010 testing tools in a rather small room with a poxy little screen but still managed to pull it off – I just had to get a photo with him :)


Brian Keller from Channel 9

My personal fav, Kevin Turner COO for Microsoft, I always love this guy and this time round he did not disappoint. He attacked their competitors Apple, VMware, Linux, Oracle and Google and painted a picture of how Microsoft's strategy for mobile, desktop and cloud will allow them to beat their top competitors. Turner also described market share gains against Google, VMware and even Apple (picture below taken on my Apple iPhone 4)!


Kevin Turner COO for Microsoft

I have to say one of the highlights for me was the UK party, a much sought after event with people from all over the world trying to gatecrash (they did let Jon Roskill in though). It was a 007 Casino Royal theme and everyone had a great night although I wasn’t to sure why they had a Dame Edna Everage impersonator roaming round, that was somewhat surreal!


Danny Crone (Technical Director for nFocus) and Jon Roskill (Corporate Vice President for the Worldwide Partner Group) having a good time at the UK Bond Party.

So roll on WPC11 which is due to be hosted in Los Angeles…

Thursday, 1 July 2010

Oi! What happened to my test results???

Hi Guys, I’ve been busy working on an agile project in the city using VS and TFS 2010 and thought I would share one of my experiences as we stood to lose some very valuable information as a result of a default option that is not that obvious from the get go! I have reported the issue to MS and they are going to change the default option in future releases.

First, a little bit of history with the project, we were experiencing late delivery of builds or incomplete builds that could only have minimal testing applied to them, so we didn’t notice any real problems at first. Then we started getting into high levels of testing and one of my testers came to me and explained that one of the tests he had executed before in the Test Manager was now showing as “Active” (meaning needs to be executed) which was a bit odd :( When we looked into it, it wasn’t just that one test, there was loads of them (but not all) that had gone walkabout :o(

After a lot of digging around in all kinds of places, I decided to check the build definition that we were using and noticed something I hadn’t noticed before…..the retention policy, it was set to keep only the last 10 successful builds! I was onto something!! It appears that the default (certainly under the MSF for Agile template) was to delete everything INCLUDING TEST RESULTS!!! The good news was that we hadn’t lost too much and the option can be modified to delete everything else but the test results (screenshots below).

This is the build definition, please note the "Specify what to Delete" option in the drop down.

Here, you can uncheck the Test Results check box to ensure that they don't get deleted along with the build.

I would recommend to everyone that they take a look at their build definitions and modify them appropriately to avoid getting into this situation. If you have any questions, then please feel free to comment here, tweet me @dannycrone or even use ye olde email address :)

Thursday, 20 May 2010

nFocus Software Testing Strategy Workshop at Intellect

As you may know, nFocus facilitated a Test Strategy Workshop in conjunction with Intellect on the 28th April 2010. The demand for the workshop was incredibly high, and was sold out weeks before the day.

The day was facilitated by one of our Principal Consultants, Sam Clarke (featured in the photo above). It was interesting how the audience encompassed a wide range of positions including directors, heads of development, QA, and operations. Both Sam and I agreed it was refreshing to see some of these senior people taking an interest in the value of testing.

The format of the day was made up of talks, exercises and discussions. This format worked well allowing tips, approaches and information to be freely traded by likeminded people as well as Sam’s impressive knowledge, expertise and helpful anecdotes.

“An enjoyable day, It was very satisfying to work with a group of open minded and receptive IT professionals who also contributed to the session” said Sam after the event.

We had some great feedback, including:

* “Came away with a lot of ideas to improve current testing processes and add new areas to the process”
* “Pitched very much at right level”
* “Very interesting, good anecdotes”
* “Sam delivered the subject in a clear and concise way”

So all in all, a really useful, and interesting day. Due to the overwhelming success and high demand for the course, we have scheduled a number of other Test Strategy Workshops, for more information please click here.

Tuesday, 20 April 2010

Testing at the Architect Insight Conference

It has been a couple of weeks since I presented at Microsoft’s Architect Insight Conference and I thought it was about time I got round to writing up my thoughts around the session. For those who didn’t manage to get to the event, the session was PM day one, and it was titled, “Testing, why it should be fully integrated in the application lifecycle?”

We ended up running the session in one of the two main speaker rooms, and a spare seat was hard to find. In all I would guess we had around 50 people in the room! (nerves I hear you ask? Yes a healthy amount!) In any case, it was great to see so many people interested in a testing session at an architect’s conference.

I have put a copy of the slide deck up here for reference but I was pleased that the session provoked plenty of questions around architecting solutions in agile projects with a focus on testing. Much of the questions seemed to be around unit testing, test driven development and automated testing and it seemed that some people are still getting some resistance to performing unit testing even though this is, in my opinion, a cornerstone for agile projects.

I stayed at the end for the networking and was glad that I did, as there was plenty of people who sought me out to discuss some of the challenges that they were facing.

I am hoping to record a re-run of this session as a video blog article as soon as possible, although I can’t promise it will be up there in the next week because I simply can’t find the time- sorry guys. In the mean time check out the Microsoft page for further event information and watch this space.

As ever, if you have any questions, please feel free to comment or tweet me @dannycrone.

Monday, 1 March 2010

VS2010 - Test Impact, one giant leap for testkind!

Hi there, finally got round to another blog article; this one is in the form of a vlog.
In the agile world, understanding the impact of code churn is vital, using the test impact collector in VS2010 testers can be told which manual and automated tests need to be rerun, I've put together this demo to show it working.
Soo...full screen the video, sit back, enjoy and please give us your feedback!

Please note the full screen button is located next to the progress bar and volume controls whilst hovering the cursor at the bottom of the video.
The vlog can also be viewed at our website

Thursday, 25 February 2010

nFocus to facilitate Intellect Software Testing Strategy Workshop: London, April 2010

I thought this may be of interest to some of our community.

"Reduce risk and increase your confidence that the quality of your software product is at a known quantifiable level."

This course takes you through the key aspects of forming a test policy and strategy allowing you to establish your weak and strong points and a plan of action. In essence a process of “getting to know what you don’t know” so that informed decisions can be taken on appropriate investment in testing.

The workshop approach allows interaction between attendees establishing common issues and their solutions. The class will be run as a sequence of short talks each followed by an interactive session allowing delegates to focus on areas of common concern. In addition to class material the key discussion points from the day will also be documented and circulated to the attendees.

Want more information? Please click here.

Wednesday, 17 February 2010

New video blog post due shortly...

Sorry we have been so quiet on the blog front at nFocus recently... it has been a really busy period! We hope to have a new video blog post on here shortly, along with a series of articles around VS2010, so please watch this space!

Thank you for bearing with us,

The nFocus team.

Tuesday, 19 January 2010

It’s testing Jim but not as we know it...

This is a short post to let you know nFocus and Microsoft will be facilitating a session at this year's Test Management Summit (27/01/2010).

The challenges of applying test discipline in the world of agile projects continues to be problematic. The session will explore how tightly integrating testing into the application lifecycle can protect the reputation of the business. This approach will be demonstrated using a state of the art Application Lifecycle Management tool which supports development and testing in a dynamic changing environment.

For more information and booking details click here.