Upload
bernd-beersma
View
82
Download
1
Tags:
Embed Size (px)
Citation preview
.......MAGAZINE
August 2013 $10.95
The Testing Tsunami
Rethinking the Role of AutomAted softwARe
testing
good things Come: Syncronization in HP Functional Tester (QTP)
Software teStingAutomAtedAutomAtedAn AUTOMATED TESTING INSTITUTE Publication - www.automatedtestinginstitute.com
Continuous Load Testing
Agility Applied to AutomAted peRfoRmAnCe
testing
PlannedAgility
Selecting the Right Tool
the pyRAmid AppRoACh to seleCting A tool to meet
youR needs
22 Automated Software Testing Magazine www.automatedtestinginstitute.com August 2013
Selecting the Right Tool
The Pyramid Approach to Test Tool
Selection
By Bernd Beersma
www.automatedtestinginstitute.comAugust 2013 23Automated Software Testing Magazine
A test tool is one of the best tools
to help us throughout the test cycle
because test tools can support almost
all activities we do in testing: test
management; test design; etc. But if we
do a Google search, we get hundreds of
thousands of hits for test tools. So how
do you as a tester, test manager or test
specialist know how to select the best
tool for your day-to-day business – one
that fits your purpose?
“
“
Selecting a tool, that eventually fails to meet your requirements or expectations, can lead to:
• Poor automation
• Frustration
• Lack of commitment
• Bad business case
• Shelfware
The tool selection pyramid is a structured selection approach that will help to avoid these pitfalls.
The Tool Selection PyramidThe Tool Selection Pyramid is composed of four different phases:
1. Long List
2. Short List
3. Proof of Concept
4. Pilot
Phase 1: Long ListThe outcome of the long-list phase is, as the name states, a long list of test tools. But getting there requires a few steps:
1. Identify your stakeholders.
If we want to implement a test tool in our organizations, this will influence the test team and other departments, IT and the maintenance departments, for example. We want to identify all those who are involved
2. Define measurable, feasible goals for Test Automation (TA).
24 Automated Software Testing Magazine www.automatedtestinginstitute.com August 2013
Be sure to involve all the stakeholders. This again creates commitment, but the other departments can also add goals of their own. For instance, business wants to decrease the time to market when you select a tool for automated regression testing.
3. Create a business case for TA. We need to set up a business case to be sure the project has enough support in the organization, and can be brought to life.
4. Set up a project team. The next thing we need to do is set up a project team for the tool selection project. Again, be sure that people from the different identified departments are on this project team, although the majority of the team can be from the test department. Someone from IT and maintenance – but also a developer – can be of real value to your team not only to create the support within the organization, but they need to help identifying requirements that are not test-related but are an essential part of the selection process.
5. Define general requirements. In this step, focus on defining general requirements like the need to automate
regression test or need to support a defect-management process. These are on a high level, and mind mapping is a good way to gather these requirements. This approach helps to get the high-level requirements for the tool in a fast and efficient way.
6. Gather general tool information. After defining the general requirements, explore the tool market. Look at tool-related websites and magazines, and events.
7. Create a long list. This list contains the tools and vendors we think should enter the next phase in our tool selection pyramid, the short-list phase.
Before entering the next phase, there is always a moment of consideration about whether or not we are ready to enter the next phase. This go/no-go moment gives us the possibility to restart the previous phase or start the selection process over.
Phase 2: Short ListThis phase also consists of several steps:
1. Define detailed requirements.
In the long-list phase we came up with
general requirements. In this phase, we need to go a little deeper into these requirements. For example, we needed a tool for automated regression testing. Now, with a mind map, we find the need for tools to support the use of external data, or tools to support cross-browser testing.
2. Gather the detailed requirements.
Add them to a scoring card and determine weight and priority. The scoring card is the basis for the next step, the Request
for Information (RFI).
3. An RFI goes to all the vendors on the long list.
Sometimes this is not possible if there is no vendor or party to whom to send the RFI. But if you still want to score, you can always see if you know an independent consultant who has experience with that tool to help you to fill out the form, or find a third party experienced with the tool and ask them to fill in the RFI. In the worst case, the project team has to fill it out. After getting back the RFI, you need to determine overall score. As you can see, this is only an example, but already 20 different main categories are identified here. So this shows tool
After drafting the requirements we invite
the vendor(s) to set up a plan for the pilot
together with members of the project
team. This needs to be a solid plan, with
fixed timelines and dates, because this
pilot phase needs to have an end, after
which the project team can start the
evaluation.
www.automatedtestinginstitute.comAugust 2013 25Automated Software Testing Magazine
selection is not that easy.
4. A demo.
Invite at max the top five scored tools to do the demo. The demo is like first contact, and you never get a second chance to make a first impression. Be sure to invite all the project members, stakeholders to let them see the demo. They can help you to decide afterwards which tools to put on your shortlist. Preferably three vendors will end up on the short list and this completes our second phase.
Again, before entering the next phase, there always is a moment of consideration if we are really ready to enter the next phase. This go/no-go moment will always give us the possibility to restart the previous phase or even start all over again with the selection process.
Phase 3: Proof of ConceptNow we entered our Proof of Concept or POC phase, during which several vendors or consultants are invited to do a proof of concept. This is like a test drive for the tool, to see how the tool performs in our situation and environment, and to see if the tool meets our requirements. Again, this phase is completed by executing several steps.
First of all, we draft the requirements for the successful completion of the POC. These requirements are the same for all participants of course, to create a fair competition. The tool must support all technologies that are part of a certain business process chain. After completion, the requirements drafted will be also part of the acceptance criteria for the POC.
Second we invite the vendors to do the POC, but this is a combined effort between us and the vendor. We need to be clear on what we expect of the vendor. This not only counts for the availability of the Software Under Test (SUT) but also of internal resources who will be executing the POC with the vendor. We need to setup a plan so we are sure the POC will be executed and finished on
time. Be sure that all systems are ready when starting with the next step in this phase, the actual execution of the POC.
Keep in mind that user accounts have to be set up, as does a test date, etc. If you are sure that everything is good to go, start executing the POC. This is also a combined effort. The more you involve the team by the selection process, the better the results of the selection will be.
After you and the vendors finished all the POCs, an evaluation of the POCs needs to be conducted. We will do a session with the project team and will see if the POC meets our requirements. The goal of this step is to choose the best tool, preferably one tool; you have selected two tools that meet all the requirements. If this is the case, no worries, we will come up with the definitive selection in the pilot phase.
But, we haven’t completed this phase yet, the next thing to do is to send a Request for Proposal. Even though I strongly believe that money is a bad driver, we still want to know what licenses, trainings, maintenance cost. We need this input to validate our business case. In the case of an open-source tool, the same procedure as for an RFI is started. Now with all this information gathered we are ready to update the business case and confirm we are still on track with our tool selection.
This completes this phase and we will be entering the pilot phase, but not before we do a go/no-go, but again, we are always allowed to restart the phase or even the whole selection project.
Phase 4: The PilotDuring this phase, one or two tools will be used in the pilot project for a longer period of time. It’s best to setup a separate project for this phase in a test environment.
Like in all of the other phases, we have to follow a few steps to complete this phase. But, after completing these final steps we have selected the best tool to help us with our testing, so completing this phase is worth the wait.
Again we start with the requirements for completion of this phase. We want to evaluate the outcome of course against our acceptance requirements. An example of a pilot requirement can be, “testers must be able to automate the regression test by themselves.”
After drafting the requirements we invite the vendor(s) to set up a plan for the pilot together with members of the project team. This needs to be a solid plan, with fixed timelines and dates, because this pilot phase needs to have an end, after which the project team can start the evaluation. After setting up this plan, we start the actual pilot with a kick off, so all participants in the pilot know what is expected of them, but also to create a feeling of collaboration between the involved parties.
No we are ready to actually start with the execution of the pilot. This is a combined effort with shared responsibility between the vendor and the project team. The goal is to prove that the tool can be implemented within the project and the organization.
Now that we have executed the pilot, final step is evaluation of the pilot. Gather the project team and see if the selected tool meets the requirements and acceptance criteria. And if all requirements are met and all systems are go, we have completed our selection project and the best tool for the job is ready to be implemented.
That leaves me with some final words of advice. Whenever you start selecting a tool, be sure to create awareness within your organization. Make it a combined effort and a shared responsibility between you and all the possible tool vendors or consultants, and always keep in mind that if you don’t feel comfortable after completing a phase, start again.
Learn more at TestKIT OnDemand:
http://www.ondemand.testkitconference.com
Wait! there’s more!
selection pyramid