Upload
elizabethdiazqa
View
326
Download
1
Embed Size (px)
Citation preview
Start, Grow, and Perfect Exploratory Testing on Your Team
Kevin DunneDirector of Product Strategy
QASymphony
Kimberly StockettQA Manager
KMS Technology
Agenda
1. What is exploratory testing (ET)?
2. What is NOT ET?
3. How can I start an ET practice?
4. How can I expand the footprint of my ET practice?
5. How can I measure & improve the effectiveness of my ET practice?
What is exploratory testing?
1. Parallel test planning, test design, and test execution
2. Specific yet flexible
3. Aligned towards investigation of potential opportunities
4. Values depth and attention to detail during testing
5. Fosters knowledge sharing and awareness
Parallel Planning, Design & Execution
Unlike traditional testing techniques, planning, design, and execution happen concurrently, allowing efficiencies of time as well as flexibility in approach
Plan Design Execute Report
Plan
DesignExecute
Report
Specific yet Flexible
Exploratory testing provides a specific lens through which to perform testing – whether that be a user persona, functionality, criteria (i.e. localization), etc.
However, it allows testers to use the tool as an end user would, not necessarily as the product owner envisioned it
Manual Scripted Testing
I tested the application as the script prescribedExploratory Testing
I tested the application as the end user would
Investigating Opportunities
Exploratory testing rewards testers who identify unknown areas of “opportunity” within the application, as they are essential in maintaining a backlog of future test charters
Manual Scripted Testing Exploratory Testing
Knowledge Sharing
Exploratory testing relies on knowledge sharing to reach full potential – developing testers who understand the impact of more areas of the application allows them to identify more areas of risk and opportunity
Plan
DesignExecute
Report Transfer Learning
Example Questions to Ask
• Have you seen this before?
• What am I not considering?
• Why would someone do this?
• How would you have tested this?
What is Not Exploratory Testing
1. Exploratory testing is NOT unstructured testing
2. Exploratory testing is NOT the only form of testing
3. Exploratory testing is NOT throwaway work
4. Exploratory testing is NOT impossible in a regulated environment
ET is NOT Unstructured Testing
While exploratory testing allows for flexibility in the exact path of the application that is tested, it is NOT unstructured, in that it still contains parameters such as:
1. A goal of the exploration
2. A log of the activity performed
3. A lens through which the testing is performed (i.e. a user persona)
Performing exploratory testing without involving some parameters such as the above allows a greater risk of unsuccessful implementation of exploratory testing
ET is NOT the Only Form of Testing
Exploratory testing is best suited as a complement to automated and manual scripted test cases. It can feed these types of testing to create greater depth in testing, and also to identify any potential gaps in coverage.
Potential New Feature Testing Cycle
Code Developer Unit Test
Exploratory Testing
Manual Scripted
Test
Automation Regression
Test
Exploratory Testing
Feature “Delivered”
“Let’s make sure this is worth writing
scripts against yet”
“Let’s make sure were still testing all
aspects of this”
ET is NOT Throwaway Work
Exploratory testing does NOT need to be extra work done on top of other testing methods – it can count on its own towards testing progress and coverage if properly accounted for. Some of the necessary information needed to manage it:
1. Charter2. Session Sheet3. Oral report4. Debrief5. Data Files6. Logs
"Any testing approach is manageable when you choose to manage it.” – Michael Boltonhttp://www.developsense.com/blog/2010/01/exploratory-testing-is-accountable/
ET is NOT Impossible in a Regulated Environment
Contrary to rumor and popular belief, exploratory testing is not only allowed in most regulated environments, it is also essential.
Starting an Exploratory Testing Practice
1. Who - choosing the pilot group
2. What - choosing the application
3. Where - choosing the team/location
4. When - choosing the release cycle
5. How - choosing the Exploratory Testing framework/approach
Who will be the first to explore?
Finding the right group of initial members to pilot exploratory testing is one of the most critical aspects of successful implementation. Pilot members should be:
• Comfortable with uncertainty and flexible
• Well connected in the organization
• A group that covers the ranges of skill and experience on the team
• Solid communicators and willing to speak up about their concerns
• Attuned towards new experiences as well as personal and team improvement
What will be the first application we explore?
The pilot application should be one that minimizes the risk to the business of serious impact if exploratory testing is unsuccessful. Some factors to look out for in a pilot application are:
SizeSmall to medium sized application – easier to track coverage
Time on MarketMature application – easier to see net impact of explorations
Business CriticalityNot the flagship product – executives will allow a longer period to pilot
Where will the first exploratory team be located?
To ensure the highest likelihood of success, it is important to choose a team that is located centrally with easy access to developers and visibility to executives overseeing the implementation
Centrally LocatedAllow for collaboration and knowledge sharing early on
Access to DevelopersWhen developers are near by, testers can provide feedback quickly
Executive VisibilityLocating the team near execs means better visibility of impact
When will we first introduce exploratory testing?
It is critical to introduce exploratory testing during a time where the pilot group can contribute their full attention to the new initiative and they are not being pulled in other directions:
• Avoid “busy season” – typically around big end of year releases• Avoid releases/sprints that overlap with long holidays• Avoid times of the year when application usage is high (i.e. year end closing for an
accounting application)
Be cautious of implementing exploratory testing while other change initiatives are going on. It can be helpful as there is a change “tailwind”, but be aware exploratory testing might be falsely attributed to other initiatives’ failure
How will we structure our exploratory testing?
Introducing exploratory testing within a framework will greatly increase the odds of success, and will reduce fear and uncertainty among the practitioners as well as executives.
Session Based Test Management is a popular framework for this, as it tracks all the important data on testing:
More info on SBTM: http://www.satisfice.com/articles/sbtm.pdf
• Session charter (includes a mission statement, and areas to be tested) • Testers involved Date and time executed• Task breakdown• Data files• Test notes• Issues • Bugs
Expanding the Footprint
1. Build champions within the organization
2. Share best practices and tips
3. Understand the limits and opportunities
Team #3 Team #3 Team #3
Build Champions within the Organization
There are two main ways we can build champions to spread the word about exploratory testing in our organization:
Initial Team
New Team
New Team
“Bowling Pin” Model “Infiltration” Model
Initial Team
Tester #1
Tester #2
Tester #3
Share Best Practices and Tips
A common threat in Agile organizations is the “information hoarding.” As loyalty to the Agile team grows, teams can lose site of the importance of improving the organization as a whole, and strive instead to improve there position as the best team.
Ways to mitigate against this include:
• Greater alignment with organizational goals• Transfer team members periodically
• Reward teams for sharing knowledge• Provide greater visibility across teams
Understand the Limits and Opportunities
There is a good chance that you will find early success with exploratory testing.
However, it is important to temper your excitement to roll out exploratory testing across your organization.
Executives often have an incredible memory when it comes to failed initiatives. Often, it is better to follow a measured approach than risk having exploratory testing barred from your organization entirely.
Measure and Improve ET Practices
1. What are we trying to accomplish by implementing ET?
2. How will we measure ET’s ability to accomplish these goals?
3. How will accomplishing these goals improve our ability to expand ET in the organization?
Benefits of Exploratory Testing
1. Increases focus on unique scenarios with greater potential to find new bugs
2. Drives engagement from testers and fosters critical thinking during the testing process
3. Boosts team morale and provides a change of pace from manual scripted testing
4. Encourages testers to think as end users and the "voice of the customer" and drive improvement in the product
Making the Case for Exploratory Testing
• Claim: ET will create more educated testers• Metric - increase in average # of testers capable of test a given
scenario• Claim: ET will provide faster feedback on new features
• Metric - reduction in user stories revised in subsequent sprints• Claim: ET will increase the alignment with customer needs
• Metric - reduction in support tickets/feature requests• Metric - increase in net promoter score/referenceable customers
• Claim: ET will garner communication product, dev, and testing• Metric - reduction in open defect time• Metric - increase in acceptance criteria met
Continuous Improvement
The path to success with exploratory testing does NOT end when implementation is complete.
You must continually improve to make sure that your program continues to add value in the organization
Plan
Implement
Evaluate
Reassess
Continuous Improvement
KMS has successfully implemented Exploratory testing techniques for all clients seeing benefit in quality and efficiency to the act of testing with an added bonus of boosted morale for the QA Analysts.
• Benefits we have seen
• Time boxed testing creates structure and limits
• Risk is apparent from application reviews
• Faster and deeper product understanding with hands on exploration
• Proactive over reactive approach in discovering bugs or workflow issues
• Items to keep in mind
• Worm holes in testing – knowing when to end a testing path is based on expereince of the QA Analyst
• Experiences drives when to annotate in the tool (QASymphony eXplorer) to make defect referencing easier
• Session definition – too much or too little
KMS is a fast growing global service provider with over 500 professionals trained on QA Symphony products. We are a Platinum Partner with QASymphony with over 3 years experience in Exploratory Testing Techniques.
About KMS Application of ET
EfficiencyIn a month long project consisting of 4 1 week sprints the team uncovered 146 bugs in 143 hours averaging 1.02 bug/hour
QualityThe team completed Mind Maps outlining 154 Charters completed 144 charters in 143 true testing session hours identified high priority bugs
Dramatic & Measurable Results from ET
Value (Bugs Found)20 High priority at 14%
57 Medium Priority at 39%66 Low Priority at 45%3 Closed as N/A at 2%
Resources/Thought Leaders
• James Bach - http://www.satisfice.com/• Jonathan Bach - https://jonbox.wordpress.com/• Michael Bolton - http://www.developsense.com/• Paul Holland -
http://testingthoughts.com/blog/author/testthought• Keith Klain - http://qualityremarks.com/• Brian Osman - https://bjosman.wordpress.com/
Questions?Kevin Dunne
[email protected]: @kevindunneQA
Linkedin: www.linkedin.com/in/kevindunneQABlog: http://www.qasymphony.com/blog/
Kimberly [email protected]