View
220
Download
0
Category
Preview:
Citation preview
8/14/2019 Access Your Way to Highest Test Team Performance
1/26
Copyright (c) 2009 LogiGear CorporationAll Rights Reserved. 11
.
Access Your Way to
Highest Test Team
Performance
Michael Hackett
Copyright (c) 2009 LogiGear CorporationAll Rights Reserved. 2
Why change?
Testing is often looked upon by
some people as an unmanageable,unpredictable, unorganized practicewith little structure.
Testers know this is not true.
This debate aside- there is alwaysroom for an improved Test Strategy.As quality professionals ContinuousImprovementis a rallying cry.
The first step in improving ourstrategy and turning our team into ahigher performance team is getting agrasp on where we are now!
8/14/2019 Access Your Way to Highest Test Team Performance
2/26
Copyright (c) 2009 LogiGear CorporationAll Rights Reserved. 3
What are we trying tosolve?
We commonly hear questions orcomplaints in development such as:
What is testing doing?
Testing takes too long
Testers have negative attitudes
There can be organization issues suchas:
Customer complaintsThe blame game
Product quality problems
You want to know:
What testing is effective?
Are we testing the right things at the right
time?Do I need a staffing upgrade?
What training does our team need?
How does the Product team value the testeffort?
Copyright (c) 2009 LogiGear CorporationAll Rights Reserved. 4
Objective
In this session my objective is togive you a framework forassessing your team.
- how to plan for an assessment,
- how to execute the assessment and
judge your current performance,- what to do with the information
- how to chart an improvement plantoward higher performance.
8/14/2019 Access Your Way to Highest Test Team Performance
3/26
Copyright (c) 2009 LogiGear CorporationAll Rights Reserved. 5
Overview
I will conduct this session based
upon practical experience ofconducting a number ofassessment programs at a varietyof development organizationsdomestically and internationally.
I will use my lessons learnedtohelp you avoid pitfalls and failureof your efforts
Copyright (c) 2009 LogiGear CorporationAll Rights Reserved. 6
Overview
Having decided:
Lets look the issue straight inthe eye. Lets deal with theseissues. We want to improve ourperformance and contribution tothe team.
The goal of doing a test processassessment is to get a clearpicture of what is going on in
testing, the good things, theproblems, possible paths toimprovement.
8/14/2019 Access Your Way to Highest Test Team Performance
4/26
Copyright (c) 2009 LogiGear CorporationAll Rights Reserved. 7
What is an Assessment?
Assess:
To sit beside, assist in the office of
a judge.
Merriam-Webster Online Dictionary
A test assessment is a datagathering process. To makedecisions we need data.
Copyright (c) 2009 LogiGear CorporationAll Rights Reserved. 8
Important note
This effort is much larger than test
team.
Issues will arise over:
Who owns quality?
What is goal of testing?
If this assessment is done well youmay step on some toes!
8/14/2019 Access Your Way to Highest Test Team Performance
5/26
Copyright (c) 2009 LogiGear CorporationAll Rights Reserved. 9
Standards
There are many standards or
assessment processes that are welldocumented for developmentorganizations.
CMM and ISO9000 are the mostcommonly mentioned.
The process of improving or aguideline for standardized processesis not as well agreed upon for testteams.
They are also expensive, timeconsuming, and for most US basedorganizations- what is the goal ofusing them?
Copyright (c) 2009 LogiGear CorporationAll Rights Reserved. 10
Introduction
In our view, a Test Strategy has 3
components that need to work togetherto produce an effective test effort.
Source: LogiGear developed this model by learning
from Mitchell Levy, Value Framework Institute
SP3
8/14/2019 Access Your Way to Highest Test Team Performance
6/26
Copyright (c) 2009 LogiGear CorporationAll Rights Reserved. 11
Introduction
We will focus our assessment on:
People, Process for software developmentand test,
Practice, methods and tools yourteam employs to accomplish thetesting task.
Copyright (c) 2009 LogiGear CorporationAll Rights Reserved. 12
Caution!
As a result of an assessment, yourwork may increase. There may be:
more demands for documentation
more metrics
more responsibility for communication
and visibility into testing.
8/14/2019 Access Your Way to Highest Test Team Performance
7/26
Copyright (c) 2009 LogiGear CorporationAll Rights Reserved. 13
Lessons Learned
The process will be doomed if:-no executive sponsorship
-no measurement program built
-no tools to support change
-risk averse culture
-the exercise is an effort to blametesting for project-wide failures
-no commitment about the goal oftesting
-no understanding of testing or qualityassurance across product team
-lack of responsibility for quality
Copyright (c) 2009 LogiGear CorporationAll Rights Reserved. 14
The Assessment Process
Phase 1 Pre-Assessment
PlanningGoal- set expectation, plan project, set
a timeline, firm executivesponsorship.
Action- meet with management ofgroups, layout expectation for resultof process, describe plan, establishtimeline
Result- agreement of expectation andbuy-in of the assessment processand follow-up commitment for
improvement.Deliverable- schedule and project plan.
Hint- make sure team understands theassessment is only the beginning.
8/14/2019 Access Your Way to Highest Test Team Performance
8/26
Copyright (c) 2009 LogiGear CorporationAll Rights Reserved. 15
The Assessment ProcessBefore you begin
Get executive buy-in.
Make a schedule and stick to it.
Give a presentation of what you aredoing, why and what you hope to getout of it.
Make a statement of goals or outline of
work as a commitment.
Copyright (c) 2009 LogiGear CorporationAll Rights Reserved. 16
The Assessment ProcessBefore you begin
Make a Scope Document a pre-approval/budget deliverable.
What is in scope and out?
The requirements process?
How much unit testing the developers do?
The Product Management team?The Automation Framework and Program?
8/14/2019 Access Your Way to Highest Test Team Performance
9/26
Copyright (c) 2009 LogiGear CorporationAll Rights Reserved. 17
The Assessment Process
Phase 2- Information
GatheringGoal- Develop interview questions and
surveys which become the backboneof your findings. Gather informationand develop more questions.
Action- gather documentation, developinterview questions, develop testteam survey.
Result- you are ready to begin theassessment.
Deliverable- Development Process
Documentation set complete,Interview questions, Tester Survey.
Hint- you will be collecting muchinformation. Get help! Stayorganized.
Copyright (c) 2009 LogiGear CorporationAll Rights Reserved. 18
Sample Documents toGather
Project/Developers
SDLC Document Engineering Requirements Doc
Test Docs
Test Plan
Template and example
Test Case Template and example
Status Reports
Test Summary Reports
8/14/2019 Access Your Way to Highest Test Team Performance
10/26
Copyright (c) 2009 LogiGear CorporationAll Rights Reserved. 19
The Interviews
The Interviews need to cover a
wide range of issues. Forexample:
Development Process
Test Process
Requirements issues
Change ControlAutomation
Tool use
Developer Unit testing
Opinions about the test team from
other groupsExpectation of the test effort
Political Problems
Communication issues
Copyright (c) 2009 LogiGear CorporationAll Rights Reserved. 20
The Interviews
In our assessment work we use arange of issues partially based onTPI developed by Tim Koomen,Martin Pol in:
Test Process Improvement : Step-By-Step Guide to Structured Testing
June 1999, Addison-Wesley
8/14/2019 Access Your Way to Highest Test Team Performance
11/26
Copyright (c) 2009 LogiGear CorporationAll Rights Reserved. 21
The Interviews
What we like about TPI is the structurethat enables us to frame thepresentation, discussion andimprovement effort.
However, it is essential to think aboutthe definitions of various attributes
being evaluated. Dont be afraid to useyou own attributes and definitions. Theprocess is not cookie-cutterand needsto be customized to cover all basesduring the assessment.
It is essential to formulate targetedquestions particular to tools andtechnologies so you get the informationyou want.
Copyright (c) 2009 LogiGear CorporationAll Rights Reserved. 22
Sample InterviewQuestions
One topic
The Development Process What do you think about the overall
development process?
Do you think the processes havebeen followed effectively?
Have people been trained in theprocess?
What is your opinion of the process?
How is the process constructed?
What do you think is theeffectiveness of the overall quality
process, including, requirementstracking, unit testing, buildengineering?
8/14/2019 Access Your Way to Highest Test Team Performance
12/26
Copyright (c) 2009 LogiGear CorporationAll Rights Reserved. 23
The Survey
In addition to the interviews, thesurvey and the doc review are themain sources of informationgathering we do.
Short answer, multiple choice,true/false is easiest to process.
These surveys go to the test teamonly. You are trying to discoverhow much knowledgethe teamhas in a few areas, such as:
Domain of the product
Technical knowledge
Software development lifecycle &process
Test/QA knowledge
Quality theory, test methods
Copyright (c) 2009 LogiGear CorporationAll Rights Reserved. 24
Sample SurveyQuestions
What are the strengths andweaknesses of your group?
How do you think other groupswould characterize the competencyof your group? What makes youdraw that conclusion?
What are the MOST effectivethings in your current approach totesting?
What are the LEAST effectivethings in your current approach totesting?
What would you like to CHANGEor IMPROVE if you could (pleaselist at least 3 items)?
8/14/2019 Access Your Way to Highest Test Team Performance
13/26
Copyright (c) 2009 LogiGear CorporationAll Rights Reserved. 25
Pre- Assessment
Just before the Assessment
begins you may want to havea meeting with all staff thatwill be interviewed andsurveyed so that they willunderstand the process and
know you are sampling from awide group. You may want totalk about confidentiality
here covering this may be a
key to success of failure ofthe assessment.
Copyright (c) 2009 LogiGear CorporationAll Rights Reserved. 26
The Assessment Process
Phase 3- AssessmentGoal- Conduct interviews and develop
preliminary findings
Action- gather and reviewdocumentation, conduct interviews,
send out and collect surveysResult- you build a mountain ofmaterial and information to review.
8/14/2019 Access Your Way to Highest Test Team Performance
14/26
Copyright (c) 2009 LogiGear CorporationAll Rights Reserved. 27
The Interviews
The interview process is the mostvaluable aspect of the assessment.
In my experience the interviewquestions do not all have to beanswered- they may be but they alsomay be jumping off points fordiscussions or issues that are on the
mind of the interviewee.
Keep it under 90 minutes.
Hint: Since it is your job to take notesand facilitate discussion and listen
and perhaps ask more questions- setaside time between scheduledinterviews to synthesize, take notesand catch your breath.
Copyright (c) 2009 LogiGear CorporationAll Rights Reserved. 28
The Interviews
Who you choose to interview:
Be careful not to pick only friends or
people who will tell you what youwant to hear.
Pick the widest range possible- thelongest term employees, the newest,the biggest complainer, the quietest,
the least techie, the most tech. Dont interview too many people but
you make sure you get a great mix orpeople from inside and outside thetest group.
This process does not only work to
focus on testers.
8/14/2019 Access Your Way to Highest Test Team Performance
15/26
Copyright (c) 2009 LogiGear CorporationAll Rights Reserved. 29
The Assessment Process
This job can be very large. Criticalreview of the documentation,
interviews, review of surveys is a fulltime job if the project is to be done ina timely fashion.
Copyright (c) 2009 LogiGear CorporationAll Rights Reserved. 30
The Assessment Process
Phase 4- PostAssessment
Goal- Synthesize information into a listof findings
Action- Review, collate, think, formopinions, make postulations.
Result- You have taken all theinformation gathered from the
document review, interviews andsurvey and developed a list offindings.
Hint-get help.
Deliverable- List of Findings, Collated
Survey answers, Collated Interviewresponses, Staff Assessment, TestGroup Maturity Ranking
8/14/2019 Access Your Way to Highest Test Team Performance
16/26
8/14/2019 Access Your Way to Highest Test Team Performance
17/26
Copyright (c) 2009 LogiGear CorporationAll Rights Reserved. 33
The Assessment ProcessPost-Assessment
Findings can be categorized into:
PeopleTechnical Skill
Interpersonal skills
Process
Documentation
Test Process
SDLC
Practice
Strategy
Automation
Environment
Tools
And more subcategories that suit your need.
Copyright (c) 2009 LogiGear CorporationAll Rights Reserved. 34
Staff Assessment
Rank of Test Team based upon ourown assessment scale
(2 sample categories):
III. Mathematical Skills
Understand concepts behind data processing algorithm
Apply mathematical operations to such tasks as frequency distribution
Apply statistical analysis techniques
Determine testability, reliability and validity
IV. Reasoning Skills
Define problems
Collect, interpret and analyze data
Establish facts and draw reasonable if not valid conclusions
Interpret complex technical instructions in mathematical or diagram form
Deal with several abstract and concrete variables
Trouble-shoot logically and reasonably
0 = No Knowledge 1 = Some Knowledge R = Required
2 = Knowledgeable 3 = Lead Expert ise D = Des irable
8/14/2019 Access Your Way to Highest Test Team Performance
18/26
Copyright (c) 2009 LogiGear CorporationAll Rights Reserved. 35
TPIhas twenty areas to rank.Here are an example three ofthe lifecycle items.
Test Group Maturity
Copyright (c) 2009 LogiGear CorporationAll Rights Reserved. 36
The Assessment Process
Phase 5- Presentation of
findings with project sponsor,executive sponsor and team.Goal- Present Preliminary findings to
executives and project sponsor. Getagreement on the highest priorityimprovement areas.
Hint- be prepared for a very differentinterpretation of the findings than youperceived.
Deliverable- Improvement Roadmap
8/14/2019 Access Your Way to Highest Test Team Performance
19/26
Copyright (c) 2009 LogiGear CorporationAll Rights Reserved. 37
Presentation of Findings
Be prepared for difficult areas.
Before you have a presentation tothe whole team about the results ofyour work, have a review of findingsand roadmap with the executivesponsor to discuss any difficultareas, confidential issues.
Pick the top 5 or top 10 to becomethe catalysts for change or roadmapof actionable improvement.
Copyright (c) 2009 LogiGear CorporationAll Rights Reserved. 38
The Assessment Process
Phase 6 - Implementation ofRoadmap
Goal- setout goals with timelines andmilestones and subtasks to accomplishthe tasks agreed upon for improvement.
Action- Develop a schedule forimplementation of the improvementplan.
Hint- Get some aspect of the projectimplemented immediately so people cansee tangible results right away- even ifthey are the smallest or easiestimprovement tasks. Roadmap tasksmust also be achievable and budgeted.
Deliverable- The Roadmap forImprovement
8/14/2019 Access Your Way to Highest Test Team Performance
20/26
Copyright (c) 2009 LogiGear CorporationAll Rights Reserved. 39
Roadmap Task Example
Title: Integrate better between onshoreand offshore teams to gain higher
productivityPriority: 4
Specific:
Provide training and support to enablestaff on both sides to collaborate moreeffectively
Structure the work environment andrelationship to make it more conducive tosuccess
Provide an effective productivitymanagement framework help facilitateoffshore activities
Solicit feedback and address thefeedback accordingly
Schedule:
Date for completion
Copyright (c) 2009 LogiGear CorporationAll Rights Reserved. 40
Roadmap Task Example
Title: Change/refine test automation strategy,specifically, methods and tools.
Priority: 4Specific:
Improve test automation productivity,failure analysis, scalability, and reducemaintenance costs
Make test automation efforts visible andquantifiable in a meaningful way
Make test automation effortsmanageable
Schedule:
Date for completion
8/14/2019 Access Your Way to Highest Test Team Performance
21/26
Copyright (c) 2009 LogiGear CorporationAll Rights Reserved. 41
The Assessment Process
Month 1 Month 2 Month 3 Month 4 Month 5 Month 6 Month 7 Month 8 Month 9 Month 10 Month 11 Month 12
Task 2:
Test Strategy Workshop
Metric Development and
Improvement Workshop
Task 3:
Final Presentation of Test
Strategy and Execution
Task 4:
Test Planning Design
Analysis Skill Development
Workshop
(3 individual workshops)
Task 1:
Presentation of Findings
Long Term Goal
Implimentation: TBD
Task 5::
Follow-Up Assessment and
Coaching
Copyright (c) 2009 LogiGear CorporationAll Rights Reserved. 42
The Assessment Process
Once the Assessment andPresentation of Findings iscomplete it is typical to reformulatethe Test Strategy.
One of the most common findings
from an test assessment isdiscovering that the does what itdoes because that is what it hasalways done- not because it iswhat is most effective.
8/14/2019 Access Your Way to Highest Test Team Performance
22/26
Copyright (c) 2009 LogiGear CorporationAll Rights Reserved. 43
Testing Strategy in Context
A test strategy is a holistic
plan that starts with a clearunderstanding of the coreobjective of testing, fromwhich we derive a structure for
testing by selecting from manytesting styles and approachesavailable to help us meet ourobjectives.
Copyright (c) 2009 LogiGear CorporationAll Rights Reserved. 44
Test Strategy
Source: LogiGear developed this model by learning
from Mitchell Levy, Value Framework Institute
SP3
8/14/2019 Access Your Way to Highest Test Team Performance
23/26
8/14/2019 Access Your Way to Highest Test Team Performance
24/26
Copyright (c) 2009 LogiGear CorporationAll Rights Reserved. 47
The Roadmap to change
Metrics Program EvaluationWhile doing the assessment you will,of course, assess the success of themetrics program you have in place.
If you dont have an effective metricsprogram in place now is the time tostart one.
In addition you will want to addmetrics to observe what changes inthe SDLC and the product as a resultof the assessment.
Copyright (c) 2009 LogiGear CorporationAll Rights Reserved. 48
Metrics
If we dont measure we cannever improve!
Anything worth doing is worth
measuring.
8/14/2019 Access Your Way to Highest Test Team Performance
25/26
Copyright (c) 2009 LogiGear CorporationAll Rights Reserved. 49
Metrics
Metrics have 2 goals for yourorganization. They are notmutually exclusive.
First- ship an improved productwith a better understanding ofrisk and test coverage.
Second- improve the process sothe next product can be releasemore efficiently with less stressand greater knowledge of the
risk.
Copyright (c) 2009 LogiGear CorporationAll Rights Reserved. 50
Metrics must be Actionable
Project Management- getting the
product out. These are typicallymeasures of software stability andActivity How many test cases done Defect counts Hours tested against a build
Code turmoil Requirements stability
Process Improvement Defect Aging Requirement Stability Valid Defects found vs. test method Bugs by severity post release
8/14/2019 Access Your Way to Highest Test Team Performance
26/26
Copyright (c) 2009 LogiGear CorporationAll Rights Reserved. 51
LogiGear Corporation
Contact:Michael Hackett
michaelh@logigear.com
www.logigear.com
Recommended