Embed Size (px)
Citation preview
Risk-Based Testing for Agile Projects
PRISMA®
1
Erik van Veenendaal
www.erikvanveendaal.nl
Erik van VeenendaalErik van Veenendaal
� In testing since 1989 working for many different clients and in many different roles
� Author “TMap”, “ISTQB Foundation” and “TMMi” and many other books and papers
� Former Vice-President International Software
www. erikvanveenendaal.nl
� Former Vice-President International Software Testing Qualifications Board (ISTQB)
� TMMi Foundation board member� Regular keynote and tutorial speaker� Winner Testing Excellence Awards� Living in Bonaire (Caribbean island)� Table tennis player and coach
Improve IT Services BV 2
Testing ChallengesTesting Challenges
� Increasing importance and size of software in society as a whole
− e.g., amount of software in consumer product doubles every 24 months,
− number of defects is almost constant− number of defects is almost constant
� High Competition & Outsourcing− Time-To-Market, Product Quality, Price Levels are essential for
business success
� Testing often takes 30 - 40% of project costs
�Risk-based testing deals with both effectiveness and efficiency
Improve IT Services BV 3
AgileAgile ≠ ≠ SilverSilver BullitBullit
100
1000
10000D
efec
ts
Improve IT Services BV 4
1
10
100
10 100 1000
Project Size (KLOC)
Def
ects
“despite continuous growth in adoption of agile, many organizations are stillfacing the the same challenges as in previous years”, World Quality Report
What is Risk?What is Risk?
� “A factor that could result in a future negativeconsequence; usually expressed as impact and likelihood” (ISTQB Glossary)
�Testers ‘only’ have the responsibility to identify the
Improve IT Services BV 5
�Testers ‘only’ have the responsibility to identify the risks and provide information on their status
� “to dare to undertake”− management attitude and style……..
Testing = Risk ManagementTesting = Risk Management
� Objective: most feasible coverage− effective usage of limited resources− optimize test effort
� Limited resources�staffing�staffing�infrastructure�time !�..
� The right level and type of coverage on the rightparts at the right time
� Always applicable, whether it is V-model, Agile or ....Improve IT Services BV 6
Our Challenge….Our Challenge…. if if only we only we knew !!knew !!
Improve IT Services BV 7
RiskRisk--Based TestingBased Testing
�Risk identification looks at ways of establishing what the risks are and where they are
�Risk analysis looks into the critical, complex �Risk analysis looks into the critical, complex and potential error prone areas
�Then we build tests to mitigate the risk
�Subsequently we monitor and reportregarding the risks
Improve IT Services BV 8
Based on Practical ExperiencesBased on Practical Experiences
Improve IT Services BV 9
9
Risk IdentificationRisk Identification
� Split up test basis in functional or technical items� V-model: based on requirements� Agile: backlog items / user stories for next sprint� Can also be based on a brainstorm session� Maximum number of appr. 35 risk items
Improve IT Services BV 10
Risk item 1 Functionality
Risk item 2 Security
Risk item 3 Functionality
Risk item 4 Interoperability
Risk AnalysesRisk Analyses
� Risk = impact x likelihood− What is the impact for the business ?− What is the likelihood that there are defects ?
� Determine factors based on previous projects, e.g., � Determine factors based on previous projects, e.g., defect patterns
Improve IT Services BV 11
Impact – business risk
Like
lihoo
d
tech
nica
l ris
k
You already know this !Exercise: Risk Factors
Factors From PracticeFactors From Practice
� Likelihood− complexity
− new development (level of re-uses)
� Impact− business importance
(“selling item”)
− financial (or other) damage (e.g. safety)
defect patterns / history
Improve IT Services BV 12
− interfacing
− size
− technology
− geographical spread
− inexperience (of development team)
damage (e.g. safety)
− usage intensity
− external visibility
− cost of rework
Customizationrequired
Risk Assessment in a NutshellRisk Assessment in a Nutshell
1) Stakeholder analysis: Assigning Factors2) Individual scoring
Business
importance
Usage
intensity
Safety
Item 155they shall
3) Consensus meeting4) Position the risk items in a Product Risk Matrix5) Define a differentiated risk-based test approach
Improve IT Services BV 13
Item 2
Item 3
Item 4
5
5
4
5
5
4
3
2
they shallmake
choices
Product Risk MatrixProduct Risk Matrix
IIIIII
H
xx
x
Must TestCould Testfocus of
unit testing
focus of
MoSCoW priorities
Improve IT Services BV 14
IVIVIIIIIIL
M
L HM
x
x
x
Likelihood
Impact
Should Test“Won’t Test”
focus of acceptance
testing
Play the Card Game: Risk PokerPlay the Card Game: Risk Poker
� Poker Planning / Wide Band Delphi based
Improve IT Services BV 15
Wide Band DelphiWide Band Delphi-- Poker Planning (1)Poker Planning (1)
� Bottom-up estimation techniques- wide variance in estimates- commitment of people involved- accurate estimation we use the collective wisdom of the team
� Each estimator has cards with valid estimates
Improve IT Services BV 16
� Each estimator has cards with valid estimates� A user story (product backlog item) is discussed� Each estimator selects an estimate� Cards are turned over� Discuss differences (especially outliers)� Repeat until consensus
16
User Story
Explain
Estimates
Moderator Estimator
Wide Band DelphiWide Band Delphi--Poker Planning (2)Poker Planning (2)
Improve IT Services BV 17
Collect
Share infoOK?
Finalestimate
OK?
17
Risk Poker the same, but…Risk Poker the same, but…
� Scoring separately for likelihood and impact� Discussion shall be factor based� Limited number of cards for each value (color)� Agreement, Product owner (impact) and lead-
Improve IT Services BV 18
� Agreement, Product owner (impact) and lead-developer (likelihood) act as tie breaker
18
Example User StoryAs as bank account holder, I want a yearly financialoverview of my accounts, so that I can submit my taxes
Impact ?Likelihood ?
Let’s PlayLet’s Play
Impact:
19Improve IT Services BV
Likelihood:
AgileAgile Product Product RiskRisk MatrixMatrix
exampleUser Story
Improve IT Services BV 20
User Story
See any “issues” with this?
Differentiated Test Approach !!Differentiated Test Approach !!
� Test design techniques
� Reviews & inspection� Test design start-up
� Level of independence� More time & effort� Most experienced
person
Improve IT Services BV 21
meetings� Reviews of test design� Level of detail of test
cases� Exit (DoD) criteria,
e.g., coverage
� Priority setting� Regression testing� Re-testing
without this a risk assessmentdoesn’t make much sense !!
RealReal--LifeLife AgileAgile ExampleExample
IIIIII
HH
MM
xxx
Must TestCould TestTest design techniquesReview designSupport module testsReview module tests
Exploratory TestingSupport module testsReview module tests
Improve IT Services BV 22
IVIVIIIIII
LL
MM
LL HMM
x
x
xLikelihood
Impact
Should Test“Won’t Test” Test design techniquesReview design
Exploratory Testing
AgileAgile ““OneOne Page Test Plan”Page Test Plan”
Must Test….. Test Approach …..
Should test…… Test Approach …..
Shall includea definition of “done”
Improve IT Services BV 23
…… Test Approach …..
Could Test….. Test Approach …..
Would Test….. Test Approach …..A 1-2 hour session
to achieve a one pageSprint Test Plan
CommunicationCommunication
Improve IT Services BV 24
Product Risk Matrix Product Risk Matrix –– InitialInitial
US
58 US
27TechnicalRisk
IIIUS
31US
49
Impact - Business risk
Likelihood-Technical
IVIII
25Improve IT Services BV
Product Risk Matrix Product Risk Matrix –– ProgressProgress
US
58 US
27TechnicalRisk
IIIUS
31US
49
Impact - Business risk
Likelihood-Technical
IVIII
26Improve IT Services BV
Key learning pointsKey learning points
��A structured and A structured and practical approachpractical approach for for riskrisk--based based testing is testing is availableavailable
��Make it Make it AgileAgile , but use the formal rules, but use the formal rules
Improve IT Services BV 27
��Efficient, Effective and AttractiveEfficient, Effective and Attractive
��Define Define a a riskrisk--based based differentiated test differentiated test approachapproach
��Practice riskPractice risk--based based trackingtracking & “& “reporting”reporting”
DownloadsDownloads……
Go to www.erikvanveenendaal.nl for:� A full PRISMA white paper (e-book ) � Paper : 10 Recommendations on
Risk-Based TestingRisk-Based Testing� Paper : Product Risk Assessments in Agile Projects
(Risk-Poker)
� To acquire the PRISMA freeware tool sent email
� Book “Practical Risk-Based Testing; The PRISMA Approach”, ISBN9-78490-986070 (www.utn.nl)
Improve IT Services BV 28
Any questions.....?Any questions.....?
30
Thank you !!Thank you !!Improve IT Services B.V.
