Test Improvement with TMMiTest Improvement with TMMi
Process Improvement for Present and Future
Erik van Veenendaal
www.erikvanveenendaal.nl
Erik van VeenendaalErik van Veenendaal
� Founder and major shareholder ImproveQS
� In testing since 1989 working for many different clients and in many different roles
� Author “TMap”, “The Little TMMi” and many other books and papers
www. erikvanveenendaal.nl
Improve Quality Services BV 2
other books and papers
� Vice-President International Software Testing Qualifications Board (ISTQB) 2005 - 2008
� Vice-Chair TMMi Foundation
� Keynote speaker, e.g. EuroSTAR, STAR-East
� Winner of the European Testing Excellence Award (2007)
ImproveImprove QualityQuality Services BVServices BV
� Service organisation in the area ofTesting, Requirements Engineering and Quality Management
� Consultancy, Subcontracting and Training
www. improveqs.nl
Improve Quality Services BV 3
� Consultancy, Subcontracting and Training
SW Process ImprovementQuality AssuranceIT-AuditingRequirements Engineering& management (IREB)
Testing (TMap, TMMi)Test Process ImprovementCertification (ISTQB)- incl. Advanced !!Inspections / Reviews
� Increasing size and complexity�Amount of software in consumer
products doubles every 24 months (Philips)
�Number of requirements for mobile phones doubles every 6 months (Nokia)
Computable, Sept.’09
ChallengesChallenges
Improve Quality Services BV 4
every 6 months (Nokia)
� # Defects hardly decreases�Defect density per KLOC is almost constant in the
last 10 years (Les Hatton)
� Deadlines, Competition & Outsourcing
� Testing often takes 30 - 40% of project costs
PeoplePeople
Test Improvement DirectionsTest Improvement Directions
Improve Quality Services BV 5
Test ProcessTest Process
People SkillsPeople Skills
Test knowledgeTest knowledge-- Test principlesTest principles
-- TechniquesTechniques
IT knowledgeIT knowledge-- ScriptingScripting
-- Requirements Requirements (IREB)(IREB)
via (exchanging) via (exchanging) Practical Practical ExperiencesExperiences, , CoachingCoaching
and (formal) and (formal) TrainingTraining
Improve Quality Services BV 6
-- TechniquesTechniques
-- Tools, etc.Tools, etc.
-- Requirements Requirements (IREB)(IREB)
-- Configuration mgt.Configuration mgt.
DomainDomainknowledgeknowledge-- Business processBusiness process
-- User characteristicsUser characteristics
Soft skillsSoft skills-- CommunicationCommunication
-- Critical mindsetCritical mindset
-- Presentation & reportingPresentation & reporting
CCMMMM((II)) MMaattuurriittyy
lleevveell
DDeessiiggnn FFaauullttss // KKLLOOCC
((KKeeeennee))
DDeelliivveerreedd DDeeffeeccttss // FFPP
((JJoonneess))
SShhiippppeedd DDeeffeeccttss // KKLLOOCC
((KKrraassnneerr))
RReellaattiivvee DDeeffeecctt DDeennssiittyy
((WWiilllliiaammss))
SShhiippppeedd DDeeffeeccttss
((RRiiffkkiinn))
55 00,,55 00,,0055 00,,55 00,,0055 11
44 11 00,,1144 22,,55 00,,11 55
Trends in Software Trends in Software QualityQuality
Improve Quality Services BV 7
44 11 00,,1144 22,,55 00,,11 55
33 22 00,,2277 33,,55 00,,22 77
22 33 00,,4444 66 00,,44 1122
11 55--66 00,,7755 3300 11,,00 6611
Readiness for Test ImprovementReadiness for Test Improvement
�� Some fundamental practices availableSome fundamental practices available��Requirements documented and managedRequirements documented and managed
��Project planning existsProject planning exists
��Configuration management Configuration management
Improve Quality Services BV 8
��Configuration management Configuration management
�� ManagementManagement “needs” / supports better testing“needs” / supports better testing��Quality is important enoughQuality is important enough
�� A budget for process improvementA budget for process improvement
��or reverse quality push……..!!or reverse quality push……..!!
TMMi modelTMMi model
�� Test Maturity Model (TMM) FrameworkTest Maturity Model (TMM) Framework��IlinoisIlinois Institute of Technology Institute of Technology -- CMM basedCMM based
�� TMMi SourcesTMMi Sources��CMMI, ISTQB, TMM, IEEE, TPICMMI, ISTQB, TMM, IEEE, TPI
BB
Improve Quality Services BV 9
��CMMI, ISTQB, TMM, IEEE, TPICMMI, ISTQB, TMM, IEEE, TPI
�� Starting pointsStarting points��Staged representation, CMMI structureStaged representation, CMMI structure
��Independent or in combination with CMMIIndependent or in combination with CMMI
��TMMiTMMi V3.1 released November 2010V3.1 released November 2010
AA
TMMi process areas by levelTMMi process areas by level5 : Optimization5 : Optimization-- Test process optimizationTest process optimization-- Quality controlQuality control-- Defect preventionDefect prevention
Helps to set priorities Helps to set priorities and focus the testand focus the testimprovement processimprovement process
Helps to set priorities Helps to set priorities and focus the testand focus the testimprovement processimprovement process
4 : Management and measurement4 : Management and measurement-- Product quality evaluationProduct quality evaluation-- Test measurement programmeTest measurement programme-- Advanced reviewsAdvanced reviews
Improve Quality Services BV 10
3 : Integration3 : Integration-- Test organizationTest organization-- Test training programTest training program-- Test life cycle and integrationTest life cycle and integration-- NonNon--Functional testingFunctional testing-- Peer ReviewsPeer Reviews
2 : Managed2 : Managed-- Test policy and strategyTest policy and strategy-- Test planningTest planning-- Test monitoring and controlTest monitoring and control-- Test design and executionTest design and execution-- Test environmentTest environment
± 2 years
± 1 year
improvement processimprovement processimprovement processimprovement process -- Advanced reviewsAdvanced reviews
CertificationCertification
�� Requirements defined regardingRequirements defined regarding
��Assessment requirements (TAMAR) Assessment requirements (TAMAR) based on ARC (B/C) and ISO 15504based on ARC (B/C) and ISO 15504
��LeadLead--Assessor and Assessor Assessor and Assessor
Improve Quality Services BV 11
��LeadLead--Assessor and Assessor Assessor and Assessor
�� Organizations and persons to be accredited by the Organizations and persons to be accredited by the TMMi FoundationTMMi Foundation
�� Register of (lead) assessorsRegister of (lead) assessors
��Formal TMMi Certification possibleFormal TMMi Certification possible
Lead Assessor / AssessorLead Assessor / AssessorLead assessorLead assessor
(formal assessments)(formal assessments)
AssessorAssessor
(informal assessments)(informal assessments)
TestingTesting > 5 years> 5 years
Multiple levels & domainsMultiple levels & domains
ISTQB AdvancedISTQB Advanced
> 3 years> 3 years
Multiple levels & domainsMultiple levels & domains
ISTQB FoundationISTQB Foundation
Improve Quality Services BV 12
Test Process Test Process ImprovementImprovement
> 2 years> 2 years
(2 years SPI = 1 year TPI)(2 years SPI = 1 year TPI)
> 1 years> 1 years
TMMiTMMi TrainingTraining
Practical experiencePractical experience
TrainingTraining
Practical experiencePractical experience
AssessmentsAssessments Training Training
20 days practical experience20 days practical experience
Training Training
10 days practical experience10 days practical experience
Level 2 : ManagedLevel 2 : Managed
�� Test Policy and StrategyTest Policy and Strategy
�� Test PlanningTest Planning
�� Test Monitoring and ControlTest Monitoring and Control
Improve Quality Services BV 13
�� Test Monitoring and ControlTest Monitoring and Control
�� Test Design and ExecutionTest Design and Execution
�� Test EnvironmentTest Environment
Ensures CMMI complianceEnsures CMMI complianceVerification & ValidationVerification & Validation
TMMi level 2TMMi level 2Lead timeLead time
reduction !!reduction !!
to provide visibility regarding the quality and outstanding risks of software productsto provide visibility regarding the quality and outstanding risks of software products
to contribute to the software product quality by finding defectsto contribute to the software product quality by finding defects
to contribute to customer satifaction regarding the released software productsto contribute to customer satifaction regarding the released software products
….….
Test Policy 20XX
.
Mission / Ambition
Definition Testingbased on
business policy
Improve Quality Services BV 14
Test targets
Employees
Organization
Test process
Customers
Test Process ImprovementTest Process Improvement-- objectives, performance indicators, modelobjectives, performance indicators, model
Example Performance IndicatorExample Performance Indicator
15
20
25
30
35
40Improvement focusImprovement focus
Improve Quality Services BV 15
0
5
10
15
w eek 1-10 w eek 11-23 w eek 24-36
Development.percentage of TOTAL FAT.percentage of TOTAL
UAT.Percentage of TOTAL Production.Percentage of TOTAL
“this is what we are doing !!”“this is what we are doing !!”
Level 3 : IntegrationLevel 3 : Integration
�� Test OrganizationTest Organization
�� Test Training ProgramTest Training Program
�� Test Life cycle and IntegrationTest Life cycle and Integration
InstitutionalizationInstitutionalization
Improve Quality Services BV 16
16
�� Test Life cycle and IntegrationTest Life cycle and Integration
�� NonNon--Functional TestingFunctional Testing
�� Peer ReviewsPeer Reviews
reqreq gdgd dddd implimpl testtest operoper
Life cycleLife cycle
Req.Design
Improve Quality Services BV 17
Code
TestTest planTest design
Test script
Execution
Test ImprovementsTest Improvements
�� Testing as a profession Testing as a profession
�� Clear responsibilitiesClear responsibilities
�� Test plan(ning) Test plan(ning)
�� RiskRisk--based testing based testing
Risk Matrix ISR 2.2
0
8
15
0 25 50
probability
Impa
ct
Q/R
US/MR
ECG
S/R
Vasc
USVW
Lic
Ptrac
Log
Srch
Impd
Merge
DB
DCMlib
Dbkill
Vwregr
Top PriorityTop Priority
engineeringview
PRISMA®
Improve Quality Services BV 18
�� RiskRisk--based testing based testing
�� Testing part of development life cycleTesting part of development life cycle
�� Early involvement Early involvement
�� Test design techniquesTest design techniques
�� Controlled test processControlled test process
view
Reported ResultsReported Results
5
10
15
20alpha / beta test lead time
556065707580859095
100Defect Detection Percentage
Improve Quality Services BV 19
0Y1 Y2 Y3 Y4
0
20
40
60
80
100
120
140
160
% deviation - spent vs. planned
5055
Y1 Y2 Y3
TMMi 1
TMMi 2
TMMi 3
Employee satisfaction
“Getting Started”“Getting Started”
��Test policyTest policy
��Management commitmentManagement commitment
�� Improvement is a projectImprovement is a project
1
Improve Quality Services BV 20
�� Improvement is a projectImprovement is a project
��Resources and effort percentageResources and effort percentage
��Maturity of development organizationMaturity of development organization
2
3
“Getting the job done”“Getting the job done”
�� Set both long term and short term goalsSet both long term and short term goals
�� Use what is availableUse what is available
�� Don’t make an external consultant Don’t make an external consultant responsibleresponsible
Improve Quality Services BV 21
responsibleresponsible
�� Institutionalize the improvementsInstitutionalize the improvements
�� Change management skillsChange management skills
�� Review against the overall objectives Review against the overall objectives (policy) almost continuously(policy) almost continuously
Test Improvement ManifestoTest Improvement Manifesto
•• FlexibilityFlexibility over Detailed Processes
•• Best PracticesBest Practices over Templates
•• Deployment orientationDeployment orientation over Process
Making TMMi work ….Making TMMi work ….
Improve Quality Services BV 22
•• Deployment orientationDeployment orientation over Process orientation
•• ReviewsReviews over Quality Assurance (departments)
•• Business drivenBusiness driven over Model driven
TMMi Supporting OrganizationsTMMi Supporting Organizations
Improve Quality Services BV 23
More information on TMMiavailable at www.TMMiFoundation.org