Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
Listen | Challenge | Understand | Interpret | Create
Private amp Confidential Experimentus Ltd 17a Dorset Square London NW1 6QB T +44 (0)207 871 2300 wwwexperimentuscom
Test Process Improvement Measurement is Critical toSuccess
Presented byBrian Wells
UKSMA 22nd Conference London27-28 October 2010
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus Ltd2
Need to Improve
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Need to focus on quality
3
sect Projects need to balance delivery of quality on time and to budgetsect Historical delivery focuses on cost and timescale alone - often results in
increased costs and produces lower quality outputndash Projects invariably being over budget and behind schedulendash Evidence of increased costs in production support - the major hidden cost of
software development
sect Quality requires focus on the ldquoDo it once and do it right principlesect Focus on quality to achieve three project objectives will achieve controlled
costs controlled timescales and higher product quality
This is progress
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The test process is critical
4
sect Testing demonstrates levels of quality and by finding and removing defectswill improve quality
sect Any activity that identifies issues is ldquotestingrdquo in its fullest sensendash All types of testing activities consumes approximately 50+ of the IT budget
sect If you are able to improve the quality of the test process and the product theprocess deliversndash You will reduce delivery and support costsndash You will deliver on or before timendash More chance to deliver what the client wants
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Evaluate your capability
5
sect Start byndash Identifying where you are today ndash strengths and opportunitiesndash Identify where you need to go (and why)
sect TMMi model isndash The emerging global standard test process reference modelndash The ldquobest of breedrdquo taken from existing models standards and practitionersndash A model that includes the test industry recognised good practicesndash Generic ndashapplicable to all development and delivery models including industry
sectorsndash Specific ndashcovers all required activities and deliverables expected of a good quality
test process
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
TMMi Output
6
sect A TMMi Assessment will producendash A structured identification of strengths within the test processndash A structured definition of process improvement requirements
sect Any process improvement requirements indicatendash Where there is a poor quality process elementndash A potential for poor product quality being delivered
sect The (optional) ability to calculate potential Return on Investment of processimprovements to the business to demonstrate the value that is achievable
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus Ltd7
Approaching ProcessImprovement
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
What is test process improvement
8
Continuous improvement of the quality and the efficiency of the testingprocess in the context of the whole software development life cycle
The ISTQB glossary seems to have got its focus wrong
when it defines test process improvement as
A continuous framework for test process improvement that describesthe key elements of an effective test process especially targeted at
system testing and acceptance testing
A continuous framework for test process improvement that describesthe key elements of an effective test process especially targeted at
system testing and acceptance testing
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The process improvement process
9
PrioritisePlan
InitiateMeasure
DefineRedefineOperateValidateEvolve
M
RP
O
EV
Confirm the objective choose appropriate model
List the possible process improvements
Undertake the assessment
Createamend the processes
Process improvements are deployed
Benefits are realised and success criteria met
Stopstart the next stage of improvement
I
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Measure
Prioritiseamp Plan
Define Re-define
Operate
Validate
Evolve
Methodology - IMPROVE
10
Initiate
Organisation
Process Technology
bull Action Planningbull Target Setting
bull New processdefinitionrefinement
bull Trainingbull Pilotbull Full deployment
bull Monitoring of resultsagainst targets
bull Evolution of programmeapproach amp targets
bull Assess Processmonitoragainst plan
bull Obtain agreementto progress
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The business case
11
sect To define objective and prioritiessect Need to define the benefits ndash donrsquot over-exaggerate
ndash Intangiblendash Tangible
sect Understand the costs involvedsect Timescalessect Quick wins
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Intangible benefits
12
sect Clearer and well understood processessect Repeatabilitysect Reduction in key person dependency ndash single point of failuresect Measurement of progresssect Measurement of processsect Clear roadmap of what to do nextsect Better career opportunitiessect Happier staff
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Financial benefits
1313
PIPID
PIP Summary EstimatedPersonDay
EstimatedCost
PotentialBenefit
Net PotentialBenefit
1 Institutionalisation and mandating of generic(consolidated) Test Framework elements 48 48000 214290 166290
2 Define approach process procedures templatestraining guidance for Risk Based Prioritisation 15 15000 114097 99097
3Document the structured procedures templatestraining tools to undertake informal andstructured reviews 15 15000 586170 571170
4 Consolidate elements into comprehensiveMonitoring amp Control Process definition 24 24000 196065 172065
5Evaluate and define Test Standard specific suiteof training modules and make available viaLearning Zone 15 15000 0 -15000
117000 1110622 99362295 105509150 175500
879591
TOTAL NET POTENTIAL BENEFITSConfidence Factor
Implementation FactorADJUSTED NET POTENTIAL BENEFITS
Potential Benefits and Implementation Estimates Summary
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Define-redefine
14
sect Based on the process improvements identified and agreed - eitherndash define any new methods processes and templatesOrndash where existing processes exist but need updating they are re-defined to bring
them up to date
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Operate
15
sect This is the rollout phasesect Will be at least 80 of the improvement project
sect Agree the improvements to be rolled outndash pilot project or piece of workndash with the support team to see if they have any questionsndash feedback to the process
sect Provide trainingsect Provide mentoring
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Validate
16
sect Benefits realisation stage as well as refinement stagesect Measurement of benefits ndash depends on having a good understanding of the
start positionsect Both tangible and intangiblesect Have the improvement projectrsquos success or exit criteria been metsect Close of the formal improvement project
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Assess and developImprovements
Implementation
17
ImplementationBAU
review
Initiate ndash Measure ndashPrioritise Plan and
DefineIMP Redefine
OperateRO ValidateEvolveVE
Ent
husi
asm
MaximumHype
Actual UseDisillusionment
Practical Benefit
RIP
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus Ltd18
To Measure
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
How to approach measuring
19
sect Agree what message needs to be deliveredsect Keep It Simple Short (KISS) relevant and easy to capture the informationsect Use Goal Question Measure (GQM) approach
ndash Add scales and thresholds
sect Define a data catalogue owners repositoriessect Define the collection and analysis proceduressect Define the presentation and reporting requirements
ndash Produce regular output reports to demonstrate progress ndash give explanations toresults to ensure the right message is delivered
ndash Consistent interpretation
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus Ltd20
Working in Practice
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The Organisation
21
sect Major testing services provider located in Europesect 90 of employees are test professionals (qualified experienced dedicated to
test activities)sect Deliver contract-driven specific test services to all major industry sectorssect Most services deliver system and system integration test levels
ndash 80+ contracts delivering services via a Test Factoryndash They are not usually involved in other parts of test and delivery life cycle(s)ndash Internal ethos of ldquojust do itrdquo instead of ldquomonitor and learnrdquondash A formal restrained communication and information exchange between
organisation and clients
sect The strategic and organisational objective was to achieve TMMi Level 3certification
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The Road Map
22
sect TMMi Baseline assessment undertaken to Level 3 in March 2010ndash Numerous process gaps evidenced in all process areas for both levels 2 amp 3
sect Process Improvement project initiatedndash Involved senior test and quality management resources to demonstrate
commitment to the rest of the organisationndash Aggressive timescales to define and implement improvements were setndash Assessment provider undertook quality assurance and TMMi progress checkpoint
activities
sect Robust approach in line with the IMPROVE modelsect An implementation plan ensured the organisation had implemented in time to
be re-assessed in December 2010ndash TMMi Level 3 certification achieved with no issues
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Oh to measure
23
sect Strategic objective only to achieve certificationndash No stated requirement to measure before and after implementation of changendash Measure only because TMMi says so
sect At March 2010 only 2 Key Performance Indicators were definedndash Service Level Agreements stated ldquo must deliver a) on time and b) on budgetrdquondash No declared interest in process or product quality
sect No requirement to identify cost of process weaknessessect Initial assessment data identified product quality ldquoKPIrdquo as well
ndash Organisation is not paid if ANY severity 1 issues found by client that SHOULD havebeen found before
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Measurement as implemented
24
sect Using GQM defined 6 internal KPIs with thresholdsndash Requirements Coverage ndash requirements planned to be tested amp successfully testedndash Quality of Incidents found ndash what level of duplication rejection was evidencedndash Incidents per Test Cases ndash good test cases finding incidentsndash Incident detection trend ndash when in the life cycle does testing find the incidentsndash Actual vs Planned test effort and deviation levelndash Test effort per test case
sect Added customer satisfaction datasect Retained on time and budget contractual SLArsquossect Integrated into test monitoring control and reporting within delivery projects
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Andhellip
25
sect The organisation achieved certification on timesect They also used the KPI data to observe that in the first 6 months of 2011
ndash Test case specificationsect Efficiency had increased 8 between January and June
sect Effectiveness had increased 11 between January and June
ndash Test case executionsect Efficiency had increased 5 between January and June
sect Effectiveness had increased 5 between January and June
ndash Quality of test cases had increased 9ndash Quality of issues detected had increased 6They also think that client relationships have become more open and positive and thepost delivery ldquonegotiationsrdquo are much easier
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
And the message is
26
sect With no requirement for measurement beyond the needs of a processreference modelndash A few well defined easy to understand KPIsndash Implemented collected analysed and reported quicklyndash Demonstrated to all parties the improvements that were accruing following
implemented change
It is good to measure (but make it easy to do so)
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd27
Thank you
Brian Wellsinfoexperimentuscom
+44(0)207 871 2300
wwwexperimentuscom
wwwtmmifoundationorg
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Who owns TMMireg
sect The TMMi Foundation
ndash Maintains ownership of the TMMireg model and the management of the ongoingintellectual property rights
ndash Creation and management of an independent central data repository and provisionof industry and other analysis services
ndash Provision of an independent accreditation process for TMMireg assessment methodsand Assessor resources
ndash Provision of a public forum of interested parties
sect For more information visit wwwtmmifoundationorg
28
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
TMMireg Foundation Contacts
29
sect Full information on the work of the Foundation can be found onwwwtmmifoundationorg
sect If interested in becoming a member of the Foundation (currently freeregistration) please email your details to infotmmifoundationorg
sect If your organisation is interested in supporting the work of the Foundationplease email sponsorshiptmmifoundationorg
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus Ltd2
Need to Improve
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Need to focus on quality
3
sect Projects need to balance delivery of quality on time and to budgetsect Historical delivery focuses on cost and timescale alone - often results in
increased costs and produces lower quality outputndash Projects invariably being over budget and behind schedulendash Evidence of increased costs in production support - the major hidden cost of
software development
sect Quality requires focus on the ldquoDo it once and do it right principlesect Focus on quality to achieve three project objectives will achieve controlled
costs controlled timescales and higher product quality
This is progress
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The test process is critical
4
sect Testing demonstrates levels of quality and by finding and removing defectswill improve quality
sect Any activity that identifies issues is ldquotestingrdquo in its fullest sensendash All types of testing activities consumes approximately 50+ of the IT budget
sect If you are able to improve the quality of the test process and the product theprocess deliversndash You will reduce delivery and support costsndash You will deliver on or before timendash More chance to deliver what the client wants
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Evaluate your capability
5
sect Start byndash Identifying where you are today ndash strengths and opportunitiesndash Identify where you need to go (and why)
sect TMMi model isndash The emerging global standard test process reference modelndash The ldquobest of breedrdquo taken from existing models standards and practitionersndash A model that includes the test industry recognised good practicesndash Generic ndashapplicable to all development and delivery models including industry
sectorsndash Specific ndashcovers all required activities and deliverables expected of a good quality
test process
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
TMMi Output
6
sect A TMMi Assessment will producendash A structured identification of strengths within the test processndash A structured definition of process improvement requirements
sect Any process improvement requirements indicatendash Where there is a poor quality process elementndash A potential for poor product quality being delivered
sect The (optional) ability to calculate potential Return on Investment of processimprovements to the business to demonstrate the value that is achievable
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus Ltd7
Approaching ProcessImprovement
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
What is test process improvement
8
Continuous improvement of the quality and the efficiency of the testingprocess in the context of the whole software development life cycle
The ISTQB glossary seems to have got its focus wrong
when it defines test process improvement as
A continuous framework for test process improvement that describesthe key elements of an effective test process especially targeted at
system testing and acceptance testing
A continuous framework for test process improvement that describesthe key elements of an effective test process especially targeted at
system testing and acceptance testing
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The process improvement process
9
PrioritisePlan
InitiateMeasure
DefineRedefineOperateValidateEvolve
M
RP
O
EV
Confirm the objective choose appropriate model
List the possible process improvements
Undertake the assessment
Createamend the processes
Process improvements are deployed
Benefits are realised and success criteria met
Stopstart the next stage of improvement
I
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Measure
Prioritiseamp Plan
Define Re-define
Operate
Validate
Evolve
Methodology - IMPROVE
10
Initiate
Organisation
Process Technology
bull Action Planningbull Target Setting
bull New processdefinitionrefinement
bull Trainingbull Pilotbull Full deployment
bull Monitoring of resultsagainst targets
bull Evolution of programmeapproach amp targets
bull Assess Processmonitoragainst plan
bull Obtain agreementto progress
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The business case
11
sect To define objective and prioritiessect Need to define the benefits ndash donrsquot over-exaggerate
ndash Intangiblendash Tangible
sect Understand the costs involvedsect Timescalessect Quick wins
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Intangible benefits
12
sect Clearer and well understood processessect Repeatabilitysect Reduction in key person dependency ndash single point of failuresect Measurement of progresssect Measurement of processsect Clear roadmap of what to do nextsect Better career opportunitiessect Happier staff
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Financial benefits
1313
PIPID
PIP Summary EstimatedPersonDay
EstimatedCost
PotentialBenefit
Net PotentialBenefit
1 Institutionalisation and mandating of generic(consolidated) Test Framework elements 48 48000 214290 166290
2 Define approach process procedures templatestraining guidance for Risk Based Prioritisation 15 15000 114097 99097
3Document the structured procedures templatestraining tools to undertake informal andstructured reviews 15 15000 586170 571170
4 Consolidate elements into comprehensiveMonitoring amp Control Process definition 24 24000 196065 172065
5Evaluate and define Test Standard specific suiteof training modules and make available viaLearning Zone 15 15000 0 -15000
117000 1110622 99362295 105509150 175500
879591
TOTAL NET POTENTIAL BENEFITSConfidence Factor
Implementation FactorADJUSTED NET POTENTIAL BENEFITS
Potential Benefits and Implementation Estimates Summary
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Define-redefine
14
sect Based on the process improvements identified and agreed - eitherndash define any new methods processes and templatesOrndash where existing processes exist but need updating they are re-defined to bring
them up to date
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Operate
15
sect This is the rollout phasesect Will be at least 80 of the improvement project
sect Agree the improvements to be rolled outndash pilot project or piece of workndash with the support team to see if they have any questionsndash feedback to the process
sect Provide trainingsect Provide mentoring
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Validate
16
sect Benefits realisation stage as well as refinement stagesect Measurement of benefits ndash depends on having a good understanding of the
start positionsect Both tangible and intangiblesect Have the improvement projectrsquos success or exit criteria been metsect Close of the formal improvement project
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Assess and developImprovements
Implementation
17
ImplementationBAU
review
Initiate ndash Measure ndashPrioritise Plan and
DefineIMP Redefine
OperateRO ValidateEvolveVE
Ent
husi
asm
MaximumHype
Actual UseDisillusionment
Practical Benefit
RIP
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus Ltd18
To Measure
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
How to approach measuring
19
sect Agree what message needs to be deliveredsect Keep It Simple Short (KISS) relevant and easy to capture the informationsect Use Goal Question Measure (GQM) approach
ndash Add scales and thresholds
sect Define a data catalogue owners repositoriessect Define the collection and analysis proceduressect Define the presentation and reporting requirements
ndash Produce regular output reports to demonstrate progress ndash give explanations toresults to ensure the right message is delivered
ndash Consistent interpretation
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus Ltd20
Working in Practice
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The Organisation
21
sect Major testing services provider located in Europesect 90 of employees are test professionals (qualified experienced dedicated to
test activities)sect Deliver contract-driven specific test services to all major industry sectorssect Most services deliver system and system integration test levels
ndash 80+ contracts delivering services via a Test Factoryndash They are not usually involved in other parts of test and delivery life cycle(s)ndash Internal ethos of ldquojust do itrdquo instead of ldquomonitor and learnrdquondash A formal restrained communication and information exchange between
organisation and clients
sect The strategic and organisational objective was to achieve TMMi Level 3certification
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The Road Map
22
sect TMMi Baseline assessment undertaken to Level 3 in March 2010ndash Numerous process gaps evidenced in all process areas for both levels 2 amp 3
sect Process Improvement project initiatedndash Involved senior test and quality management resources to demonstrate
commitment to the rest of the organisationndash Aggressive timescales to define and implement improvements were setndash Assessment provider undertook quality assurance and TMMi progress checkpoint
activities
sect Robust approach in line with the IMPROVE modelsect An implementation plan ensured the organisation had implemented in time to
be re-assessed in December 2010ndash TMMi Level 3 certification achieved with no issues
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Oh to measure
23
sect Strategic objective only to achieve certificationndash No stated requirement to measure before and after implementation of changendash Measure only because TMMi says so
sect At March 2010 only 2 Key Performance Indicators were definedndash Service Level Agreements stated ldquo must deliver a) on time and b) on budgetrdquondash No declared interest in process or product quality
sect No requirement to identify cost of process weaknessessect Initial assessment data identified product quality ldquoKPIrdquo as well
ndash Organisation is not paid if ANY severity 1 issues found by client that SHOULD havebeen found before
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Measurement as implemented
24
sect Using GQM defined 6 internal KPIs with thresholdsndash Requirements Coverage ndash requirements planned to be tested amp successfully testedndash Quality of Incidents found ndash what level of duplication rejection was evidencedndash Incidents per Test Cases ndash good test cases finding incidentsndash Incident detection trend ndash when in the life cycle does testing find the incidentsndash Actual vs Planned test effort and deviation levelndash Test effort per test case
sect Added customer satisfaction datasect Retained on time and budget contractual SLArsquossect Integrated into test monitoring control and reporting within delivery projects
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Andhellip
25
sect The organisation achieved certification on timesect They also used the KPI data to observe that in the first 6 months of 2011
ndash Test case specificationsect Efficiency had increased 8 between January and June
sect Effectiveness had increased 11 between January and June
ndash Test case executionsect Efficiency had increased 5 between January and June
sect Effectiveness had increased 5 between January and June
ndash Quality of test cases had increased 9ndash Quality of issues detected had increased 6They also think that client relationships have become more open and positive and thepost delivery ldquonegotiationsrdquo are much easier
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
And the message is
26
sect With no requirement for measurement beyond the needs of a processreference modelndash A few well defined easy to understand KPIsndash Implemented collected analysed and reported quicklyndash Demonstrated to all parties the improvements that were accruing following
implemented change
It is good to measure (but make it easy to do so)
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd27
Thank you
Brian Wellsinfoexperimentuscom
+44(0)207 871 2300
wwwexperimentuscom
wwwtmmifoundationorg
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Who owns TMMireg
sect The TMMi Foundation
ndash Maintains ownership of the TMMireg model and the management of the ongoingintellectual property rights
ndash Creation and management of an independent central data repository and provisionof industry and other analysis services
ndash Provision of an independent accreditation process for TMMireg assessment methodsand Assessor resources
ndash Provision of a public forum of interested parties
sect For more information visit wwwtmmifoundationorg
28
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
TMMireg Foundation Contacts
29
sect Full information on the work of the Foundation can be found onwwwtmmifoundationorg
sect If interested in becoming a member of the Foundation (currently freeregistration) please email your details to infotmmifoundationorg
sect If your organisation is interested in supporting the work of the Foundationplease email sponsorshiptmmifoundationorg
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Need to focus on quality
3
sect Projects need to balance delivery of quality on time and to budgetsect Historical delivery focuses on cost and timescale alone - often results in
increased costs and produces lower quality outputndash Projects invariably being over budget and behind schedulendash Evidence of increased costs in production support - the major hidden cost of
software development
sect Quality requires focus on the ldquoDo it once and do it right principlesect Focus on quality to achieve three project objectives will achieve controlled
costs controlled timescales and higher product quality
This is progress
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The test process is critical
4
sect Testing demonstrates levels of quality and by finding and removing defectswill improve quality
sect Any activity that identifies issues is ldquotestingrdquo in its fullest sensendash All types of testing activities consumes approximately 50+ of the IT budget
sect If you are able to improve the quality of the test process and the product theprocess deliversndash You will reduce delivery and support costsndash You will deliver on or before timendash More chance to deliver what the client wants
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Evaluate your capability
5
sect Start byndash Identifying where you are today ndash strengths and opportunitiesndash Identify where you need to go (and why)
sect TMMi model isndash The emerging global standard test process reference modelndash The ldquobest of breedrdquo taken from existing models standards and practitionersndash A model that includes the test industry recognised good practicesndash Generic ndashapplicable to all development and delivery models including industry
sectorsndash Specific ndashcovers all required activities and deliverables expected of a good quality
test process
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
TMMi Output
6
sect A TMMi Assessment will producendash A structured identification of strengths within the test processndash A structured definition of process improvement requirements
sect Any process improvement requirements indicatendash Where there is a poor quality process elementndash A potential for poor product quality being delivered
sect The (optional) ability to calculate potential Return on Investment of processimprovements to the business to demonstrate the value that is achievable
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus Ltd7
Approaching ProcessImprovement
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
What is test process improvement
8
Continuous improvement of the quality and the efficiency of the testingprocess in the context of the whole software development life cycle
The ISTQB glossary seems to have got its focus wrong
when it defines test process improvement as
A continuous framework for test process improvement that describesthe key elements of an effective test process especially targeted at
system testing and acceptance testing
A continuous framework for test process improvement that describesthe key elements of an effective test process especially targeted at
system testing and acceptance testing
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The process improvement process
9
PrioritisePlan
InitiateMeasure
DefineRedefineOperateValidateEvolve
M
RP
O
EV
Confirm the objective choose appropriate model
List the possible process improvements
Undertake the assessment
Createamend the processes
Process improvements are deployed
Benefits are realised and success criteria met
Stopstart the next stage of improvement
I
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Measure
Prioritiseamp Plan
Define Re-define
Operate
Validate
Evolve
Methodology - IMPROVE
10
Initiate
Organisation
Process Technology
bull Action Planningbull Target Setting
bull New processdefinitionrefinement
bull Trainingbull Pilotbull Full deployment
bull Monitoring of resultsagainst targets
bull Evolution of programmeapproach amp targets
bull Assess Processmonitoragainst plan
bull Obtain agreementto progress
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The business case
11
sect To define objective and prioritiessect Need to define the benefits ndash donrsquot over-exaggerate
ndash Intangiblendash Tangible
sect Understand the costs involvedsect Timescalessect Quick wins
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Intangible benefits
12
sect Clearer and well understood processessect Repeatabilitysect Reduction in key person dependency ndash single point of failuresect Measurement of progresssect Measurement of processsect Clear roadmap of what to do nextsect Better career opportunitiessect Happier staff
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Financial benefits
1313
PIPID
PIP Summary EstimatedPersonDay
EstimatedCost
PotentialBenefit
Net PotentialBenefit
1 Institutionalisation and mandating of generic(consolidated) Test Framework elements 48 48000 214290 166290
2 Define approach process procedures templatestraining guidance for Risk Based Prioritisation 15 15000 114097 99097
3Document the structured procedures templatestraining tools to undertake informal andstructured reviews 15 15000 586170 571170
4 Consolidate elements into comprehensiveMonitoring amp Control Process definition 24 24000 196065 172065
5Evaluate and define Test Standard specific suiteof training modules and make available viaLearning Zone 15 15000 0 -15000
117000 1110622 99362295 105509150 175500
879591
TOTAL NET POTENTIAL BENEFITSConfidence Factor
Implementation FactorADJUSTED NET POTENTIAL BENEFITS
Potential Benefits and Implementation Estimates Summary
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Define-redefine
14
sect Based on the process improvements identified and agreed - eitherndash define any new methods processes and templatesOrndash where existing processes exist but need updating they are re-defined to bring
them up to date
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Operate
15
sect This is the rollout phasesect Will be at least 80 of the improvement project
sect Agree the improvements to be rolled outndash pilot project or piece of workndash with the support team to see if they have any questionsndash feedback to the process
sect Provide trainingsect Provide mentoring
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Validate
16
sect Benefits realisation stage as well as refinement stagesect Measurement of benefits ndash depends on having a good understanding of the
start positionsect Both tangible and intangiblesect Have the improvement projectrsquos success or exit criteria been metsect Close of the formal improvement project
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Assess and developImprovements
Implementation
17
ImplementationBAU
review
Initiate ndash Measure ndashPrioritise Plan and
DefineIMP Redefine
OperateRO ValidateEvolveVE
Ent
husi
asm
MaximumHype
Actual UseDisillusionment
Practical Benefit
RIP
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus Ltd18
To Measure
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
How to approach measuring
19
sect Agree what message needs to be deliveredsect Keep It Simple Short (KISS) relevant and easy to capture the informationsect Use Goal Question Measure (GQM) approach
ndash Add scales and thresholds
sect Define a data catalogue owners repositoriessect Define the collection and analysis proceduressect Define the presentation and reporting requirements
ndash Produce regular output reports to demonstrate progress ndash give explanations toresults to ensure the right message is delivered
ndash Consistent interpretation
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus Ltd20
Working in Practice
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The Organisation
21
sect Major testing services provider located in Europesect 90 of employees are test professionals (qualified experienced dedicated to
test activities)sect Deliver contract-driven specific test services to all major industry sectorssect Most services deliver system and system integration test levels
ndash 80+ contracts delivering services via a Test Factoryndash They are not usually involved in other parts of test and delivery life cycle(s)ndash Internal ethos of ldquojust do itrdquo instead of ldquomonitor and learnrdquondash A formal restrained communication and information exchange between
organisation and clients
sect The strategic and organisational objective was to achieve TMMi Level 3certification
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The Road Map
22
sect TMMi Baseline assessment undertaken to Level 3 in March 2010ndash Numerous process gaps evidenced in all process areas for both levels 2 amp 3
sect Process Improvement project initiatedndash Involved senior test and quality management resources to demonstrate
commitment to the rest of the organisationndash Aggressive timescales to define and implement improvements were setndash Assessment provider undertook quality assurance and TMMi progress checkpoint
activities
sect Robust approach in line with the IMPROVE modelsect An implementation plan ensured the organisation had implemented in time to
be re-assessed in December 2010ndash TMMi Level 3 certification achieved with no issues
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Oh to measure
23
sect Strategic objective only to achieve certificationndash No stated requirement to measure before and after implementation of changendash Measure only because TMMi says so
sect At March 2010 only 2 Key Performance Indicators were definedndash Service Level Agreements stated ldquo must deliver a) on time and b) on budgetrdquondash No declared interest in process or product quality
sect No requirement to identify cost of process weaknessessect Initial assessment data identified product quality ldquoKPIrdquo as well
ndash Organisation is not paid if ANY severity 1 issues found by client that SHOULD havebeen found before
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Measurement as implemented
24
sect Using GQM defined 6 internal KPIs with thresholdsndash Requirements Coverage ndash requirements planned to be tested amp successfully testedndash Quality of Incidents found ndash what level of duplication rejection was evidencedndash Incidents per Test Cases ndash good test cases finding incidentsndash Incident detection trend ndash when in the life cycle does testing find the incidentsndash Actual vs Planned test effort and deviation levelndash Test effort per test case
sect Added customer satisfaction datasect Retained on time and budget contractual SLArsquossect Integrated into test monitoring control and reporting within delivery projects
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Andhellip
25
sect The organisation achieved certification on timesect They also used the KPI data to observe that in the first 6 months of 2011
ndash Test case specificationsect Efficiency had increased 8 between January and June
sect Effectiveness had increased 11 between January and June
ndash Test case executionsect Efficiency had increased 5 between January and June
sect Effectiveness had increased 5 between January and June
ndash Quality of test cases had increased 9ndash Quality of issues detected had increased 6They also think that client relationships have become more open and positive and thepost delivery ldquonegotiationsrdquo are much easier
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
And the message is
26
sect With no requirement for measurement beyond the needs of a processreference modelndash A few well defined easy to understand KPIsndash Implemented collected analysed and reported quicklyndash Demonstrated to all parties the improvements that were accruing following
implemented change
It is good to measure (but make it easy to do so)
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd27
Thank you
Brian Wellsinfoexperimentuscom
+44(0)207 871 2300
wwwexperimentuscom
wwwtmmifoundationorg
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Who owns TMMireg
sect The TMMi Foundation
ndash Maintains ownership of the TMMireg model and the management of the ongoingintellectual property rights
ndash Creation and management of an independent central data repository and provisionof industry and other analysis services
ndash Provision of an independent accreditation process for TMMireg assessment methodsand Assessor resources
ndash Provision of a public forum of interested parties
sect For more information visit wwwtmmifoundationorg
28
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
TMMireg Foundation Contacts
29
sect Full information on the work of the Foundation can be found onwwwtmmifoundationorg
sect If interested in becoming a member of the Foundation (currently freeregistration) please email your details to infotmmifoundationorg
sect If your organisation is interested in supporting the work of the Foundationplease email sponsorshiptmmifoundationorg
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The test process is critical
4
sect Testing demonstrates levels of quality and by finding and removing defectswill improve quality
sect Any activity that identifies issues is ldquotestingrdquo in its fullest sensendash All types of testing activities consumes approximately 50+ of the IT budget
sect If you are able to improve the quality of the test process and the product theprocess deliversndash You will reduce delivery and support costsndash You will deliver on or before timendash More chance to deliver what the client wants
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Evaluate your capability
5
sect Start byndash Identifying where you are today ndash strengths and opportunitiesndash Identify where you need to go (and why)
sect TMMi model isndash The emerging global standard test process reference modelndash The ldquobest of breedrdquo taken from existing models standards and practitionersndash A model that includes the test industry recognised good practicesndash Generic ndashapplicable to all development and delivery models including industry
sectorsndash Specific ndashcovers all required activities and deliverables expected of a good quality
test process
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
TMMi Output
6
sect A TMMi Assessment will producendash A structured identification of strengths within the test processndash A structured definition of process improvement requirements
sect Any process improvement requirements indicatendash Where there is a poor quality process elementndash A potential for poor product quality being delivered
sect The (optional) ability to calculate potential Return on Investment of processimprovements to the business to demonstrate the value that is achievable
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus Ltd7
Approaching ProcessImprovement
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
What is test process improvement
8
Continuous improvement of the quality and the efficiency of the testingprocess in the context of the whole software development life cycle
The ISTQB glossary seems to have got its focus wrong
when it defines test process improvement as
A continuous framework for test process improvement that describesthe key elements of an effective test process especially targeted at
system testing and acceptance testing
A continuous framework for test process improvement that describesthe key elements of an effective test process especially targeted at
system testing and acceptance testing
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The process improvement process
9
PrioritisePlan
InitiateMeasure
DefineRedefineOperateValidateEvolve
M
RP
O
EV
Confirm the objective choose appropriate model
List the possible process improvements
Undertake the assessment
Createamend the processes
Process improvements are deployed
Benefits are realised and success criteria met
Stopstart the next stage of improvement
I
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Measure
Prioritiseamp Plan
Define Re-define
Operate
Validate
Evolve
Methodology - IMPROVE
10
Initiate
Organisation
Process Technology
bull Action Planningbull Target Setting
bull New processdefinitionrefinement
bull Trainingbull Pilotbull Full deployment
bull Monitoring of resultsagainst targets
bull Evolution of programmeapproach amp targets
bull Assess Processmonitoragainst plan
bull Obtain agreementto progress
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The business case
11
sect To define objective and prioritiessect Need to define the benefits ndash donrsquot over-exaggerate
ndash Intangiblendash Tangible
sect Understand the costs involvedsect Timescalessect Quick wins
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Intangible benefits
12
sect Clearer and well understood processessect Repeatabilitysect Reduction in key person dependency ndash single point of failuresect Measurement of progresssect Measurement of processsect Clear roadmap of what to do nextsect Better career opportunitiessect Happier staff
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Financial benefits
1313
PIPID
PIP Summary EstimatedPersonDay
EstimatedCost
PotentialBenefit
Net PotentialBenefit
1 Institutionalisation and mandating of generic(consolidated) Test Framework elements 48 48000 214290 166290
2 Define approach process procedures templatestraining guidance for Risk Based Prioritisation 15 15000 114097 99097
3Document the structured procedures templatestraining tools to undertake informal andstructured reviews 15 15000 586170 571170
4 Consolidate elements into comprehensiveMonitoring amp Control Process definition 24 24000 196065 172065
5Evaluate and define Test Standard specific suiteof training modules and make available viaLearning Zone 15 15000 0 -15000
117000 1110622 99362295 105509150 175500
879591
TOTAL NET POTENTIAL BENEFITSConfidence Factor
Implementation FactorADJUSTED NET POTENTIAL BENEFITS
Potential Benefits and Implementation Estimates Summary
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Define-redefine
14
sect Based on the process improvements identified and agreed - eitherndash define any new methods processes and templatesOrndash where existing processes exist but need updating they are re-defined to bring
them up to date
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Operate
15
sect This is the rollout phasesect Will be at least 80 of the improvement project
sect Agree the improvements to be rolled outndash pilot project or piece of workndash with the support team to see if they have any questionsndash feedback to the process
sect Provide trainingsect Provide mentoring
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Validate
16
sect Benefits realisation stage as well as refinement stagesect Measurement of benefits ndash depends on having a good understanding of the
start positionsect Both tangible and intangiblesect Have the improvement projectrsquos success or exit criteria been metsect Close of the formal improvement project
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Assess and developImprovements
Implementation
17
ImplementationBAU
review
Initiate ndash Measure ndashPrioritise Plan and
DefineIMP Redefine
OperateRO ValidateEvolveVE
Ent
husi
asm
MaximumHype
Actual UseDisillusionment
Practical Benefit
RIP
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus Ltd18
To Measure
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
How to approach measuring
19
sect Agree what message needs to be deliveredsect Keep It Simple Short (KISS) relevant and easy to capture the informationsect Use Goal Question Measure (GQM) approach
ndash Add scales and thresholds
sect Define a data catalogue owners repositoriessect Define the collection and analysis proceduressect Define the presentation and reporting requirements
ndash Produce regular output reports to demonstrate progress ndash give explanations toresults to ensure the right message is delivered
ndash Consistent interpretation
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus Ltd20
Working in Practice
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The Organisation
21
sect Major testing services provider located in Europesect 90 of employees are test professionals (qualified experienced dedicated to
test activities)sect Deliver contract-driven specific test services to all major industry sectorssect Most services deliver system and system integration test levels
ndash 80+ contracts delivering services via a Test Factoryndash They are not usually involved in other parts of test and delivery life cycle(s)ndash Internal ethos of ldquojust do itrdquo instead of ldquomonitor and learnrdquondash A formal restrained communication and information exchange between
organisation and clients
sect The strategic and organisational objective was to achieve TMMi Level 3certification
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The Road Map
22
sect TMMi Baseline assessment undertaken to Level 3 in March 2010ndash Numerous process gaps evidenced in all process areas for both levels 2 amp 3
sect Process Improvement project initiatedndash Involved senior test and quality management resources to demonstrate
commitment to the rest of the organisationndash Aggressive timescales to define and implement improvements were setndash Assessment provider undertook quality assurance and TMMi progress checkpoint
activities
sect Robust approach in line with the IMPROVE modelsect An implementation plan ensured the organisation had implemented in time to
be re-assessed in December 2010ndash TMMi Level 3 certification achieved with no issues
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Oh to measure
23
sect Strategic objective only to achieve certificationndash No stated requirement to measure before and after implementation of changendash Measure only because TMMi says so
sect At March 2010 only 2 Key Performance Indicators were definedndash Service Level Agreements stated ldquo must deliver a) on time and b) on budgetrdquondash No declared interest in process or product quality
sect No requirement to identify cost of process weaknessessect Initial assessment data identified product quality ldquoKPIrdquo as well
ndash Organisation is not paid if ANY severity 1 issues found by client that SHOULD havebeen found before
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Measurement as implemented
24
sect Using GQM defined 6 internal KPIs with thresholdsndash Requirements Coverage ndash requirements planned to be tested amp successfully testedndash Quality of Incidents found ndash what level of duplication rejection was evidencedndash Incidents per Test Cases ndash good test cases finding incidentsndash Incident detection trend ndash when in the life cycle does testing find the incidentsndash Actual vs Planned test effort and deviation levelndash Test effort per test case
sect Added customer satisfaction datasect Retained on time and budget contractual SLArsquossect Integrated into test monitoring control and reporting within delivery projects
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Andhellip
25
sect The organisation achieved certification on timesect They also used the KPI data to observe that in the first 6 months of 2011
ndash Test case specificationsect Efficiency had increased 8 between January and June
sect Effectiveness had increased 11 between January and June
ndash Test case executionsect Efficiency had increased 5 between January and June
sect Effectiveness had increased 5 between January and June
ndash Quality of test cases had increased 9ndash Quality of issues detected had increased 6They also think that client relationships have become more open and positive and thepost delivery ldquonegotiationsrdquo are much easier
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
And the message is
26
sect With no requirement for measurement beyond the needs of a processreference modelndash A few well defined easy to understand KPIsndash Implemented collected analysed and reported quicklyndash Demonstrated to all parties the improvements that were accruing following
implemented change
It is good to measure (but make it easy to do so)
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd27
Thank you
Brian Wellsinfoexperimentuscom
+44(0)207 871 2300
wwwexperimentuscom
wwwtmmifoundationorg
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Who owns TMMireg
sect The TMMi Foundation
ndash Maintains ownership of the TMMireg model and the management of the ongoingintellectual property rights
ndash Creation and management of an independent central data repository and provisionof industry and other analysis services
ndash Provision of an independent accreditation process for TMMireg assessment methodsand Assessor resources
ndash Provision of a public forum of interested parties
sect For more information visit wwwtmmifoundationorg
28
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
TMMireg Foundation Contacts
29
sect Full information on the work of the Foundation can be found onwwwtmmifoundationorg
sect If interested in becoming a member of the Foundation (currently freeregistration) please email your details to infotmmifoundationorg
sect If your organisation is interested in supporting the work of the Foundationplease email sponsorshiptmmifoundationorg
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Evaluate your capability
5
sect Start byndash Identifying where you are today ndash strengths and opportunitiesndash Identify where you need to go (and why)
sect TMMi model isndash The emerging global standard test process reference modelndash The ldquobest of breedrdquo taken from existing models standards and practitionersndash A model that includes the test industry recognised good practicesndash Generic ndashapplicable to all development and delivery models including industry
sectorsndash Specific ndashcovers all required activities and deliverables expected of a good quality
test process
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
TMMi Output
6
sect A TMMi Assessment will producendash A structured identification of strengths within the test processndash A structured definition of process improvement requirements
sect Any process improvement requirements indicatendash Where there is a poor quality process elementndash A potential for poor product quality being delivered
sect The (optional) ability to calculate potential Return on Investment of processimprovements to the business to demonstrate the value that is achievable
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus Ltd7
Approaching ProcessImprovement
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
What is test process improvement
8
Continuous improvement of the quality and the efficiency of the testingprocess in the context of the whole software development life cycle
The ISTQB glossary seems to have got its focus wrong
when it defines test process improvement as
A continuous framework for test process improvement that describesthe key elements of an effective test process especially targeted at
system testing and acceptance testing
A continuous framework for test process improvement that describesthe key elements of an effective test process especially targeted at
system testing and acceptance testing
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The process improvement process
9
PrioritisePlan
InitiateMeasure
DefineRedefineOperateValidateEvolve
M
RP
O
EV
Confirm the objective choose appropriate model
List the possible process improvements
Undertake the assessment
Createamend the processes
Process improvements are deployed
Benefits are realised and success criteria met
Stopstart the next stage of improvement
I
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Measure
Prioritiseamp Plan
Define Re-define
Operate
Validate
Evolve
Methodology - IMPROVE
10
Initiate
Organisation
Process Technology
bull Action Planningbull Target Setting
bull New processdefinitionrefinement
bull Trainingbull Pilotbull Full deployment
bull Monitoring of resultsagainst targets
bull Evolution of programmeapproach amp targets
bull Assess Processmonitoragainst plan
bull Obtain agreementto progress
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The business case
11
sect To define objective and prioritiessect Need to define the benefits ndash donrsquot over-exaggerate
ndash Intangiblendash Tangible
sect Understand the costs involvedsect Timescalessect Quick wins
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Intangible benefits
12
sect Clearer and well understood processessect Repeatabilitysect Reduction in key person dependency ndash single point of failuresect Measurement of progresssect Measurement of processsect Clear roadmap of what to do nextsect Better career opportunitiessect Happier staff
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Financial benefits
1313
PIPID
PIP Summary EstimatedPersonDay
EstimatedCost
PotentialBenefit
Net PotentialBenefit
1 Institutionalisation and mandating of generic(consolidated) Test Framework elements 48 48000 214290 166290
2 Define approach process procedures templatestraining guidance for Risk Based Prioritisation 15 15000 114097 99097
3Document the structured procedures templatestraining tools to undertake informal andstructured reviews 15 15000 586170 571170
4 Consolidate elements into comprehensiveMonitoring amp Control Process definition 24 24000 196065 172065
5Evaluate and define Test Standard specific suiteof training modules and make available viaLearning Zone 15 15000 0 -15000
117000 1110622 99362295 105509150 175500
879591
TOTAL NET POTENTIAL BENEFITSConfidence Factor
Implementation FactorADJUSTED NET POTENTIAL BENEFITS
Potential Benefits and Implementation Estimates Summary
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Define-redefine
14
sect Based on the process improvements identified and agreed - eitherndash define any new methods processes and templatesOrndash where existing processes exist but need updating they are re-defined to bring
them up to date
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Operate
15
sect This is the rollout phasesect Will be at least 80 of the improvement project
sect Agree the improvements to be rolled outndash pilot project or piece of workndash with the support team to see if they have any questionsndash feedback to the process
sect Provide trainingsect Provide mentoring
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Validate
16
sect Benefits realisation stage as well as refinement stagesect Measurement of benefits ndash depends on having a good understanding of the
start positionsect Both tangible and intangiblesect Have the improvement projectrsquos success or exit criteria been metsect Close of the formal improvement project
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Assess and developImprovements
Implementation
17
ImplementationBAU
review
Initiate ndash Measure ndashPrioritise Plan and
DefineIMP Redefine
OperateRO ValidateEvolveVE
Ent
husi
asm
MaximumHype
Actual UseDisillusionment
Practical Benefit
RIP
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus Ltd18
To Measure
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
How to approach measuring
19
sect Agree what message needs to be deliveredsect Keep It Simple Short (KISS) relevant and easy to capture the informationsect Use Goal Question Measure (GQM) approach
ndash Add scales and thresholds
sect Define a data catalogue owners repositoriessect Define the collection and analysis proceduressect Define the presentation and reporting requirements
ndash Produce regular output reports to demonstrate progress ndash give explanations toresults to ensure the right message is delivered
ndash Consistent interpretation
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus Ltd20
Working in Practice
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The Organisation
21
sect Major testing services provider located in Europesect 90 of employees are test professionals (qualified experienced dedicated to
test activities)sect Deliver contract-driven specific test services to all major industry sectorssect Most services deliver system and system integration test levels
ndash 80+ contracts delivering services via a Test Factoryndash They are not usually involved in other parts of test and delivery life cycle(s)ndash Internal ethos of ldquojust do itrdquo instead of ldquomonitor and learnrdquondash A formal restrained communication and information exchange between
organisation and clients
sect The strategic and organisational objective was to achieve TMMi Level 3certification
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The Road Map
22
sect TMMi Baseline assessment undertaken to Level 3 in March 2010ndash Numerous process gaps evidenced in all process areas for both levels 2 amp 3
sect Process Improvement project initiatedndash Involved senior test and quality management resources to demonstrate
commitment to the rest of the organisationndash Aggressive timescales to define and implement improvements were setndash Assessment provider undertook quality assurance and TMMi progress checkpoint
activities
sect Robust approach in line with the IMPROVE modelsect An implementation plan ensured the organisation had implemented in time to
be re-assessed in December 2010ndash TMMi Level 3 certification achieved with no issues
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Oh to measure
23
sect Strategic objective only to achieve certificationndash No stated requirement to measure before and after implementation of changendash Measure only because TMMi says so
sect At March 2010 only 2 Key Performance Indicators were definedndash Service Level Agreements stated ldquo must deliver a) on time and b) on budgetrdquondash No declared interest in process or product quality
sect No requirement to identify cost of process weaknessessect Initial assessment data identified product quality ldquoKPIrdquo as well
ndash Organisation is not paid if ANY severity 1 issues found by client that SHOULD havebeen found before
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Measurement as implemented
24
sect Using GQM defined 6 internal KPIs with thresholdsndash Requirements Coverage ndash requirements planned to be tested amp successfully testedndash Quality of Incidents found ndash what level of duplication rejection was evidencedndash Incidents per Test Cases ndash good test cases finding incidentsndash Incident detection trend ndash when in the life cycle does testing find the incidentsndash Actual vs Planned test effort and deviation levelndash Test effort per test case
sect Added customer satisfaction datasect Retained on time and budget contractual SLArsquossect Integrated into test monitoring control and reporting within delivery projects
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Andhellip
25
sect The organisation achieved certification on timesect They also used the KPI data to observe that in the first 6 months of 2011
ndash Test case specificationsect Efficiency had increased 8 between January and June
sect Effectiveness had increased 11 between January and June
ndash Test case executionsect Efficiency had increased 5 between January and June
sect Effectiveness had increased 5 between January and June
ndash Quality of test cases had increased 9ndash Quality of issues detected had increased 6They also think that client relationships have become more open and positive and thepost delivery ldquonegotiationsrdquo are much easier
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
And the message is
26
sect With no requirement for measurement beyond the needs of a processreference modelndash A few well defined easy to understand KPIsndash Implemented collected analysed and reported quicklyndash Demonstrated to all parties the improvements that were accruing following
implemented change
It is good to measure (but make it easy to do so)
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd27
Thank you
Brian Wellsinfoexperimentuscom
+44(0)207 871 2300
wwwexperimentuscom
wwwtmmifoundationorg
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Who owns TMMireg
sect The TMMi Foundation
ndash Maintains ownership of the TMMireg model and the management of the ongoingintellectual property rights
ndash Creation and management of an independent central data repository and provisionof industry and other analysis services
ndash Provision of an independent accreditation process for TMMireg assessment methodsand Assessor resources
ndash Provision of a public forum of interested parties
sect For more information visit wwwtmmifoundationorg
28
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
TMMireg Foundation Contacts
29
sect Full information on the work of the Foundation can be found onwwwtmmifoundationorg
sect If interested in becoming a member of the Foundation (currently freeregistration) please email your details to infotmmifoundationorg
sect If your organisation is interested in supporting the work of the Foundationplease email sponsorshiptmmifoundationorg
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
TMMi Output
6
sect A TMMi Assessment will producendash A structured identification of strengths within the test processndash A structured definition of process improvement requirements
sect Any process improvement requirements indicatendash Where there is a poor quality process elementndash A potential for poor product quality being delivered
sect The (optional) ability to calculate potential Return on Investment of processimprovements to the business to demonstrate the value that is achievable
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus Ltd7
Approaching ProcessImprovement
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
What is test process improvement
8
Continuous improvement of the quality and the efficiency of the testingprocess in the context of the whole software development life cycle
The ISTQB glossary seems to have got its focus wrong
when it defines test process improvement as
A continuous framework for test process improvement that describesthe key elements of an effective test process especially targeted at
system testing and acceptance testing
A continuous framework for test process improvement that describesthe key elements of an effective test process especially targeted at
system testing and acceptance testing
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The process improvement process
9
PrioritisePlan
InitiateMeasure
DefineRedefineOperateValidateEvolve
M
RP
O
EV
Confirm the objective choose appropriate model
List the possible process improvements
Undertake the assessment
Createamend the processes
Process improvements are deployed
Benefits are realised and success criteria met
Stopstart the next stage of improvement
I
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Measure
Prioritiseamp Plan
Define Re-define
Operate
Validate
Evolve
Methodology - IMPROVE
10
Initiate
Organisation
Process Technology
bull Action Planningbull Target Setting
bull New processdefinitionrefinement
bull Trainingbull Pilotbull Full deployment
bull Monitoring of resultsagainst targets
bull Evolution of programmeapproach amp targets
bull Assess Processmonitoragainst plan
bull Obtain agreementto progress
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The business case
11
sect To define objective and prioritiessect Need to define the benefits ndash donrsquot over-exaggerate
ndash Intangiblendash Tangible
sect Understand the costs involvedsect Timescalessect Quick wins
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Intangible benefits
12
sect Clearer and well understood processessect Repeatabilitysect Reduction in key person dependency ndash single point of failuresect Measurement of progresssect Measurement of processsect Clear roadmap of what to do nextsect Better career opportunitiessect Happier staff
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Financial benefits
1313
PIPID
PIP Summary EstimatedPersonDay
EstimatedCost
PotentialBenefit
Net PotentialBenefit
1 Institutionalisation and mandating of generic(consolidated) Test Framework elements 48 48000 214290 166290
2 Define approach process procedures templatestraining guidance for Risk Based Prioritisation 15 15000 114097 99097
3Document the structured procedures templatestraining tools to undertake informal andstructured reviews 15 15000 586170 571170
4 Consolidate elements into comprehensiveMonitoring amp Control Process definition 24 24000 196065 172065
5Evaluate and define Test Standard specific suiteof training modules and make available viaLearning Zone 15 15000 0 -15000
117000 1110622 99362295 105509150 175500
879591
TOTAL NET POTENTIAL BENEFITSConfidence Factor
Implementation FactorADJUSTED NET POTENTIAL BENEFITS
Potential Benefits and Implementation Estimates Summary
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Define-redefine
14
sect Based on the process improvements identified and agreed - eitherndash define any new methods processes and templatesOrndash where existing processes exist but need updating they are re-defined to bring
them up to date
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Operate
15
sect This is the rollout phasesect Will be at least 80 of the improvement project
sect Agree the improvements to be rolled outndash pilot project or piece of workndash with the support team to see if they have any questionsndash feedback to the process
sect Provide trainingsect Provide mentoring
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Validate
16
sect Benefits realisation stage as well as refinement stagesect Measurement of benefits ndash depends on having a good understanding of the
start positionsect Both tangible and intangiblesect Have the improvement projectrsquos success or exit criteria been metsect Close of the formal improvement project
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Assess and developImprovements
Implementation
17
ImplementationBAU
review
Initiate ndash Measure ndashPrioritise Plan and
DefineIMP Redefine
OperateRO ValidateEvolveVE
Ent
husi
asm
MaximumHype
Actual UseDisillusionment
Practical Benefit
RIP
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus Ltd18
To Measure
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
How to approach measuring
19
sect Agree what message needs to be deliveredsect Keep It Simple Short (KISS) relevant and easy to capture the informationsect Use Goal Question Measure (GQM) approach
ndash Add scales and thresholds
sect Define a data catalogue owners repositoriessect Define the collection and analysis proceduressect Define the presentation and reporting requirements
ndash Produce regular output reports to demonstrate progress ndash give explanations toresults to ensure the right message is delivered
ndash Consistent interpretation
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus Ltd20
Working in Practice
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The Organisation
21
sect Major testing services provider located in Europesect 90 of employees are test professionals (qualified experienced dedicated to
test activities)sect Deliver contract-driven specific test services to all major industry sectorssect Most services deliver system and system integration test levels
ndash 80+ contracts delivering services via a Test Factoryndash They are not usually involved in other parts of test and delivery life cycle(s)ndash Internal ethos of ldquojust do itrdquo instead of ldquomonitor and learnrdquondash A formal restrained communication and information exchange between
organisation and clients
sect The strategic and organisational objective was to achieve TMMi Level 3certification
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The Road Map
22
sect TMMi Baseline assessment undertaken to Level 3 in March 2010ndash Numerous process gaps evidenced in all process areas for both levels 2 amp 3
sect Process Improvement project initiatedndash Involved senior test and quality management resources to demonstrate
commitment to the rest of the organisationndash Aggressive timescales to define and implement improvements were setndash Assessment provider undertook quality assurance and TMMi progress checkpoint
activities
sect Robust approach in line with the IMPROVE modelsect An implementation plan ensured the organisation had implemented in time to
be re-assessed in December 2010ndash TMMi Level 3 certification achieved with no issues
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Oh to measure
23
sect Strategic objective only to achieve certificationndash No stated requirement to measure before and after implementation of changendash Measure only because TMMi says so
sect At March 2010 only 2 Key Performance Indicators were definedndash Service Level Agreements stated ldquo must deliver a) on time and b) on budgetrdquondash No declared interest in process or product quality
sect No requirement to identify cost of process weaknessessect Initial assessment data identified product quality ldquoKPIrdquo as well
ndash Organisation is not paid if ANY severity 1 issues found by client that SHOULD havebeen found before
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Measurement as implemented
24
sect Using GQM defined 6 internal KPIs with thresholdsndash Requirements Coverage ndash requirements planned to be tested amp successfully testedndash Quality of Incidents found ndash what level of duplication rejection was evidencedndash Incidents per Test Cases ndash good test cases finding incidentsndash Incident detection trend ndash when in the life cycle does testing find the incidentsndash Actual vs Planned test effort and deviation levelndash Test effort per test case
sect Added customer satisfaction datasect Retained on time and budget contractual SLArsquossect Integrated into test monitoring control and reporting within delivery projects
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Andhellip
25
sect The organisation achieved certification on timesect They also used the KPI data to observe that in the first 6 months of 2011
ndash Test case specificationsect Efficiency had increased 8 between January and June
sect Effectiveness had increased 11 between January and June
ndash Test case executionsect Efficiency had increased 5 between January and June
sect Effectiveness had increased 5 between January and June
ndash Quality of test cases had increased 9ndash Quality of issues detected had increased 6They also think that client relationships have become more open and positive and thepost delivery ldquonegotiationsrdquo are much easier
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
And the message is
26
sect With no requirement for measurement beyond the needs of a processreference modelndash A few well defined easy to understand KPIsndash Implemented collected analysed and reported quicklyndash Demonstrated to all parties the improvements that were accruing following
implemented change
It is good to measure (but make it easy to do so)
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd27
Thank you
Brian Wellsinfoexperimentuscom
+44(0)207 871 2300
wwwexperimentuscom
wwwtmmifoundationorg
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Who owns TMMireg
sect The TMMi Foundation
ndash Maintains ownership of the TMMireg model and the management of the ongoingintellectual property rights
ndash Creation and management of an independent central data repository and provisionof industry and other analysis services
ndash Provision of an independent accreditation process for TMMireg assessment methodsand Assessor resources
ndash Provision of a public forum of interested parties
sect For more information visit wwwtmmifoundationorg
28
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
TMMireg Foundation Contacts
29
sect Full information on the work of the Foundation can be found onwwwtmmifoundationorg
sect If interested in becoming a member of the Foundation (currently freeregistration) please email your details to infotmmifoundationorg
sect If your organisation is interested in supporting the work of the Foundationplease email sponsorshiptmmifoundationorg
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus Ltd7
Approaching ProcessImprovement
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
What is test process improvement
8
Continuous improvement of the quality and the efficiency of the testingprocess in the context of the whole software development life cycle
The ISTQB glossary seems to have got its focus wrong
when it defines test process improvement as
A continuous framework for test process improvement that describesthe key elements of an effective test process especially targeted at
system testing and acceptance testing
A continuous framework for test process improvement that describesthe key elements of an effective test process especially targeted at
system testing and acceptance testing
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The process improvement process
9
PrioritisePlan
InitiateMeasure
DefineRedefineOperateValidateEvolve
M
RP
O
EV
Confirm the objective choose appropriate model
List the possible process improvements
Undertake the assessment
Createamend the processes
Process improvements are deployed
Benefits are realised and success criteria met
Stopstart the next stage of improvement
I
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Measure
Prioritiseamp Plan
Define Re-define
Operate
Validate
Evolve
Methodology - IMPROVE
10
Initiate
Organisation
Process Technology
bull Action Planningbull Target Setting
bull New processdefinitionrefinement
bull Trainingbull Pilotbull Full deployment
bull Monitoring of resultsagainst targets
bull Evolution of programmeapproach amp targets
bull Assess Processmonitoragainst plan
bull Obtain agreementto progress
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The business case
11
sect To define objective and prioritiessect Need to define the benefits ndash donrsquot over-exaggerate
ndash Intangiblendash Tangible
sect Understand the costs involvedsect Timescalessect Quick wins
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Intangible benefits
12
sect Clearer and well understood processessect Repeatabilitysect Reduction in key person dependency ndash single point of failuresect Measurement of progresssect Measurement of processsect Clear roadmap of what to do nextsect Better career opportunitiessect Happier staff
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Financial benefits
1313
PIPID
PIP Summary EstimatedPersonDay
EstimatedCost
PotentialBenefit
Net PotentialBenefit
1 Institutionalisation and mandating of generic(consolidated) Test Framework elements 48 48000 214290 166290
2 Define approach process procedures templatestraining guidance for Risk Based Prioritisation 15 15000 114097 99097
3Document the structured procedures templatestraining tools to undertake informal andstructured reviews 15 15000 586170 571170
4 Consolidate elements into comprehensiveMonitoring amp Control Process definition 24 24000 196065 172065
5Evaluate and define Test Standard specific suiteof training modules and make available viaLearning Zone 15 15000 0 -15000
117000 1110622 99362295 105509150 175500
879591
TOTAL NET POTENTIAL BENEFITSConfidence Factor
Implementation FactorADJUSTED NET POTENTIAL BENEFITS
Potential Benefits and Implementation Estimates Summary
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Define-redefine
14
sect Based on the process improvements identified and agreed - eitherndash define any new methods processes and templatesOrndash where existing processes exist but need updating they are re-defined to bring
them up to date
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Operate
15
sect This is the rollout phasesect Will be at least 80 of the improvement project
sect Agree the improvements to be rolled outndash pilot project or piece of workndash with the support team to see if they have any questionsndash feedback to the process
sect Provide trainingsect Provide mentoring
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Validate
16
sect Benefits realisation stage as well as refinement stagesect Measurement of benefits ndash depends on having a good understanding of the
start positionsect Both tangible and intangiblesect Have the improvement projectrsquos success or exit criteria been metsect Close of the formal improvement project
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Assess and developImprovements
Implementation
17
ImplementationBAU
review
Initiate ndash Measure ndashPrioritise Plan and
DefineIMP Redefine
OperateRO ValidateEvolveVE
Ent
husi
asm
MaximumHype
Actual UseDisillusionment
Practical Benefit
RIP
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus Ltd18
To Measure
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
How to approach measuring
19
sect Agree what message needs to be deliveredsect Keep It Simple Short (KISS) relevant and easy to capture the informationsect Use Goal Question Measure (GQM) approach
ndash Add scales and thresholds
sect Define a data catalogue owners repositoriessect Define the collection and analysis proceduressect Define the presentation and reporting requirements
ndash Produce regular output reports to demonstrate progress ndash give explanations toresults to ensure the right message is delivered
ndash Consistent interpretation
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus Ltd20
Working in Practice
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The Organisation
21
sect Major testing services provider located in Europesect 90 of employees are test professionals (qualified experienced dedicated to
test activities)sect Deliver contract-driven specific test services to all major industry sectorssect Most services deliver system and system integration test levels
ndash 80+ contracts delivering services via a Test Factoryndash They are not usually involved in other parts of test and delivery life cycle(s)ndash Internal ethos of ldquojust do itrdquo instead of ldquomonitor and learnrdquondash A formal restrained communication and information exchange between
organisation and clients
sect The strategic and organisational objective was to achieve TMMi Level 3certification
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The Road Map
22
sect TMMi Baseline assessment undertaken to Level 3 in March 2010ndash Numerous process gaps evidenced in all process areas for both levels 2 amp 3
sect Process Improvement project initiatedndash Involved senior test and quality management resources to demonstrate
commitment to the rest of the organisationndash Aggressive timescales to define and implement improvements were setndash Assessment provider undertook quality assurance and TMMi progress checkpoint
activities
sect Robust approach in line with the IMPROVE modelsect An implementation plan ensured the organisation had implemented in time to
be re-assessed in December 2010ndash TMMi Level 3 certification achieved with no issues
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Oh to measure
23
sect Strategic objective only to achieve certificationndash No stated requirement to measure before and after implementation of changendash Measure only because TMMi says so
sect At March 2010 only 2 Key Performance Indicators were definedndash Service Level Agreements stated ldquo must deliver a) on time and b) on budgetrdquondash No declared interest in process or product quality
sect No requirement to identify cost of process weaknessessect Initial assessment data identified product quality ldquoKPIrdquo as well
ndash Organisation is not paid if ANY severity 1 issues found by client that SHOULD havebeen found before
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Measurement as implemented
24
sect Using GQM defined 6 internal KPIs with thresholdsndash Requirements Coverage ndash requirements planned to be tested amp successfully testedndash Quality of Incidents found ndash what level of duplication rejection was evidencedndash Incidents per Test Cases ndash good test cases finding incidentsndash Incident detection trend ndash when in the life cycle does testing find the incidentsndash Actual vs Planned test effort and deviation levelndash Test effort per test case
sect Added customer satisfaction datasect Retained on time and budget contractual SLArsquossect Integrated into test monitoring control and reporting within delivery projects
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Andhellip
25
sect The organisation achieved certification on timesect They also used the KPI data to observe that in the first 6 months of 2011
ndash Test case specificationsect Efficiency had increased 8 between January and June
sect Effectiveness had increased 11 between January and June
ndash Test case executionsect Efficiency had increased 5 between January and June
sect Effectiveness had increased 5 between January and June
ndash Quality of test cases had increased 9ndash Quality of issues detected had increased 6They also think that client relationships have become more open and positive and thepost delivery ldquonegotiationsrdquo are much easier
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
And the message is
26
sect With no requirement for measurement beyond the needs of a processreference modelndash A few well defined easy to understand KPIsndash Implemented collected analysed and reported quicklyndash Demonstrated to all parties the improvements that were accruing following
implemented change
It is good to measure (but make it easy to do so)
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd27
Thank you
Brian Wellsinfoexperimentuscom
+44(0)207 871 2300
wwwexperimentuscom
wwwtmmifoundationorg
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Who owns TMMireg
sect The TMMi Foundation
ndash Maintains ownership of the TMMireg model and the management of the ongoingintellectual property rights
ndash Creation and management of an independent central data repository and provisionof industry and other analysis services
ndash Provision of an independent accreditation process for TMMireg assessment methodsand Assessor resources
ndash Provision of a public forum of interested parties
sect For more information visit wwwtmmifoundationorg
28
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
TMMireg Foundation Contacts
29
sect Full information on the work of the Foundation can be found onwwwtmmifoundationorg
sect If interested in becoming a member of the Foundation (currently freeregistration) please email your details to infotmmifoundationorg
sect If your organisation is interested in supporting the work of the Foundationplease email sponsorshiptmmifoundationorg
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
What is test process improvement
8
Continuous improvement of the quality and the efficiency of the testingprocess in the context of the whole software development life cycle
The ISTQB glossary seems to have got its focus wrong
when it defines test process improvement as
A continuous framework for test process improvement that describesthe key elements of an effective test process especially targeted at
system testing and acceptance testing
A continuous framework for test process improvement that describesthe key elements of an effective test process especially targeted at
system testing and acceptance testing
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The process improvement process
9
PrioritisePlan
InitiateMeasure
DefineRedefineOperateValidateEvolve
M
RP
O
EV
Confirm the objective choose appropriate model
List the possible process improvements
Undertake the assessment
Createamend the processes
Process improvements are deployed
Benefits are realised and success criteria met
Stopstart the next stage of improvement
I
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Measure
Prioritiseamp Plan
Define Re-define
Operate
Validate
Evolve
Methodology - IMPROVE
10
Initiate
Organisation
Process Technology
bull Action Planningbull Target Setting
bull New processdefinitionrefinement
bull Trainingbull Pilotbull Full deployment
bull Monitoring of resultsagainst targets
bull Evolution of programmeapproach amp targets
bull Assess Processmonitoragainst plan
bull Obtain agreementto progress
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The business case
11
sect To define objective and prioritiessect Need to define the benefits ndash donrsquot over-exaggerate
ndash Intangiblendash Tangible
sect Understand the costs involvedsect Timescalessect Quick wins
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Intangible benefits
12
sect Clearer and well understood processessect Repeatabilitysect Reduction in key person dependency ndash single point of failuresect Measurement of progresssect Measurement of processsect Clear roadmap of what to do nextsect Better career opportunitiessect Happier staff
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Financial benefits
1313
PIPID
PIP Summary EstimatedPersonDay
EstimatedCost
PotentialBenefit
Net PotentialBenefit
1 Institutionalisation and mandating of generic(consolidated) Test Framework elements 48 48000 214290 166290
2 Define approach process procedures templatestraining guidance for Risk Based Prioritisation 15 15000 114097 99097
3Document the structured procedures templatestraining tools to undertake informal andstructured reviews 15 15000 586170 571170
4 Consolidate elements into comprehensiveMonitoring amp Control Process definition 24 24000 196065 172065
5Evaluate and define Test Standard specific suiteof training modules and make available viaLearning Zone 15 15000 0 -15000
117000 1110622 99362295 105509150 175500
879591
TOTAL NET POTENTIAL BENEFITSConfidence Factor
Implementation FactorADJUSTED NET POTENTIAL BENEFITS
Potential Benefits and Implementation Estimates Summary
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Define-redefine
14
sect Based on the process improvements identified and agreed - eitherndash define any new methods processes and templatesOrndash where existing processes exist but need updating they are re-defined to bring
them up to date
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Operate
15
sect This is the rollout phasesect Will be at least 80 of the improvement project
sect Agree the improvements to be rolled outndash pilot project or piece of workndash with the support team to see if they have any questionsndash feedback to the process
sect Provide trainingsect Provide mentoring
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Validate
16
sect Benefits realisation stage as well as refinement stagesect Measurement of benefits ndash depends on having a good understanding of the
start positionsect Both tangible and intangiblesect Have the improvement projectrsquos success or exit criteria been metsect Close of the formal improvement project
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Assess and developImprovements
Implementation
17
ImplementationBAU
review
Initiate ndash Measure ndashPrioritise Plan and
DefineIMP Redefine
OperateRO ValidateEvolveVE
Ent
husi
asm
MaximumHype
Actual UseDisillusionment
Practical Benefit
RIP
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus Ltd18
To Measure
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
How to approach measuring
19
sect Agree what message needs to be deliveredsect Keep It Simple Short (KISS) relevant and easy to capture the informationsect Use Goal Question Measure (GQM) approach
ndash Add scales and thresholds
sect Define a data catalogue owners repositoriessect Define the collection and analysis proceduressect Define the presentation and reporting requirements
ndash Produce regular output reports to demonstrate progress ndash give explanations toresults to ensure the right message is delivered
ndash Consistent interpretation
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus Ltd20
Working in Practice
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The Organisation
21
sect Major testing services provider located in Europesect 90 of employees are test professionals (qualified experienced dedicated to
test activities)sect Deliver contract-driven specific test services to all major industry sectorssect Most services deliver system and system integration test levels
ndash 80+ contracts delivering services via a Test Factoryndash They are not usually involved in other parts of test and delivery life cycle(s)ndash Internal ethos of ldquojust do itrdquo instead of ldquomonitor and learnrdquondash A formal restrained communication and information exchange between
organisation and clients
sect The strategic and organisational objective was to achieve TMMi Level 3certification
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The Road Map
22
sect TMMi Baseline assessment undertaken to Level 3 in March 2010ndash Numerous process gaps evidenced in all process areas for both levels 2 amp 3
sect Process Improvement project initiatedndash Involved senior test and quality management resources to demonstrate
commitment to the rest of the organisationndash Aggressive timescales to define and implement improvements were setndash Assessment provider undertook quality assurance and TMMi progress checkpoint
activities
sect Robust approach in line with the IMPROVE modelsect An implementation plan ensured the organisation had implemented in time to
be re-assessed in December 2010ndash TMMi Level 3 certification achieved with no issues
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Oh to measure
23
sect Strategic objective only to achieve certificationndash No stated requirement to measure before and after implementation of changendash Measure only because TMMi says so
sect At March 2010 only 2 Key Performance Indicators were definedndash Service Level Agreements stated ldquo must deliver a) on time and b) on budgetrdquondash No declared interest in process or product quality
sect No requirement to identify cost of process weaknessessect Initial assessment data identified product quality ldquoKPIrdquo as well
ndash Organisation is not paid if ANY severity 1 issues found by client that SHOULD havebeen found before
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Measurement as implemented
24
sect Using GQM defined 6 internal KPIs with thresholdsndash Requirements Coverage ndash requirements planned to be tested amp successfully testedndash Quality of Incidents found ndash what level of duplication rejection was evidencedndash Incidents per Test Cases ndash good test cases finding incidentsndash Incident detection trend ndash when in the life cycle does testing find the incidentsndash Actual vs Planned test effort and deviation levelndash Test effort per test case
sect Added customer satisfaction datasect Retained on time and budget contractual SLArsquossect Integrated into test monitoring control and reporting within delivery projects
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Andhellip
25
sect The organisation achieved certification on timesect They also used the KPI data to observe that in the first 6 months of 2011
ndash Test case specificationsect Efficiency had increased 8 between January and June
sect Effectiveness had increased 11 between January and June
ndash Test case executionsect Efficiency had increased 5 between January and June
sect Effectiveness had increased 5 between January and June
ndash Quality of test cases had increased 9ndash Quality of issues detected had increased 6They also think that client relationships have become more open and positive and thepost delivery ldquonegotiationsrdquo are much easier
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
And the message is
26
sect With no requirement for measurement beyond the needs of a processreference modelndash A few well defined easy to understand KPIsndash Implemented collected analysed and reported quicklyndash Demonstrated to all parties the improvements that were accruing following
implemented change
It is good to measure (but make it easy to do so)
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd27
Thank you
Brian Wellsinfoexperimentuscom
+44(0)207 871 2300
wwwexperimentuscom
wwwtmmifoundationorg
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Who owns TMMireg
sect The TMMi Foundation
ndash Maintains ownership of the TMMireg model and the management of the ongoingintellectual property rights
ndash Creation and management of an independent central data repository and provisionof industry and other analysis services
ndash Provision of an independent accreditation process for TMMireg assessment methodsand Assessor resources
ndash Provision of a public forum of interested parties
sect For more information visit wwwtmmifoundationorg
28
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
TMMireg Foundation Contacts
29
sect Full information on the work of the Foundation can be found onwwwtmmifoundationorg
sect If interested in becoming a member of the Foundation (currently freeregistration) please email your details to infotmmifoundationorg
sect If your organisation is interested in supporting the work of the Foundationplease email sponsorshiptmmifoundationorg
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The process improvement process
9
PrioritisePlan
InitiateMeasure
DefineRedefineOperateValidateEvolve
M
RP
O
EV
Confirm the objective choose appropriate model
List the possible process improvements
Undertake the assessment
Createamend the processes
Process improvements are deployed
Benefits are realised and success criteria met
Stopstart the next stage of improvement
I
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Measure
Prioritiseamp Plan
Define Re-define
Operate
Validate
Evolve
Methodology - IMPROVE
10
Initiate
Organisation
Process Technology
bull Action Planningbull Target Setting
bull New processdefinitionrefinement
bull Trainingbull Pilotbull Full deployment
bull Monitoring of resultsagainst targets
bull Evolution of programmeapproach amp targets
bull Assess Processmonitoragainst plan
bull Obtain agreementto progress
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The business case
11
sect To define objective and prioritiessect Need to define the benefits ndash donrsquot over-exaggerate
ndash Intangiblendash Tangible
sect Understand the costs involvedsect Timescalessect Quick wins
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Intangible benefits
12
sect Clearer and well understood processessect Repeatabilitysect Reduction in key person dependency ndash single point of failuresect Measurement of progresssect Measurement of processsect Clear roadmap of what to do nextsect Better career opportunitiessect Happier staff
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Financial benefits
1313
PIPID
PIP Summary EstimatedPersonDay
EstimatedCost
PotentialBenefit
Net PotentialBenefit
1 Institutionalisation and mandating of generic(consolidated) Test Framework elements 48 48000 214290 166290
2 Define approach process procedures templatestraining guidance for Risk Based Prioritisation 15 15000 114097 99097
3Document the structured procedures templatestraining tools to undertake informal andstructured reviews 15 15000 586170 571170
4 Consolidate elements into comprehensiveMonitoring amp Control Process definition 24 24000 196065 172065
5Evaluate and define Test Standard specific suiteof training modules and make available viaLearning Zone 15 15000 0 -15000
117000 1110622 99362295 105509150 175500
879591
TOTAL NET POTENTIAL BENEFITSConfidence Factor
Implementation FactorADJUSTED NET POTENTIAL BENEFITS
Potential Benefits and Implementation Estimates Summary
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Define-redefine
14
sect Based on the process improvements identified and agreed - eitherndash define any new methods processes and templatesOrndash where existing processes exist but need updating they are re-defined to bring
them up to date
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Operate
15
sect This is the rollout phasesect Will be at least 80 of the improvement project
sect Agree the improvements to be rolled outndash pilot project or piece of workndash with the support team to see if they have any questionsndash feedback to the process
sect Provide trainingsect Provide mentoring
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Validate
16
sect Benefits realisation stage as well as refinement stagesect Measurement of benefits ndash depends on having a good understanding of the
start positionsect Both tangible and intangiblesect Have the improvement projectrsquos success or exit criteria been metsect Close of the formal improvement project
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Assess and developImprovements
Implementation
17
ImplementationBAU
review
Initiate ndash Measure ndashPrioritise Plan and
DefineIMP Redefine
OperateRO ValidateEvolveVE
Ent
husi
asm
MaximumHype
Actual UseDisillusionment
Practical Benefit
RIP
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus Ltd18
To Measure
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
How to approach measuring
19
sect Agree what message needs to be deliveredsect Keep It Simple Short (KISS) relevant and easy to capture the informationsect Use Goal Question Measure (GQM) approach
ndash Add scales and thresholds
sect Define a data catalogue owners repositoriessect Define the collection and analysis proceduressect Define the presentation and reporting requirements
ndash Produce regular output reports to demonstrate progress ndash give explanations toresults to ensure the right message is delivered
ndash Consistent interpretation
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus Ltd20
Working in Practice
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The Organisation
21
sect Major testing services provider located in Europesect 90 of employees are test professionals (qualified experienced dedicated to
test activities)sect Deliver contract-driven specific test services to all major industry sectorssect Most services deliver system and system integration test levels
ndash 80+ contracts delivering services via a Test Factoryndash They are not usually involved in other parts of test and delivery life cycle(s)ndash Internal ethos of ldquojust do itrdquo instead of ldquomonitor and learnrdquondash A formal restrained communication and information exchange between
organisation and clients
sect The strategic and organisational objective was to achieve TMMi Level 3certification
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The Road Map
22
sect TMMi Baseline assessment undertaken to Level 3 in March 2010ndash Numerous process gaps evidenced in all process areas for both levels 2 amp 3
sect Process Improvement project initiatedndash Involved senior test and quality management resources to demonstrate
commitment to the rest of the organisationndash Aggressive timescales to define and implement improvements were setndash Assessment provider undertook quality assurance and TMMi progress checkpoint
activities
sect Robust approach in line with the IMPROVE modelsect An implementation plan ensured the organisation had implemented in time to
be re-assessed in December 2010ndash TMMi Level 3 certification achieved with no issues
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Oh to measure
23
sect Strategic objective only to achieve certificationndash No stated requirement to measure before and after implementation of changendash Measure only because TMMi says so
sect At March 2010 only 2 Key Performance Indicators were definedndash Service Level Agreements stated ldquo must deliver a) on time and b) on budgetrdquondash No declared interest in process or product quality
sect No requirement to identify cost of process weaknessessect Initial assessment data identified product quality ldquoKPIrdquo as well
ndash Organisation is not paid if ANY severity 1 issues found by client that SHOULD havebeen found before
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Measurement as implemented
24
sect Using GQM defined 6 internal KPIs with thresholdsndash Requirements Coverage ndash requirements planned to be tested amp successfully testedndash Quality of Incidents found ndash what level of duplication rejection was evidencedndash Incidents per Test Cases ndash good test cases finding incidentsndash Incident detection trend ndash when in the life cycle does testing find the incidentsndash Actual vs Planned test effort and deviation levelndash Test effort per test case
sect Added customer satisfaction datasect Retained on time and budget contractual SLArsquossect Integrated into test monitoring control and reporting within delivery projects
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Andhellip
25
sect The organisation achieved certification on timesect They also used the KPI data to observe that in the first 6 months of 2011
ndash Test case specificationsect Efficiency had increased 8 between January and June
sect Effectiveness had increased 11 between January and June
ndash Test case executionsect Efficiency had increased 5 between January and June
sect Effectiveness had increased 5 between January and June
ndash Quality of test cases had increased 9ndash Quality of issues detected had increased 6They also think that client relationships have become more open and positive and thepost delivery ldquonegotiationsrdquo are much easier
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
And the message is
26
sect With no requirement for measurement beyond the needs of a processreference modelndash A few well defined easy to understand KPIsndash Implemented collected analysed and reported quicklyndash Demonstrated to all parties the improvements that were accruing following
implemented change
It is good to measure (but make it easy to do so)
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd27
Thank you
Brian Wellsinfoexperimentuscom
+44(0)207 871 2300
wwwexperimentuscom
wwwtmmifoundationorg
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Who owns TMMireg
sect The TMMi Foundation
ndash Maintains ownership of the TMMireg model and the management of the ongoingintellectual property rights
ndash Creation and management of an independent central data repository and provisionof industry and other analysis services
ndash Provision of an independent accreditation process for TMMireg assessment methodsand Assessor resources
ndash Provision of a public forum of interested parties
sect For more information visit wwwtmmifoundationorg
28
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
TMMireg Foundation Contacts
29
sect Full information on the work of the Foundation can be found onwwwtmmifoundationorg
sect If interested in becoming a member of the Foundation (currently freeregistration) please email your details to infotmmifoundationorg
sect If your organisation is interested in supporting the work of the Foundationplease email sponsorshiptmmifoundationorg
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Measure
Prioritiseamp Plan
Define Re-define
Operate
Validate
Evolve
Methodology - IMPROVE
10
Initiate
Organisation
Process Technology
bull Action Planningbull Target Setting
bull New processdefinitionrefinement
bull Trainingbull Pilotbull Full deployment
bull Monitoring of resultsagainst targets
bull Evolution of programmeapproach amp targets
bull Assess Processmonitoragainst plan
bull Obtain agreementto progress
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The business case
11
sect To define objective and prioritiessect Need to define the benefits ndash donrsquot over-exaggerate
ndash Intangiblendash Tangible
sect Understand the costs involvedsect Timescalessect Quick wins
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Intangible benefits
12
sect Clearer and well understood processessect Repeatabilitysect Reduction in key person dependency ndash single point of failuresect Measurement of progresssect Measurement of processsect Clear roadmap of what to do nextsect Better career opportunitiessect Happier staff
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Financial benefits
1313
PIPID
PIP Summary EstimatedPersonDay
EstimatedCost
PotentialBenefit
Net PotentialBenefit
1 Institutionalisation and mandating of generic(consolidated) Test Framework elements 48 48000 214290 166290
2 Define approach process procedures templatestraining guidance for Risk Based Prioritisation 15 15000 114097 99097
3Document the structured procedures templatestraining tools to undertake informal andstructured reviews 15 15000 586170 571170
4 Consolidate elements into comprehensiveMonitoring amp Control Process definition 24 24000 196065 172065
5Evaluate and define Test Standard specific suiteof training modules and make available viaLearning Zone 15 15000 0 -15000
117000 1110622 99362295 105509150 175500
879591
TOTAL NET POTENTIAL BENEFITSConfidence Factor
Implementation FactorADJUSTED NET POTENTIAL BENEFITS
Potential Benefits and Implementation Estimates Summary
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Define-redefine
14
sect Based on the process improvements identified and agreed - eitherndash define any new methods processes and templatesOrndash where existing processes exist but need updating they are re-defined to bring
them up to date
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Operate
15
sect This is the rollout phasesect Will be at least 80 of the improvement project
sect Agree the improvements to be rolled outndash pilot project or piece of workndash with the support team to see if they have any questionsndash feedback to the process
sect Provide trainingsect Provide mentoring
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Validate
16
sect Benefits realisation stage as well as refinement stagesect Measurement of benefits ndash depends on having a good understanding of the
start positionsect Both tangible and intangiblesect Have the improvement projectrsquos success or exit criteria been metsect Close of the formal improvement project
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Assess and developImprovements
Implementation
17
ImplementationBAU
review
Initiate ndash Measure ndashPrioritise Plan and
DefineIMP Redefine
OperateRO ValidateEvolveVE
Ent
husi
asm
MaximumHype
Actual UseDisillusionment
Practical Benefit
RIP
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus Ltd18
To Measure
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
How to approach measuring
19
sect Agree what message needs to be deliveredsect Keep It Simple Short (KISS) relevant and easy to capture the informationsect Use Goal Question Measure (GQM) approach
ndash Add scales and thresholds
sect Define a data catalogue owners repositoriessect Define the collection and analysis proceduressect Define the presentation and reporting requirements
ndash Produce regular output reports to demonstrate progress ndash give explanations toresults to ensure the right message is delivered
ndash Consistent interpretation
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus Ltd20
Working in Practice
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The Organisation
21
sect Major testing services provider located in Europesect 90 of employees are test professionals (qualified experienced dedicated to
test activities)sect Deliver contract-driven specific test services to all major industry sectorssect Most services deliver system and system integration test levels
ndash 80+ contracts delivering services via a Test Factoryndash They are not usually involved in other parts of test and delivery life cycle(s)ndash Internal ethos of ldquojust do itrdquo instead of ldquomonitor and learnrdquondash A formal restrained communication and information exchange between
organisation and clients
sect The strategic and organisational objective was to achieve TMMi Level 3certification
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The Road Map
22
sect TMMi Baseline assessment undertaken to Level 3 in March 2010ndash Numerous process gaps evidenced in all process areas for both levels 2 amp 3
sect Process Improvement project initiatedndash Involved senior test and quality management resources to demonstrate
commitment to the rest of the organisationndash Aggressive timescales to define and implement improvements were setndash Assessment provider undertook quality assurance and TMMi progress checkpoint
activities
sect Robust approach in line with the IMPROVE modelsect An implementation plan ensured the organisation had implemented in time to
be re-assessed in December 2010ndash TMMi Level 3 certification achieved with no issues
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Oh to measure
23
sect Strategic objective only to achieve certificationndash No stated requirement to measure before and after implementation of changendash Measure only because TMMi says so
sect At March 2010 only 2 Key Performance Indicators were definedndash Service Level Agreements stated ldquo must deliver a) on time and b) on budgetrdquondash No declared interest in process or product quality
sect No requirement to identify cost of process weaknessessect Initial assessment data identified product quality ldquoKPIrdquo as well
ndash Organisation is not paid if ANY severity 1 issues found by client that SHOULD havebeen found before
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Measurement as implemented
24
sect Using GQM defined 6 internal KPIs with thresholdsndash Requirements Coverage ndash requirements planned to be tested amp successfully testedndash Quality of Incidents found ndash what level of duplication rejection was evidencedndash Incidents per Test Cases ndash good test cases finding incidentsndash Incident detection trend ndash when in the life cycle does testing find the incidentsndash Actual vs Planned test effort and deviation levelndash Test effort per test case
sect Added customer satisfaction datasect Retained on time and budget contractual SLArsquossect Integrated into test monitoring control and reporting within delivery projects
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Andhellip
25
sect The organisation achieved certification on timesect They also used the KPI data to observe that in the first 6 months of 2011
ndash Test case specificationsect Efficiency had increased 8 between January and June
sect Effectiveness had increased 11 between January and June
ndash Test case executionsect Efficiency had increased 5 between January and June
sect Effectiveness had increased 5 between January and June
ndash Quality of test cases had increased 9ndash Quality of issues detected had increased 6They also think that client relationships have become more open and positive and thepost delivery ldquonegotiationsrdquo are much easier
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
And the message is
26
sect With no requirement for measurement beyond the needs of a processreference modelndash A few well defined easy to understand KPIsndash Implemented collected analysed and reported quicklyndash Demonstrated to all parties the improvements that were accruing following
implemented change
It is good to measure (but make it easy to do so)
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd27
Thank you
Brian Wellsinfoexperimentuscom
+44(0)207 871 2300
wwwexperimentuscom
wwwtmmifoundationorg
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Who owns TMMireg
sect The TMMi Foundation
ndash Maintains ownership of the TMMireg model and the management of the ongoingintellectual property rights
ndash Creation and management of an independent central data repository and provisionof industry and other analysis services
ndash Provision of an independent accreditation process for TMMireg assessment methodsand Assessor resources
ndash Provision of a public forum of interested parties
sect For more information visit wwwtmmifoundationorg
28
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
TMMireg Foundation Contacts
29
sect Full information on the work of the Foundation can be found onwwwtmmifoundationorg
sect If interested in becoming a member of the Foundation (currently freeregistration) please email your details to infotmmifoundationorg
sect If your organisation is interested in supporting the work of the Foundationplease email sponsorshiptmmifoundationorg
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The business case
11
sect To define objective and prioritiessect Need to define the benefits ndash donrsquot over-exaggerate
ndash Intangiblendash Tangible
sect Understand the costs involvedsect Timescalessect Quick wins
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Intangible benefits
12
sect Clearer and well understood processessect Repeatabilitysect Reduction in key person dependency ndash single point of failuresect Measurement of progresssect Measurement of processsect Clear roadmap of what to do nextsect Better career opportunitiessect Happier staff
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Financial benefits
1313
PIPID
PIP Summary EstimatedPersonDay
EstimatedCost
PotentialBenefit
Net PotentialBenefit
1 Institutionalisation and mandating of generic(consolidated) Test Framework elements 48 48000 214290 166290
2 Define approach process procedures templatestraining guidance for Risk Based Prioritisation 15 15000 114097 99097
3Document the structured procedures templatestraining tools to undertake informal andstructured reviews 15 15000 586170 571170
4 Consolidate elements into comprehensiveMonitoring amp Control Process definition 24 24000 196065 172065
5Evaluate and define Test Standard specific suiteof training modules and make available viaLearning Zone 15 15000 0 -15000
117000 1110622 99362295 105509150 175500
879591
TOTAL NET POTENTIAL BENEFITSConfidence Factor
Implementation FactorADJUSTED NET POTENTIAL BENEFITS
Potential Benefits and Implementation Estimates Summary
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Define-redefine
14
sect Based on the process improvements identified and agreed - eitherndash define any new methods processes and templatesOrndash where existing processes exist but need updating they are re-defined to bring
them up to date
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Operate
15
sect This is the rollout phasesect Will be at least 80 of the improvement project
sect Agree the improvements to be rolled outndash pilot project or piece of workndash with the support team to see if they have any questionsndash feedback to the process
sect Provide trainingsect Provide mentoring
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Validate
16
sect Benefits realisation stage as well as refinement stagesect Measurement of benefits ndash depends on having a good understanding of the
start positionsect Both tangible and intangiblesect Have the improvement projectrsquos success or exit criteria been metsect Close of the formal improvement project
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Assess and developImprovements
Implementation
17
ImplementationBAU
review
Initiate ndash Measure ndashPrioritise Plan and
DefineIMP Redefine
OperateRO ValidateEvolveVE
Ent
husi
asm
MaximumHype
Actual UseDisillusionment
Practical Benefit
RIP
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus Ltd18
To Measure
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
How to approach measuring
19
sect Agree what message needs to be deliveredsect Keep It Simple Short (KISS) relevant and easy to capture the informationsect Use Goal Question Measure (GQM) approach
ndash Add scales and thresholds
sect Define a data catalogue owners repositoriessect Define the collection and analysis proceduressect Define the presentation and reporting requirements
ndash Produce regular output reports to demonstrate progress ndash give explanations toresults to ensure the right message is delivered
ndash Consistent interpretation
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus Ltd20
Working in Practice
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The Organisation
21
sect Major testing services provider located in Europesect 90 of employees are test professionals (qualified experienced dedicated to
test activities)sect Deliver contract-driven specific test services to all major industry sectorssect Most services deliver system and system integration test levels
ndash 80+ contracts delivering services via a Test Factoryndash They are not usually involved in other parts of test and delivery life cycle(s)ndash Internal ethos of ldquojust do itrdquo instead of ldquomonitor and learnrdquondash A formal restrained communication and information exchange between
organisation and clients
sect The strategic and organisational objective was to achieve TMMi Level 3certification
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The Road Map
22
sect TMMi Baseline assessment undertaken to Level 3 in March 2010ndash Numerous process gaps evidenced in all process areas for both levels 2 amp 3
sect Process Improvement project initiatedndash Involved senior test and quality management resources to demonstrate
commitment to the rest of the organisationndash Aggressive timescales to define and implement improvements were setndash Assessment provider undertook quality assurance and TMMi progress checkpoint
activities
sect Robust approach in line with the IMPROVE modelsect An implementation plan ensured the organisation had implemented in time to
be re-assessed in December 2010ndash TMMi Level 3 certification achieved with no issues
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Oh to measure
23
sect Strategic objective only to achieve certificationndash No stated requirement to measure before and after implementation of changendash Measure only because TMMi says so
sect At March 2010 only 2 Key Performance Indicators were definedndash Service Level Agreements stated ldquo must deliver a) on time and b) on budgetrdquondash No declared interest in process or product quality
sect No requirement to identify cost of process weaknessessect Initial assessment data identified product quality ldquoKPIrdquo as well
ndash Organisation is not paid if ANY severity 1 issues found by client that SHOULD havebeen found before
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Measurement as implemented
24
sect Using GQM defined 6 internal KPIs with thresholdsndash Requirements Coverage ndash requirements planned to be tested amp successfully testedndash Quality of Incidents found ndash what level of duplication rejection was evidencedndash Incidents per Test Cases ndash good test cases finding incidentsndash Incident detection trend ndash when in the life cycle does testing find the incidentsndash Actual vs Planned test effort and deviation levelndash Test effort per test case
sect Added customer satisfaction datasect Retained on time and budget contractual SLArsquossect Integrated into test monitoring control and reporting within delivery projects
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Andhellip
25
sect The organisation achieved certification on timesect They also used the KPI data to observe that in the first 6 months of 2011
ndash Test case specificationsect Efficiency had increased 8 between January and June
sect Effectiveness had increased 11 between January and June
ndash Test case executionsect Efficiency had increased 5 between January and June
sect Effectiveness had increased 5 between January and June
ndash Quality of test cases had increased 9ndash Quality of issues detected had increased 6They also think that client relationships have become more open and positive and thepost delivery ldquonegotiationsrdquo are much easier
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
And the message is
26
sect With no requirement for measurement beyond the needs of a processreference modelndash A few well defined easy to understand KPIsndash Implemented collected analysed and reported quicklyndash Demonstrated to all parties the improvements that were accruing following
implemented change
It is good to measure (but make it easy to do so)
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd27
Thank you
Brian Wellsinfoexperimentuscom
+44(0)207 871 2300
wwwexperimentuscom
wwwtmmifoundationorg
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Who owns TMMireg
sect The TMMi Foundation
ndash Maintains ownership of the TMMireg model and the management of the ongoingintellectual property rights
ndash Creation and management of an independent central data repository and provisionof industry and other analysis services
ndash Provision of an independent accreditation process for TMMireg assessment methodsand Assessor resources
ndash Provision of a public forum of interested parties
sect For more information visit wwwtmmifoundationorg
28
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
TMMireg Foundation Contacts
29
sect Full information on the work of the Foundation can be found onwwwtmmifoundationorg
sect If interested in becoming a member of the Foundation (currently freeregistration) please email your details to infotmmifoundationorg
sect If your organisation is interested in supporting the work of the Foundationplease email sponsorshiptmmifoundationorg
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Intangible benefits
12
sect Clearer and well understood processessect Repeatabilitysect Reduction in key person dependency ndash single point of failuresect Measurement of progresssect Measurement of processsect Clear roadmap of what to do nextsect Better career opportunitiessect Happier staff
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Financial benefits
1313
PIPID
PIP Summary EstimatedPersonDay
EstimatedCost
PotentialBenefit
Net PotentialBenefit
1 Institutionalisation and mandating of generic(consolidated) Test Framework elements 48 48000 214290 166290
2 Define approach process procedures templatestraining guidance for Risk Based Prioritisation 15 15000 114097 99097
3Document the structured procedures templatestraining tools to undertake informal andstructured reviews 15 15000 586170 571170
4 Consolidate elements into comprehensiveMonitoring amp Control Process definition 24 24000 196065 172065
5Evaluate and define Test Standard specific suiteof training modules and make available viaLearning Zone 15 15000 0 -15000
117000 1110622 99362295 105509150 175500
879591
TOTAL NET POTENTIAL BENEFITSConfidence Factor
Implementation FactorADJUSTED NET POTENTIAL BENEFITS
Potential Benefits and Implementation Estimates Summary
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Define-redefine
14
sect Based on the process improvements identified and agreed - eitherndash define any new methods processes and templatesOrndash where existing processes exist but need updating they are re-defined to bring
them up to date
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Operate
15
sect This is the rollout phasesect Will be at least 80 of the improvement project
sect Agree the improvements to be rolled outndash pilot project or piece of workndash with the support team to see if they have any questionsndash feedback to the process
sect Provide trainingsect Provide mentoring
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Validate
16
sect Benefits realisation stage as well as refinement stagesect Measurement of benefits ndash depends on having a good understanding of the
start positionsect Both tangible and intangiblesect Have the improvement projectrsquos success or exit criteria been metsect Close of the formal improvement project
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Assess and developImprovements
Implementation
17
ImplementationBAU
review
Initiate ndash Measure ndashPrioritise Plan and
DefineIMP Redefine
OperateRO ValidateEvolveVE
Ent
husi
asm
MaximumHype
Actual UseDisillusionment
Practical Benefit
RIP
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus Ltd18
To Measure
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
How to approach measuring
19
sect Agree what message needs to be deliveredsect Keep It Simple Short (KISS) relevant and easy to capture the informationsect Use Goal Question Measure (GQM) approach
ndash Add scales and thresholds
sect Define a data catalogue owners repositoriessect Define the collection and analysis proceduressect Define the presentation and reporting requirements
ndash Produce regular output reports to demonstrate progress ndash give explanations toresults to ensure the right message is delivered
ndash Consistent interpretation
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus Ltd20
Working in Practice
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The Organisation
21
sect Major testing services provider located in Europesect 90 of employees are test professionals (qualified experienced dedicated to
test activities)sect Deliver contract-driven specific test services to all major industry sectorssect Most services deliver system and system integration test levels
ndash 80+ contracts delivering services via a Test Factoryndash They are not usually involved in other parts of test and delivery life cycle(s)ndash Internal ethos of ldquojust do itrdquo instead of ldquomonitor and learnrdquondash A formal restrained communication and information exchange between
organisation and clients
sect The strategic and organisational objective was to achieve TMMi Level 3certification
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The Road Map
22
sect TMMi Baseline assessment undertaken to Level 3 in March 2010ndash Numerous process gaps evidenced in all process areas for both levels 2 amp 3
sect Process Improvement project initiatedndash Involved senior test and quality management resources to demonstrate
commitment to the rest of the organisationndash Aggressive timescales to define and implement improvements were setndash Assessment provider undertook quality assurance and TMMi progress checkpoint
activities
sect Robust approach in line with the IMPROVE modelsect An implementation plan ensured the organisation had implemented in time to
be re-assessed in December 2010ndash TMMi Level 3 certification achieved with no issues
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Oh to measure
23
sect Strategic objective only to achieve certificationndash No stated requirement to measure before and after implementation of changendash Measure only because TMMi says so
sect At March 2010 only 2 Key Performance Indicators were definedndash Service Level Agreements stated ldquo must deliver a) on time and b) on budgetrdquondash No declared interest in process or product quality
sect No requirement to identify cost of process weaknessessect Initial assessment data identified product quality ldquoKPIrdquo as well
ndash Organisation is not paid if ANY severity 1 issues found by client that SHOULD havebeen found before
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Measurement as implemented
24
sect Using GQM defined 6 internal KPIs with thresholdsndash Requirements Coverage ndash requirements planned to be tested amp successfully testedndash Quality of Incidents found ndash what level of duplication rejection was evidencedndash Incidents per Test Cases ndash good test cases finding incidentsndash Incident detection trend ndash when in the life cycle does testing find the incidentsndash Actual vs Planned test effort and deviation levelndash Test effort per test case
sect Added customer satisfaction datasect Retained on time and budget contractual SLArsquossect Integrated into test monitoring control and reporting within delivery projects
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Andhellip
25
sect The organisation achieved certification on timesect They also used the KPI data to observe that in the first 6 months of 2011
ndash Test case specificationsect Efficiency had increased 8 between January and June
sect Effectiveness had increased 11 between January and June
ndash Test case executionsect Efficiency had increased 5 between January and June
sect Effectiveness had increased 5 between January and June
ndash Quality of test cases had increased 9ndash Quality of issues detected had increased 6They also think that client relationships have become more open and positive and thepost delivery ldquonegotiationsrdquo are much easier
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
And the message is
26
sect With no requirement for measurement beyond the needs of a processreference modelndash A few well defined easy to understand KPIsndash Implemented collected analysed and reported quicklyndash Demonstrated to all parties the improvements that were accruing following
implemented change
It is good to measure (but make it easy to do so)
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd27
Thank you
Brian Wellsinfoexperimentuscom
+44(0)207 871 2300
wwwexperimentuscom
wwwtmmifoundationorg
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Who owns TMMireg
sect The TMMi Foundation
ndash Maintains ownership of the TMMireg model and the management of the ongoingintellectual property rights
ndash Creation and management of an independent central data repository and provisionof industry and other analysis services
ndash Provision of an independent accreditation process for TMMireg assessment methodsand Assessor resources
ndash Provision of a public forum of interested parties
sect For more information visit wwwtmmifoundationorg
28
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
TMMireg Foundation Contacts
29
sect Full information on the work of the Foundation can be found onwwwtmmifoundationorg
sect If interested in becoming a member of the Foundation (currently freeregistration) please email your details to infotmmifoundationorg
sect If your organisation is interested in supporting the work of the Foundationplease email sponsorshiptmmifoundationorg
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Financial benefits
1313
PIPID
PIP Summary EstimatedPersonDay
EstimatedCost
PotentialBenefit
Net PotentialBenefit
1 Institutionalisation and mandating of generic(consolidated) Test Framework elements 48 48000 214290 166290
2 Define approach process procedures templatestraining guidance for Risk Based Prioritisation 15 15000 114097 99097
3Document the structured procedures templatestraining tools to undertake informal andstructured reviews 15 15000 586170 571170
4 Consolidate elements into comprehensiveMonitoring amp Control Process definition 24 24000 196065 172065
5Evaluate and define Test Standard specific suiteof training modules and make available viaLearning Zone 15 15000 0 -15000
117000 1110622 99362295 105509150 175500
879591
TOTAL NET POTENTIAL BENEFITSConfidence Factor
Implementation FactorADJUSTED NET POTENTIAL BENEFITS
Potential Benefits and Implementation Estimates Summary
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Define-redefine
14
sect Based on the process improvements identified and agreed - eitherndash define any new methods processes and templatesOrndash where existing processes exist but need updating they are re-defined to bring
them up to date
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Operate
15
sect This is the rollout phasesect Will be at least 80 of the improvement project
sect Agree the improvements to be rolled outndash pilot project or piece of workndash with the support team to see if they have any questionsndash feedback to the process
sect Provide trainingsect Provide mentoring
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Validate
16
sect Benefits realisation stage as well as refinement stagesect Measurement of benefits ndash depends on having a good understanding of the
start positionsect Both tangible and intangiblesect Have the improvement projectrsquos success or exit criteria been metsect Close of the formal improvement project
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Assess and developImprovements
Implementation
17
ImplementationBAU
review
Initiate ndash Measure ndashPrioritise Plan and
DefineIMP Redefine
OperateRO ValidateEvolveVE
Ent
husi
asm
MaximumHype
Actual UseDisillusionment
Practical Benefit
RIP
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus Ltd18
To Measure
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
How to approach measuring
19
sect Agree what message needs to be deliveredsect Keep It Simple Short (KISS) relevant and easy to capture the informationsect Use Goal Question Measure (GQM) approach
ndash Add scales and thresholds
sect Define a data catalogue owners repositoriessect Define the collection and analysis proceduressect Define the presentation and reporting requirements
ndash Produce regular output reports to demonstrate progress ndash give explanations toresults to ensure the right message is delivered
ndash Consistent interpretation
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus Ltd20
Working in Practice
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The Organisation
21
sect Major testing services provider located in Europesect 90 of employees are test professionals (qualified experienced dedicated to
test activities)sect Deliver contract-driven specific test services to all major industry sectorssect Most services deliver system and system integration test levels
ndash 80+ contracts delivering services via a Test Factoryndash They are not usually involved in other parts of test and delivery life cycle(s)ndash Internal ethos of ldquojust do itrdquo instead of ldquomonitor and learnrdquondash A formal restrained communication and information exchange between
organisation and clients
sect The strategic and organisational objective was to achieve TMMi Level 3certification
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The Road Map
22
sect TMMi Baseline assessment undertaken to Level 3 in March 2010ndash Numerous process gaps evidenced in all process areas for both levels 2 amp 3
sect Process Improvement project initiatedndash Involved senior test and quality management resources to demonstrate
commitment to the rest of the organisationndash Aggressive timescales to define and implement improvements were setndash Assessment provider undertook quality assurance and TMMi progress checkpoint
activities
sect Robust approach in line with the IMPROVE modelsect An implementation plan ensured the organisation had implemented in time to
be re-assessed in December 2010ndash TMMi Level 3 certification achieved with no issues
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Oh to measure
23
sect Strategic objective only to achieve certificationndash No stated requirement to measure before and after implementation of changendash Measure only because TMMi says so
sect At March 2010 only 2 Key Performance Indicators were definedndash Service Level Agreements stated ldquo must deliver a) on time and b) on budgetrdquondash No declared interest in process or product quality
sect No requirement to identify cost of process weaknessessect Initial assessment data identified product quality ldquoKPIrdquo as well
ndash Organisation is not paid if ANY severity 1 issues found by client that SHOULD havebeen found before
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Measurement as implemented
24
sect Using GQM defined 6 internal KPIs with thresholdsndash Requirements Coverage ndash requirements planned to be tested amp successfully testedndash Quality of Incidents found ndash what level of duplication rejection was evidencedndash Incidents per Test Cases ndash good test cases finding incidentsndash Incident detection trend ndash when in the life cycle does testing find the incidentsndash Actual vs Planned test effort and deviation levelndash Test effort per test case
sect Added customer satisfaction datasect Retained on time and budget contractual SLArsquossect Integrated into test monitoring control and reporting within delivery projects
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Andhellip
25
sect The organisation achieved certification on timesect They also used the KPI data to observe that in the first 6 months of 2011
ndash Test case specificationsect Efficiency had increased 8 between January and June
sect Effectiveness had increased 11 between January and June
ndash Test case executionsect Efficiency had increased 5 between January and June
sect Effectiveness had increased 5 between January and June
ndash Quality of test cases had increased 9ndash Quality of issues detected had increased 6They also think that client relationships have become more open and positive and thepost delivery ldquonegotiationsrdquo are much easier
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
And the message is
26
sect With no requirement for measurement beyond the needs of a processreference modelndash A few well defined easy to understand KPIsndash Implemented collected analysed and reported quicklyndash Demonstrated to all parties the improvements that were accruing following
implemented change
It is good to measure (but make it easy to do so)
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd27
Thank you
Brian Wellsinfoexperimentuscom
+44(0)207 871 2300
wwwexperimentuscom
wwwtmmifoundationorg
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Who owns TMMireg
sect The TMMi Foundation
ndash Maintains ownership of the TMMireg model and the management of the ongoingintellectual property rights
ndash Creation and management of an independent central data repository and provisionof industry and other analysis services
ndash Provision of an independent accreditation process for TMMireg assessment methodsand Assessor resources
ndash Provision of a public forum of interested parties
sect For more information visit wwwtmmifoundationorg
28
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
TMMireg Foundation Contacts
29
sect Full information on the work of the Foundation can be found onwwwtmmifoundationorg
sect If interested in becoming a member of the Foundation (currently freeregistration) please email your details to infotmmifoundationorg
sect If your organisation is interested in supporting the work of the Foundationplease email sponsorshiptmmifoundationorg
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Define-redefine
14
sect Based on the process improvements identified and agreed - eitherndash define any new methods processes and templatesOrndash where existing processes exist but need updating they are re-defined to bring
them up to date
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Operate
15
sect This is the rollout phasesect Will be at least 80 of the improvement project
sect Agree the improvements to be rolled outndash pilot project or piece of workndash with the support team to see if they have any questionsndash feedback to the process
sect Provide trainingsect Provide mentoring
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Validate
16
sect Benefits realisation stage as well as refinement stagesect Measurement of benefits ndash depends on having a good understanding of the
start positionsect Both tangible and intangiblesect Have the improvement projectrsquos success or exit criteria been metsect Close of the formal improvement project
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Assess and developImprovements
Implementation
17
ImplementationBAU
review
Initiate ndash Measure ndashPrioritise Plan and
DefineIMP Redefine
OperateRO ValidateEvolveVE
Ent
husi
asm
MaximumHype
Actual UseDisillusionment
Practical Benefit
RIP
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus Ltd18
To Measure
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
How to approach measuring
19
sect Agree what message needs to be deliveredsect Keep It Simple Short (KISS) relevant and easy to capture the informationsect Use Goal Question Measure (GQM) approach
ndash Add scales and thresholds
sect Define a data catalogue owners repositoriessect Define the collection and analysis proceduressect Define the presentation and reporting requirements
ndash Produce regular output reports to demonstrate progress ndash give explanations toresults to ensure the right message is delivered
ndash Consistent interpretation
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus Ltd20
Working in Practice
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The Organisation
21
sect Major testing services provider located in Europesect 90 of employees are test professionals (qualified experienced dedicated to
test activities)sect Deliver contract-driven specific test services to all major industry sectorssect Most services deliver system and system integration test levels
ndash 80+ contracts delivering services via a Test Factoryndash They are not usually involved in other parts of test and delivery life cycle(s)ndash Internal ethos of ldquojust do itrdquo instead of ldquomonitor and learnrdquondash A formal restrained communication and information exchange between
organisation and clients
sect The strategic and organisational objective was to achieve TMMi Level 3certification
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The Road Map
22
sect TMMi Baseline assessment undertaken to Level 3 in March 2010ndash Numerous process gaps evidenced in all process areas for both levels 2 amp 3
sect Process Improvement project initiatedndash Involved senior test and quality management resources to demonstrate
commitment to the rest of the organisationndash Aggressive timescales to define and implement improvements were setndash Assessment provider undertook quality assurance and TMMi progress checkpoint
activities
sect Robust approach in line with the IMPROVE modelsect An implementation plan ensured the organisation had implemented in time to
be re-assessed in December 2010ndash TMMi Level 3 certification achieved with no issues
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Oh to measure
23
sect Strategic objective only to achieve certificationndash No stated requirement to measure before and after implementation of changendash Measure only because TMMi says so
sect At March 2010 only 2 Key Performance Indicators were definedndash Service Level Agreements stated ldquo must deliver a) on time and b) on budgetrdquondash No declared interest in process or product quality
sect No requirement to identify cost of process weaknessessect Initial assessment data identified product quality ldquoKPIrdquo as well
ndash Organisation is not paid if ANY severity 1 issues found by client that SHOULD havebeen found before
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Measurement as implemented
24
sect Using GQM defined 6 internal KPIs with thresholdsndash Requirements Coverage ndash requirements planned to be tested amp successfully testedndash Quality of Incidents found ndash what level of duplication rejection was evidencedndash Incidents per Test Cases ndash good test cases finding incidentsndash Incident detection trend ndash when in the life cycle does testing find the incidentsndash Actual vs Planned test effort and deviation levelndash Test effort per test case
sect Added customer satisfaction datasect Retained on time and budget contractual SLArsquossect Integrated into test monitoring control and reporting within delivery projects
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Andhellip
25
sect The organisation achieved certification on timesect They also used the KPI data to observe that in the first 6 months of 2011
ndash Test case specificationsect Efficiency had increased 8 between January and June
sect Effectiveness had increased 11 between January and June
ndash Test case executionsect Efficiency had increased 5 between January and June
sect Effectiveness had increased 5 between January and June
ndash Quality of test cases had increased 9ndash Quality of issues detected had increased 6They also think that client relationships have become more open and positive and thepost delivery ldquonegotiationsrdquo are much easier
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
And the message is
26
sect With no requirement for measurement beyond the needs of a processreference modelndash A few well defined easy to understand KPIsndash Implemented collected analysed and reported quicklyndash Demonstrated to all parties the improvements that were accruing following
implemented change
It is good to measure (but make it easy to do so)
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd27
Thank you
Brian Wellsinfoexperimentuscom
+44(0)207 871 2300
wwwexperimentuscom
wwwtmmifoundationorg
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Who owns TMMireg
sect The TMMi Foundation
ndash Maintains ownership of the TMMireg model and the management of the ongoingintellectual property rights
ndash Creation and management of an independent central data repository and provisionof industry and other analysis services
ndash Provision of an independent accreditation process for TMMireg assessment methodsand Assessor resources
ndash Provision of a public forum of interested parties
sect For more information visit wwwtmmifoundationorg
28
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
TMMireg Foundation Contacts
29
sect Full information on the work of the Foundation can be found onwwwtmmifoundationorg
sect If interested in becoming a member of the Foundation (currently freeregistration) please email your details to infotmmifoundationorg
sect If your organisation is interested in supporting the work of the Foundationplease email sponsorshiptmmifoundationorg
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Operate
15
sect This is the rollout phasesect Will be at least 80 of the improvement project
sect Agree the improvements to be rolled outndash pilot project or piece of workndash with the support team to see if they have any questionsndash feedback to the process
sect Provide trainingsect Provide mentoring
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Validate
16
sect Benefits realisation stage as well as refinement stagesect Measurement of benefits ndash depends on having a good understanding of the
start positionsect Both tangible and intangiblesect Have the improvement projectrsquos success or exit criteria been metsect Close of the formal improvement project
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Assess and developImprovements
Implementation
17
ImplementationBAU
review
Initiate ndash Measure ndashPrioritise Plan and
DefineIMP Redefine
OperateRO ValidateEvolveVE
Ent
husi
asm
MaximumHype
Actual UseDisillusionment
Practical Benefit
RIP
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus Ltd18
To Measure
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
How to approach measuring
19
sect Agree what message needs to be deliveredsect Keep It Simple Short (KISS) relevant and easy to capture the informationsect Use Goal Question Measure (GQM) approach
ndash Add scales and thresholds
sect Define a data catalogue owners repositoriessect Define the collection and analysis proceduressect Define the presentation and reporting requirements
ndash Produce regular output reports to demonstrate progress ndash give explanations toresults to ensure the right message is delivered
ndash Consistent interpretation
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus Ltd20
Working in Practice
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The Organisation
21
sect Major testing services provider located in Europesect 90 of employees are test professionals (qualified experienced dedicated to
test activities)sect Deliver contract-driven specific test services to all major industry sectorssect Most services deliver system and system integration test levels
ndash 80+ contracts delivering services via a Test Factoryndash They are not usually involved in other parts of test and delivery life cycle(s)ndash Internal ethos of ldquojust do itrdquo instead of ldquomonitor and learnrdquondash A formal restrained communication and information exchange between
organisation and clients
sect The strategic and organisational objective was to achieve TMMi Level 3certification
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The Road Map
22
sect TMMi Baseline assessment undertaken to Level 3 in March 2010ndash Numerous process gaps evidenced in all process areas for both levels 2 amp 3
sect Process Improvement project initiatedndash Involved senior test and quality management resources to demonstrate
commitment to the rest of the organisationndash Aggressive timescales to define and implement improvements were setndash Assessment provider undertook quality assurance and TMMi progress checkpoint
activities
sect Robust approach in line with the IMPROVE modelsect An implementation plan ensured the organisation had implemented in time to
be re-assessed in December 2010ndash TMMi Level 3 certification achieved with no issues
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Oh to measure
23
sect Strategic objective only to achieve certificationndash No stated requirement to measure before and after implementation of changendash Measure only because TMMi says so
sect At March 2010 only 2 Key Performance Indicators were definedndash Service Level Agreements stated ldquo must deliver a) on time and b) on budgetrdquondash No declared interest in process or product quality
sect No requirement to identify cost of process weaknessessect Initial assessment data identified product quality ldquoKPIrdquo as well
ndash Organisation is not paid if ANY severity 1 issues found by client that SHOULD havebeen found before
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Measurement as implemented
24
sect Using GQM defined 6 internal KPIs with thresholdsndash Requirements Coverage ndash requirements planned to be tested amp successfully testedndash Quality of Incidents found ndash what level of duplication rejection was evidencedndash Incidents per Test Cases ndash good test cases finding incidentsndash Incident detection trend ndash when in the life cycle does testing find the incidentsndash Actual vs Planned test effort and deviation levelndash Test effort per test case
sect Added customer satisfaction datasect Retained on time and budget contractual SLArsquossect Integrated into test monitoring control and reporting within delivery projects
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Andhellip
25
sect The organisation achieved certification on timesect They also used the KPI data to observe that in the first 6 months of 2011
ndash Test case specificationsect Efficiency had increased 8 between January and June
sect Effectiveness had increased 11 between January and June
ndash Test case executionsect Efficiency had increased 5 between January and June
sect Effectiveness had increased 5 between January and June
ndash Quality of test cases had increased 9ndash Quality of issues detected had increased 6They also think that client relationships have become more open and positive and thepost delivery ldquonegotiationsrdquo are much easier
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
And the message is
26
sect With no requirement for measurement beyond the needs of a processreference modelndash A few well defined easy to understand KPIsndash Implemented collected analysed and reported quicklyndash Demonstrated to all parties the improvements that were accruing following
implemented change
It is good to measure (but make it easy to do so)
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd27
Thank you
Brian Wellsinfoexperimentuscom
+44(0)207 871 2300
wwwexperimentuscom
wwwtmmifoundationorg
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Who owns TMMireg
sect The TMMi Foundation
ndash Maintains ownership of the TMMireg model and the management of the ongoingintellectual property rights
ndash Creation and management of an independent central data repository and provisionof industry and other analysis services
ndash Provision of an independent accreditation process for TMMireg assessment methodsand Assessor resources
ndash Provision of a public forum of interested parties
sect For more information visit wwwtmmifoundationorg
28
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
TMMireg Foundation Contacts
29
sect Full information on the work of the Foundation can be found onwwwtmmifoundationorg
sect If interested in becoming a member of the Foundation (currently freeregistration) please email your details to infotmmifoundationorg
sect If your organisation is interested in supporting the work of the Foundationplease email sponsorshiptmmifoundationorg
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Validate
16
sect Benefits realisation stage as well as refinement stagesect Measurement of benefits ndash depends on having a good understanding of the
start positionsect Both tangible and intangiblesect Have the improvement projectrsquos success or exit criteria been metsect Close of the formal improvement project
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Assess and developImprovements
Implementation
17
ImplementationBAU
review
Initiate ndash Measure ndashPrioritise Plan and
DefineIMP Redefine
OperateRO ValidateEvolveVE
Ent
husi
asm
MaximumHype
Actual UseDisillusionment
Practical Benefit
RIP
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus Ltd18
To Measure
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
How to approach measuring
19
sect Agree what message needs to be deliveredsect Keep It Simple Short (KISS) relevant and easy to capture the informationsect Use Goal Question Measure (GQM) approach
ndash Add scales and thresholds
sect Define a data catalogue owners repositoriessect Define the collection and analysis proceduressect Define the presentation and reporting requirements
ndash Produce regular output reports to demonstrate progress ndash give explanations toresults to ensure the right message is delivered
ndash Consistent interpretation
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus Ltd20
Working in Practice
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The Organisation
21
sect Major testing services provider located in Europesect 90 of employees are test professionals (qualified experienced dedicated to
test activities)sect Deliver contract-driven specific test services to all major industry sectorssect Most services deliver system and system integration test levels
ndash 80+ contracts delivering services via a Test Factoryndash They are not usually involved in other parts of test and delivery life cycle(s)ndash Internal ethos of ldquojust do itrdquo instead of ldquomonitor and learnrdquondash A formal restrained communication and information exchange between
organisation and clients
sect The strategic and organisational objective was to achieve TMMi Level 3certification
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The Road Map
22
sect TMMi Baseline assessment undertaken to Level 3 in March 2010ndash Numerous process gaps evidenced in all process areas for both levels 2 amp 3
sect Process Improvement project initiatedndash Involved senior test and quality management resources to demonstrate
commitment to the rest of the organisationndash Aggressive timescales to define and implement improvements were setndash Assessment provider undertook quality assurance and TMMi progress checkpoint
activities
sect Robust approach in line with the IMPROVE modelsect An implementation plan ensured the organisation had implemented in time to
be re-assessed in December 2010ndash TMMi Level 3 certification achieved with no issues
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Oh to measure
23
sect Strategic objective only to achieve certificationndash No stated requirement to measure before and after implementation of changendash Measure only because TMMi says so
sect At March 2010 only 2 Key Performance Indicators were definedndash Service Level Agreements stated ldquo must deliver a) on time and b) on budgetrdquondash No declared interest in process or product quality
sect No requirement to identify cost of process weaknessessect Initial assessment data identified product quality ldquoKPIrdquo as well
ndash Organisation is not paid if ANY severity 1 issues found by client that SHOULD havebeen found before
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Measurement as implemented
24
sect Using GQM defined 6 internal KPIs with thresholdsndash Requirements Coverage ndash requirements planned to be tested amp successfully testedndash Quality of Incidents found ndash what level of duplication rejection was evidencedndash Incidents per Test Cases ndash good test cases finding incidentsndash Incident detection trend ndash when in the life cycle does testing find the incidentsndash Actual vs Planned test effort and deviation levelndash Test effort per test case
sect Added customer satisfaction datasect Retained on time and budget contractual SLArsquossect Integrated into test monitoring control and reporting within delivery projects
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Andhellip
25
sect The organisation achieved certification on timesect They also used the KPI data to observe that in the first 6 months of 2011
ndash Test case specificationsect Efficiency had increased 8 between January and June
sect Effectiveness had increased 11 between January and June
ndash Test case executionsect Efficiency had increased 5 between January and June
sect Effectiveness had increased 5 between January and June
ndash Quality of test cases had increased 9ndash Quality of issues detected had increased 6They also think that client relationships have become more open and positive and thepost delivery ldquonegotiationsrdquo are much easier
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
And the message is
26
sect With no requirement for measurement beyond the needs of a processreference modelndash A few well defined easy to understand KPIsndash Implemented collected analysed and reported quicklyndash Demonstrated to all parties the improvements that were accruing following
implemented change
It is good to measure (but make it easy to do so)
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd27
Thank you
Brian Wellsinfoexperimentuscom
+44(0)207 871 2300
wwwexperimentuscom
wwwtmmifoundationorg
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Who owns TMMireg
sect The TMMi Foundation
ndash Maintains ownership of the TMMireg model and the management of the ongoingintellectual property rights
ndash Creation and management of an independent central data repository and provisionof industry and other analysis services
ndash Provision of an independent accreditation process for TMMireg assessment methodsand Assessor resources
ndash Provision of a public forum of interested parties
sect For more information visit wwwtmmifoundationorg
28
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
TMMireg Foundation Contacts
29
sect Full information on the work of the Foundation can be found onwwwtmmifoundationorg
sect If interested in becoming a member of the Foundation (currently freeregistration) please email your details to infotmmifoundationorg
sect If your organisation is interested in supporting the work of the Foundationplease email sponsorshiptmmifoundationorg
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Assess and developImprovements
Implementation
17
ImplementationBAU
review
Initiate ndash Measure ndashPrioritise Plan and
DefineIMP Redefine
OperateRO ValidateEvolveVE
Ent
husi
asm
MaximumHype
Actual UseDisillusionment
Practical Benefit
RIP
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus Ltd18
To Measure
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
How to approach measuring
19
sect Agree what message needs to be deliveredsect Keep It Simple Short (KISS) relevant and easy to capture the informationsect Use Goal Question Measure (GQM) approach
ndash Add scales and thresholds
sect Define a data catalogue owners repositoriessect Define the collection and analysis proceduressect Define the presentation and reporting requirements
ndash Produce regular output reports to demonstrate progress ndash give explanations toresults to ensure the right message is delivered
ndash Consistent interpretation
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus Ltd20
Working in Practice
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The Organisation
21
sect Major testing services provider located in Europesect 90 of employees are test professionals (qualified experienced dedicated to
test activities)sect Deliver contract-driven specific test services to all major industry sectorssect Most services deliver system and system integration test levels
ndash 80+ contracts delivering services via a Test Factoryndash They are not usually involved in other parts of test and delivery life cycle(s)ndash Internal ethos of ldquojust do itrdquo instead of ldquomonitor and learnrdquondash A formal restrained communication and information exchange between
organisation and clients
sect The strategic and organisational objective was to achieve TMMi Level 3certification
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The Road Map
22
sect TMMi Baseline assessment undertaken to Level 3 in March 2010ndash Numerous process gaps evidenced in all process areas for both levels 2 amp 3
sect Process Improvement project initiatedndash Involved senior test and quality management resources to demonstrate
commitment to the rest of the organisationndash Aggressive timescales to define and implement improvements were setndash Assessment provider undertook quality assurance and TMMi progress checkpoint
activities
sect Robust approach in line with the IMPROVE modelsect An implementation plan ensured the organisation had implemented in time to
be re-assessed in December 2010ndash TMMi Level 3 certification achieved with no issues
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Oh to measure
23
sect Strategic objective only to achieve certificationndash No stated requirement to measure before and after implementation of changendash Measure only because TMMi says so
sect At March 2010 only 2 Key Performance Indicators were definedndash Service Level Agreements stated ldquo must deliver a) on time and b) on budgetrdquondash No declared interest in process or product quality
sect No requirement to identify cost of process weaknessessect Initial assessment data identified product quality ldquoKPIrdquo as well
ndash Organisation is not paid if ANY severity 1 issues found by client that SHOULD havebeen found before
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Measurement as implemented
24
sect Using GQM defined 6 internal KPIs with thresholdsndash Requirements Coverage ndash requirements planned to be tested amp successfully testedndash Quality of Incidents found ndash what level of duplication rejection was evidencedndash Incidents per Test Cases ndash good test cases finding incidentsndash Incident detection trend ndash when in the life cycle does testing find the incidentsndash Actual vs Planned test effort and deviation levelndash Test effort per test case
sect Added customer satisfaction datasect Retained on time and budget contractual SLArsquossect Integrated into test monitoring control and reporting within delivery projects
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Andhellip
25
sect The organisation achieved certification on timesect They also used the KPI data to observe that in the first 6 months of 2011
ndash Test case specificationsect Efficiency had increased 8 between January and June
sect Effectiveness had increased 11 between January and June
ndash Test case executionsect Efficiency had increased 5 between January and June
sect Effectiveness had increased 5 between January and June
ndash Quality of test cases had increased 9ndash Quality of issues detected had increased 6They also think that client relationships have become more open and positive and thepost delivery ldquonegotiationsrdquo are much easier
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
And the message is
26
sect With no requirement for measurement beyond the needs of a processreference modelndash A few well defined easy to understand KPIsndash Implemented collected analysed and reported quicklyndash Demonstrated to all parties the improvements that were accruing following
implemented change
It is good to measure (but make it easy to do so)
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd27
Thank you
Brian Wellsinfoexperimentuscom
+44(0)207 871 2300
wwwexperimentuscom
wwwtmmifoundationorg
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Who owns TMMireg
sect The TMMi Foundation
ndash Maintains ownership of the TMMireg model and the management of the ongoingintellectual property rights
ndash Creation and management of an independent central data repository and provisionof industry and other analysis services
ndash Provision of an independent accreditation process for TMMireg assessment methodsand Assessor resources
ndash Provision of a public forum of interested parties
sect For more information visit wwwtmmifoundationorg
28
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
TMMireg Foundation Contacts
29
sect Full information on the work of the Foundation can be found onwwwtmmifoundationorg
sect If interested in becoming a member of the Foundation (currently freeregistration) please email your details to infotmmifoundationorg
sect If your organisation is interested in supporting the work of the Foundationplease email sponsorshiptmmifoundationorg
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus Ltd18
To Measure
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
How to approach measuring
19
sect Agree what message needs to be deliveredsect Keep It Simple Short (KISS) relevant and easy to capture the informationsect Use Goal Question Measure (GQM) approach
ndash Add scales and thresholds
sect Define a data catalogue owners repositoriessect Define the collection and analysis proceduressect Define the presentation and reporting requirements
ndash Produce regular output reports to demonstrate progress ndash give explanations toresults to ensure the right message is delivered
ndash Consistent interpretation
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus Ltd20
Working in Practice
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The Organisation
21
sect Major testing services provider located in Europesect 90 of employees are test professionals (qualified experienced dedicated to
test activities)sect Deliver contract-driven specific test services to all major industry sectorssect Most services deliver system and system integration test levels
ndash 80+ contracts delivering services via a Test Factoryndash They are not usually involved in other parts of test and delivery life cycle(s)ndash Internal ethos of ldquojust do itrdquo instead of ldquomonitor and learnrdquondash A formal restrained communication and information exchange between
organisation and clients
sect The strategic and organisational objective was to achieve TMMi Level 3certification
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The Road Map
22
sect TMMi Baseline assessment undertaken to Level 3 in March 2010ndash Numerous process gaps evidenced in all process areas for both levels 2 amp 3
sect Process Improvement project initiatedndash Involved senior test and quality management resources to demonstrate
commitment to the rest of the organisationndash Aggressive timescales to define and implement improvements were setndash Assessment provider undertook quality assurance and TMMi progress checkpoint
activities
sect Robust approach in line with the IMPROVE modelsect An implementation plan ensured the organisation had implemented in time to
be re-assessed in December 2010ndash TMMi Level 3 certification achieved with no issues
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Oh to measure
23
sect Strategic objective only to achieve certificationndash No stated requirement to measure before and after implementation of changendash Measure only because TMMi says so
sect At March 2010 only 2 Key Performance Indicators were definedndash Service Level Agreements stated ldquo must deliver a) on time and b) on budgetrdquondash No declared interest in process or product quality
sect No requirement to identify cost of process weaknessessect Initial assessment data identified product quality ldquoKPIrdquo as well
ndash Organisation is not paid if ANY severity 1 issues found by client that SHOULD havebeen found before
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Measurement as implemented
24
sect Using GQM defined 6 internal KPIs with thresholdsndash Requirements Coverage ndash requirements planned to be tested amp successfully testedndash Quality of Incidents found ndash what level of duplication rejection was evidencedndash Incidents per Test Cases ndash good test cases finding incidentsndash Incident detection trend ndash when in the life cycle does testing find the incidentsndash Actual vs Planned test effort and deviation levelndash Test effort per test case
sect Added customer satisfaction datasect Retained on time and budget contractual SLArsquossect Integrated into test monitoring control and reporting within delivery projects
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Andhellip
25
sect The organisation achieved certification on timesect They also used the KPI data to observe that in the first 6 months of 2011
ndash Test case specificationsect Efficiency had increased 8 between January and June
sect Effectiveness had increased 11 between January and June
ndash Test case executionsect Efficiency had increased 5 between January and June
sect Effectiveness had increased 5 between January and June
ndash Quality of test cases had increased 9ndash Quality of issues detected had increased 6They also think that client relationships have become more open and positive and thepost delivery ldquonegotiationsrdquo are much easier
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
And the message is
26
sect With no requirement for measurement beyond the needs of a processreference modelndash A few well defined easy to understand KPIsndash Implemented collected analysed and reported quicklyndash Demonstrated to all parties the improvements that were accruing following
implemented change
It is good to measure (but make it easy to do so)
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd27
Thank you
Brian Wellsinfoexperimentuscom
+44(0)207 871 2300
wwwexperimentuscom
wwwtmmifoundationorg
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Who owns TMMireg
sect The TMMi Foundation
ndash Maintains ownership of the TMMireg model and the management of the ongoingintellectual property rights
ndash Creation and management of an independent central data repository and provisionof industry and other analysis services
ndash Provision of an independent accreditation process for TMMireg assessment methodsand Assessor resources
ndash Provision of a public forum of interested parties
sect For more information visit wwwtmmifoundationorg
28
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
TMMireg Foundation Contacts
29
sect Full information on the work of the Foundation can be found onwwwtmmifoundationorg
sect If interested in becoming a member of the Foundation (currently freeregistration) please email your details to infotmmifoundationorg
sect If your organisation is interested in supporting the work of the Foundationplease email sponsorshiptmmifoundationorg
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
How to approach measuring
19
sect Agree what message needs to be deliveredsect Keep It Simple Short (KISS) relevant and easy to capture the informationsect Use Goal Question Measure (GQM) approach
ndash Add scales and thresholds
sect Define a data catalogue owners repositoriessect Define the collection and analysis proceduressect Define the presentation and reporting requirements
ndash Produce regular output reports to demonstrate progress ndash give explanations toresults to ensure the right message is delivered
ndash Consistent interpretation
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus Ltd20
Working in Practice
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The Organisation
21
sect Major testing services provider located in Europesect 90 of employees are test professionals (qualified experienced dedicated to
test activities)sect Deliver contract-driven specific test services to all major industry sectorssect Most services deliver system and system integration test levels
ndash 80+ contracts delivering services via a Test Factoryndash They are not usually involved in other parts of test and delivery life cycle(s)ndash Internal ethos of ldquojust do itrdquo instead of ldquomonitor and learnrdquondash A formal restrained communication and information exchange between
organisation and clients
sect The strategic and organisational objective was to achieve TMMi Level 3certification
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The Road Map
22
sect TMMi Baseline assessment undertaken to Level 3 in March 2010ndash Numerous process gaps evidenced in all process areas for both levels 2 amp 3
sect Process Improvement project initiatedndash Involved senior test and quality management resources to demonstrate
commitment to the rest of the organisationndash Aggressive timescales to define and implement improvements were setndash Assessment provider undertook quality assurance and TMMi progress checkpoint
activities
sect Robust approach in line with the IMPROVE modelsect An implementation plan ensured the organisation had implemented in time to
be re-assessed in December 2010ndash TMMi Level 3 certification achieved with no issues
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Oh to measure
23
sect Strategic objective only to achieve certificationndash No stated requirement to measure before and after implementation of changendash Measure only because TMMi says so
sect At March 2010 only 2 Key Performance Indicators were definedndash Service Level Agreements stated ldquo must deliver a) on time and b) on budgetrdquondash No declared interest in process or product quality
sect No requirement to identify cost of process weaknessessect Initial assessment data identified product quality ldquoKPIrdquo as well
ndash Organisation is not paid if ANY severity 1 issues found by client that SHOULD havebeen found before
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Measurement as implemented
24
sect Using GQM defined 6 internal KPIs with thresholdsndash Requirements Coverage ndash requirements planned to be tested amp successfully testedndash Quality of Incidents found ndash what level of duplication rejection was evidencedndash Incidents per Test Cases ndash good test cases finding incidentsndash Incident detection trend ndash when in the life cycle does testing find the incidentsndash Actual vs Planned test effort and deviation levelndash Test effort per test case
sect Added customer satisfaction datasect Retained on time and budget contractual SLArsquossect Integrated into test monitoring control and reporting within delivery projects
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Andhellip
25
sect The organisation achieved certification on timesect They also used the KPI data to observe that in the first 6 months of 2011
ndash Test case specificationsect Efficiency had increased 8 between January and June
sect Effectiveness had increased 11 between January and June
ndash Test case executionsect Efficiency had increased 5 between January and June
sect Effectiveness had increased 5 between January and June
ndash Quality of test cases had increased 9ndash Quality of issues detected had increased 6They also think that client relationships have become more open and positive and thepost delivery ldquonegotiationsrdquo are much easier
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
And the message is
26
sect With no requirement for measurement beyond the needs of a processreference modelndash A few well defined easy to understand KPIsndash Implemented collected analysed and reported quicklyndash Demonstrated to all parties the improvements that were accruing following
implemented change
It is good to measure (but make it easy to do so)
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd27
Thank you
Brian Wellsinfoexperimentuscom
+44(0)207 871 2300
wwwexperimentuscom
wwwtmmifoundationorg
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Who owns TMMireg
sect The TMMi Foundation
ndash Maintains ownership of the TMMireg model and the management of the ongoingintellectual property rights
ndash Creation and management of an independent central data repository and provisionof industry and other analysis services
ndash Provision of an independent accreditation process for TMMireg assessment methodsand Assessor resources
ndash Provision of a public forum of interested parties
sect For more information visit wwwtmmifoundationorg
28
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
TMMireg Foundation Contacts
29
sect Full information on the work of the Foundation can be found onwwwtmmifoundationorg
sect If interested in becoming a member of the Foundation (currently freeregistration) please email your details to infotmmifoundationorg
sect If your organisation is interested in supporting the work of the Foundationplease email sponsorshiptmmifoundationorg
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus Ltd20
Working in Practice
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The Organisation
21
sect Major testing services provider located in Europesect 90 of employees are test professionals (qualified experienced dedicated to
test activities)sect Deliver contract-driven specific test services to all major industry sectorssect Most services deliver system and system integration test levels
ndash 80+ contracts delivering services via a Test Factoryndash They are not usually involved in other parts of test and delivery life cycle(s)ndash Internal ethos of ldquojust do itrdquo instead of ldquomonitor and learnrdquondash A formal restrained communication and information exchange between
organisation and clients
sect The strategic and organisational objective was to achieve TMMi Level 3certification
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The Road Map
22
sect TMMi Baseline assessment undertaken to Level 3 in March 2010ndash Numerous process gaps evidenced in all process areas for both levels 2 amp 3
sect Process Improvement project initiatedndash Involved senior test and quality management resources to demonstrate
commitment to the rest of the organisationndash Aggressive timescales to define and implement improvements were setndash Assessment provider undertook quality assurance and TMMi progress checkpoint
activities
sect Robust approach in line with the IMPROVE modelsect An implementation plan ensured the organisation had implemented in time to
be re-assessed in December 2010ndash TMMi Level 3 certification achieved with no issues
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Oh to measure
23
sect Strategic objective only to achieve certificationndash No stated requirement to measure before and after implementation of changendash Measure only because TMMi says so
sect At March 2010 only 2 Key Performance Indicators were definedndash Service Level Agreements stated ldquo must deliver a) on time and b) on budgetrdquondash No declared interest in process or product quality
sect No requirement to identify cost of process weaknessessect Initial assessment data identified product quality ldquoKPIrdquo as well
ndash Organisation is not paid if ANY severity 1 issues found by client that SHOULD havebeen found before
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Measurement as implemented
24
sect Using GQM defined 6 internal KPIs with thresholdsndash Requirements Coverage ndash requirements planned to be tested amp successfully testedndash Quality of Incidents found ndash what level of duplication rejection was evidencedndash Incidents per Test Cases ndash good test cases finding incidentsndash Incident detection trend ndash when in the life cycle does testing find the incidentsndash Actual vs Planned test effort and deviation levelndash Test effort per test case
sect Added customer satisfaction datasect Retained on time and budget contractual SLArsquossect Integrated into test monitoring control and reporting within delivery projects
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Andhellip
25
sect The organisation achieved certification on timesect They also used the KPI data to observe that in the first 6 months of 2011
ndash Test case specificationsect Efficiency had increased 8 between January and June
sect Effectiveness had increased 11 between January and June
ndash Test case executionsect Efficiency had increased 5 between January and June
sect Effectiveness had increased 5 between January and June
ndash Quality of test cases had increased 9ndash Quality of issues detected had increased 6They also think that client relationships have become more open and positive and thepost delivery ldquonegotiationsrdquo are much easier
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
And the message is
26
sect With no requirement for measurement beyond the needs of a processreference modelndash A few well defined easy to understand KPIsndash Implemented collected analysed and reported quicklyndash Demonstrated to all parties the improvements that were accruing following
implemented change
It is good to measure (but make it easy to do so)
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd27
Thank you
Brian Wellsinfoexperimentuscom
+44(0)207 871 2300
wwwexperimentuscom
wwwtmmifoundationorg
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Who owns TMMireg
sect The TMMi Foundation
ndash Maintains ownership of the TMMireg model and the management of the ongoingintellectual property rights
ndash Creation and management of an independent central data repository and provisionof industry and other analysis services
ndash Provision of an independent accreditation process for TMMireg assessment methodsand Assessor resources
ndash Provision of a public forum of interested parties
sect For more information visit wwwtmmifoundationorg
28
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
TMMireg Foundation Contacts
29
sect Full information on the work of the Foundation can be found onwwwtmmifoundationorg
sect If interested in becoming a member of the Foundation (currently freeregistration) please email your details to infotmmifoundationorg
sect If your organisation is interested in supporting the work of the Foundationplease email sponsorshiptmmifoundationorg
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The Organisation
21
sect Major testing services provider located in Europesect 90 of employees are test professionals (qualified experienced dedicated to
test activities)sect Deliver contract-driven specific test services to all major industry sectorssect Most services deliver system and system integration test levels
ndash 80+ contracts delivering services via a Test Factoryndash They are not usually involved in other parts of test and delivery life cycle(s)ndash Internal ethos of ldquojust do itrdquo instead of ldquomonitor and learnrdquondash A formal restrained communication and information exchange between
organisation and clients
sect The strategic and organisational objective was to achieve TMMi Level 3certification
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The Road Map
22
sect TMMi Baseline assessment undertaken to Level 3 in March 2010ndash Numerous process gaps evidenced in all process areas for both levels 2 amp 3
sect Process Improvement project initiatedndash Involved senior test and quality management resources to demonstrate
commitment to the rest of the organisationndash Aggressive timescales to define and implement improvements were setndash Assessment provider undertook quality assurance and TMMi progress checkpoint
activities
sect Robust approach in line with the IMPROVE modelsect An implementation plan ensured the organisation had implemented in time to
be re-assessed in December 2010ndash TMMi Level 3 certification achieved with no issues
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Oh to measure
23
sect Strategic objective only to achieve certificationndash No stated requirement to measure before and after implementation of changendash Measure only because TMMi says so
sect At March 2010 only 2 Key Performance Indicators were definedndash Service Level Agreements stated ldquo must deliver a) on time and b) on budgetrdquondash No declared interest in process or product quality
sect No requirement to identify cost of process weaknessessect Initial assessment data identified product quality ldquoKPIrdquo as well
ndash Organisation is not paid if ANY severity 1 issues found by client that SHOULD havebeen found before
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Measurement as implemented
24
sect Using GQM defined 6 internal KPIs with thresholdsndash Requirements Coverage ndash requirements planned to be tested amp successfully testedndash Quality of Incidents found ndash what level of duplication rejection was evidencedndash Incidents per Test Cases ndash good test cases finding incidentsndash Incident detection trend ndash when in the life cycle does testing find the incidentsndash Actual vs Planned test effort and deviation levelndash Test effort per test case
sect Added customer satisfaction datasect Retained on time and budget contractual SLArsquossect Integrated into test monitoring control and reporting within delivery projects
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Andhellip
25
sect The organisation achieved certification on timesect They also used the KPI data to observe that in the first 6 months of 2011
ndash Test case specificationsect Efficiency had increased 8 between January and June
sect Effectiveness had increased 11 between January and June
ndash Test case executionsect Efficiency had increased 5 between January and June
sect Effectiveness had increased 5 between January and June
ndash Quality of test cases had increased 9ndash Quality of issues detected had increased 6They also think that client relationships have become more open and positive and thepost delivery ldquonegotiationsrdquo are much easier
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
And the message is
26
sect With no requirement for measurement beyond the needs of a processreference modelndash A few well defined easy to understand KPIsndash Implemented collected analysed and reported quicklyndash Demonstrated to all parties the improvements that were accruing following
implemented change
It is good to measure (but make it easy to do so)
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd27
Thank you
Brian Wellsinfoexperimentuscom
+44(0)207 871 2300
wwwexperimentuscom
wwwtmmifoundationorg
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Who owns TMMireg
sect The TMMi Foundation
ndash Maintains ownership of the TMMireg model and the management of the ongoingintellectual property rights
ndash Creation and management of an independent central data repository and provisionof industry and other analysis services
ndash Provision of an independent accreditation process for TMMireg assessment methodsand Assessor resources
ndash Provision of a public forum of interested parties
sect For more information visit wwwtmmifoundationorg
28
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
TMMireg Foundation Contacts
29
sect Full information on the work of the Foundation can be found onwwwtmmifoundationorg
sect If interested in becoming a member of the Foundation (currently freeregistration) please email your details to infotmmifoundationorg
sect If your organisation is interested in supporting the work of the Foundationplease email sponsorshiptmmifoundationorg
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
The Road Map
22
sect TMMi Baseline assessment undertaken to Level 3 in March 2010ndash Numerous process gaps evidenced in all process areas for both levels 2 amp 3
sect Process Improvement project initiatedndash Involved senior test and quality management resources to demonstrate
commitment to the rest of the organisationndash Aggressive timescales to define and implement improvements were setndash Assessment provider undertook quality assurance and TMMi progress checkpoint
activities
sect Robust approach in line with the IMPROVE modelsect An implementation plan ensured the organisation had implemented in time to
be re-assessed in December 2010ndash TMMi Level 3 certification achieved with no issues
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Oh to measure
23
sect Strategic objective only to achieve certificationndash No stated requirement to measure before and after implementation of changendash Measure only because TMMi says so
sect At March 2010 only 2 Key Performance Indicators were definedndash Service Level Agreements stated ldquo must deliver a) on time and b) on budgetrdquondash No declared interest in process or product quality
sect No requirement to identify cost of process weaknessessect Initial assessment data identified product quality ldquoKPIrdquo as well
ndash Organisation is not paid if ANY severity 1 issues found by client that SHOULD havebeen found before
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Measurement as implemented
24
sect Using GQM defined 6 internal KPIs with thresholdsndash Requirements Coverage ndash requirements planned to be tested amp successfully testedndash Quality of Incidents found ndash what level of duplication rejection was evidencedndash Incidents per Test Cases ndash good test cases finding incidentsndash Incident detection trend ndash when in the life cycle does testing find the incidentsndash Actual vs Planned test effort and deviation levelndash Test effort per test case
sect Added customer satisfaction datasect Retained on time and budget contractual SLArsquossect Integrated into test monitoring control and reporting within delivery projects
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Andhellip
25
sect The organisation achieved certification on timesect They also used the KPI data to observe that in the first 6 months of 2011
ndash Test case specificationsect Efficiency had increased 8 between January and June
sect Effectiveness had increased 11 between January and June
ndash Test case executionsect Efficiency had increased 5 between January and June
sect Effectiveness had increased 5 between January and June
ndash Quality of test cases had increased 9ndash Quality of issues detected had increased 6They also think that client relationships have become more open and positive and thepost delivery ldquonegotiationsrdquo are much easier
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
And the message is
26
sect With no requirement for measurement beyond the needs of a processreference modelndash A few well defined easy to understand KPIsndash Implemented collected analysed and reported quicklyndash Demonstrated to all parties the improvements that were accruing following
implemented change
It is good to measure (but make it easy to do so)
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd27
Thank you
Brian Wellsinfoexperimentuscom
+44(0)207 871 2300
wwwexperimentuscom
wwwtmmifoundationorg
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Who owns TMMireg
sect The TMMi Foundation
ndash Maintains ownership of the TMMireg model and the management of the ongoingintellectual property rights
ndash Creation and management of an independent central data repository and provisionof industry and other analysis services
ndash Provision of an independent accreditation process for TMMireg assessment methodsand Assessor resources
ndash Provision of a public forum of interested parties
sect For more information visit wwwtmmifoundationorg
28
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
TMMireg Foundation Contacts
29
sect Full information on the work of the Foundation can be found onwwwtmmifoundationorg
sect If interested in becoming a member of the Foundation (currently freeregistration) please email your details to infotmmifoundationorg
sect If your organisation is interested in supporting the work of the Foundationplease email sponsorshiptmmifoundationorg
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Oh to measure
23
sect Strategic objective only to achieve certificationndash No stated requirement to measure before and after implementation of changendash Measure only because TMMi says so
sect At March 2010 only 2 Key Performance Indicators were definedndash Service Level Agreements stated ldquo must deliver a) on time and b) on budgetrdquondash No declared interest in process or product quality
sect No requirement to identify cost of process weaknessessect Initial assessment data identified product quality ldquoKPIrdquo as well
ndash Organisation is not paid if ANY severity 1 issues found by client that SHOULD havebeen found before
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Measurement as implemented
24
sect Using GQM defined 6 internal KPIs with thresholdsndash Requirements Coverage ndash requirements planned to be tested amp successfully testedndash Quality of Incidents found ndash what level of duplication rejection was evidencedndash Incidents per Test Cases ndash good test cases finding incidentsndash Incident detection trend ndash when in the life cycle does testing find the incidentsndash Actual vs Planned test effort and deviation levelndash Test effort per test case
sect Added customer satisfaction datasect Retained on time and budget contractual SLArsquossect Integrated into test monitoring control and reporting within delivery projects
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Andhellip
25
sect The organisation achieved certification on timesect They also used the KPI data to observe that in the first 6 months of 2011
ndash Test case specificationsect Efficiency had increased 8 between January and June
sect Effectiveness had increased 11 between January and June
ndash Test case executionsect Efficiency had increased 5 between January and June
sect Effectiveness had increased 5 between January and June
ndash Quality of test cases had increased 9ndash Quality of issues detected had increased 6They also think that client relationships have become more open and positive and thepost delivery ldquonegotiationsrdquo are much easier
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
And the message is
26
sect With no requirement for measurement beyond the needs of a processreference modelndash A few well defined easy to understand KPIsndash Implemented collected analysed and reported quicklyndash Demonstrated to all parties the improvements that were accruing following
implemented change
It is good to measure (but make it easy to do so)
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd27
Thank you
Brian Wellsinfoexperimentuscom
+44(0)207 871 2300
wwwexperimentuscom
wwwtmmifoundationorg
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Who owns TMMireg
sect The TMMi Foundation
ndash Maintains ownership of the TMMireg model and the management of the ongoingintellectual property rights
ndash Creation and management of an independent central data repository and provisionof industry and other analysis services
ndash Provision of an independent accreditation process for TMMireg assessment methodsand Assessor resources
ndash Provision of a public forum of interested parties
sect For more information visit wwwtmmifoundationorg
28
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
TMMireg Foundation Contacts
29
sect Full information on the work of the Foundation can be found onwwwtmmifoundationorg
sect If interested in becoming a member of the Foundation (currently freeregistration) please email your details to infotmmifoundationorg
sect If your organisation is interested in supporting the work of the Foundationplease email sponsorshiptmmifoundationorg
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Measurement as implemented
24
sect Using GQM defined 6 internal KPIs with thresholdsndash Requirements Coverage ndash requirements planned to be tested amp successfully testedndash Quality of Incidents found ndash what level of duplication rejection was evidencedndash Incidents per Test Cases ndash good test cases finding incidentsndash Incident detection trend ndash when in the life cycle does testing find the incidentsndash Actual vs Planned test effort and deviation levelndash Test effort per test case
sect Added customer satisfaction datasect Retained on time and budget contractual SLArsquossect Integrated into test monitoring control and reporting within delivery projects
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Andhellip
25
sect The organisation achieved certification on timesect They also used the KPI data to observe that in the first 6 months of 2011
ndash Test case specificationsect Efficiency had increased 8 between January and June
sect Effectiveness had increased 11 between January and June
ndash Test case executionsect Efficiency had increased 5 between January and June
sect Effectiveness had increased 5 between January and June
ndash Quality of test cases had increased 9ndash Quality of issues detected had increased 6They also think that client relationships have become more open and positive and thepost delivery ldquonegotiationsrdquo are much easier
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
And the message is
26
sect With no requirement for measurement beyond the needs of a processreference modelndash A few well defined easy to understand KPIsndash Implemented collected analysed and reported quicklyndash Demonstrated to all parties the improvements that were accruing following
implemented change
It is good to measure (but make it easy to do so)
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd27
Thank you
Brian Wellsinfoexperimentuscom
+44(0)207 871 2300
wwwexperimentuscom
wwwtmmifoundationorg
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Who owns TMMireg
sect The TMMi Foundation
ndash Maintains ownership of the TMMireg model and the management of the ongoingintellectual property rights
ndash Creation and management of an independent central data repository and provisionof industry and other analysis services
ndash Provision of an independent accreditation process for TMMireg assessment methodsand Assessor resources
ndash Provision of a public forum of interested parties
sect For more information visit wwwtmmifoundationorg
28
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
TMMireg Foundation Contacts
29
sect Full information on the work of the Foundation can be found onwwwtmmifoundationorg
sect If interested in becoming a member of the Foundation (currently freeregistration) please email your details to infotmmifoundationorg
sect If your organisation is interested in supporting the work of the Foundationplease email sponsorshiptmmifoundationorg
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Andhellip
25
sect The organisation achieved certification on timesect They also used the KPI data to observe that in the first 6 months of 2011
ndash Test case specificationsect Efficiency had increased 8 between January and June
sect Effectiveness had increased 11 between January and June
ndash Test case executionsect Efficiency had increased 5 between January and June
sect Effectiveness had increased 5 between January and June
ndash Quality of test cases had increased 9ndash Quality of issues detected had increased 6They also think that client relationships have become more open and positive and thepost delivery ldquonegotiationsrdquo are much easier
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
And the message is
26
sect With no requirement for measurement beyond the needs of a processreference modelndash A few well defined easy to understand KPIsndash Implemented collected analysed and reported quicklyndash Demonstrated to all parties the improvements that were accruing following
implemented change
It is good to measure (but make it easy to do so)
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd27
Thank you
Brian Wellsinfoexperimentuscom
+44(0)207 871 2300
wwwexperimentuscom
wwwtmmifoundationorg
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Who owns TMMireg
sect The TMMi Foundation
ndash Maintains ownership of the TMMireg model and the management of the ongoingintellectual property rights
ndash Creation and management of an independent central data repository and provisionof industry and other analysis services
ndash Provision of an independent accreditation process for TMMireg assessment methodsand Assessor resources
ndash Provision of a public forum of interested parties
sect For more information visit wwwtmmifoundationorg
28
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
TMMireg Foundation Contacts
29
sect Full information on the work of the Foundation can be found onwwwtmmifoundationorg
sect If interested in becoming a member of the Foundation (currently freeregistration) please email your details to infotmmifoundationorg
sect If your organisation is interested in supporting the work of the Foundationplease email sponsorshiptmmifoundationorg
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
And the message is
26
sect With no requirement for measurement beyond the needs of a processreference modelndash A few well defined easy to understand KPIsndash Implemented collected analysed and reported quicklyndash Demonstrated to all parties the improvements that were accruing following
implemented change
It is good to measure (but make it easy to do so)
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd27
Thank you
Brian Wellsinfoexperimentuscom
+44(0)207 871 2300
wwwexperimentuscom
wwwtmmifoundationorg
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Who owns TMMireg
sect The TMMi Foundation
ndash Maintains ownership of the TMMireg model and the management of the ongoingintellectual property rights
ndash Creation and management of an independent central data repository and provisionof industry and other analysis services
ndash Provision of an independent accreditation process for TMMireg assessment methodsand Assessor resources
ndash Provision of a public forum of interested parties
sect For more information visit wwwtmmifoundationorg
28
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
TMMireg Foundation Contacts
29
sect Full information on the work of the Foundation can be found onwwwtmmifoundationorg
sect If interested in becoming a member of the Foundation (currently freeregistration) please email your details to infotmmifoundationorg
sect If your organisation is interested in supporting the work of the Foundationplease email sponsorshiptmmifoundationorg
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd27
Thank you
Brian Wellsinfoexperimentuscom
+44(0)207 871 2300
wwwexperimentuscom
wwwtmmifoundationorg
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Who owns TMMireg
sect The TMMi Foundation
ndash Maintains ownership of the TMMireg model and the management of the ongoingintellectual property rights
ndash Creation and management of an independent central data repository and provisionof industry and other analysis services
ndash Provision of an independent accreditation process for TMMireg assessment methodsand Assessor resources
ndash Provision of a public forum of interested parties
sect For more information visit wwwtmmifoundationorg
28
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
TMMireg Foundation Contacts
29
sect Full information on the work of the Foundation can be found onwwwtmmifoundationorg
sect If interested in becoming a member of the Foundation (currently freeregistration) please email your details to infotmmifoundationorg
sect If your organisation is interested in supporting the work of the Foundationplease email sponsorshiptmmifoundationorg
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
Who owns TMMireg
sect The TMMi Foundation
ndash Maintains ownership of the TMMireg model and the management of the ongoingintellectual property rights
ndash Creation and management of an independent central data repository and provisionof industry and other analysis services
ndash Provision of an independent accreditation process for TMMireg assessment methodsand Assessor resources
ndash Provision of a public forum of interested parties
sect For more information visit wwwtmmifoundationorg
28
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
TMMireg Foundation Contacts
29
sect Full information on the work of the Foundation can be found onwwwtmmifoundationorg
sect If interested in becoming a member of the Foundation (currently freeregistration) please email your details to infotmmifoundationorg
sect If your organisation is interested in supporting the work of the Foundationplease email sponsorshiptmmifoundationorg
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license
Copyright copy 2010 Experimentus LtdCopyright copy 2010 Experimentus Ltd
TMMireg Foundation Contacts
29
sect Full information on the work of the Foundation can be found onwwwtmmifoundationorg
sect If interested in becoming a member of the Foundation (currently freeregistration) please email your details to infotmmifoundationorg
sect If your organisation is interested in supporting the work of the Foundationplease email sponsorshiptmmifoundationorg
This document is created with an UNREGISTERED copy of PDF Vista To remove this mark buy a license