A practical Test Strategy A practical Test Strategy
Jurian van de LaarJurian van de [email protected]@improveqs.nl
EuroSTAR Conference 2009EuroSTAR Conference 2009StockholmStockholm
Accredited provider ISTQBFoundation & all Advanced modules
Training provider IREBRequirements Engineering
Accredited Lead Assessorsformal TMMi assessments
Improve Quality ServicesImprove Quality Services
Leading in Testing & Quality ManagementLeading in Testing & Quality Management Consultancy, Subcontracting and TrainingConsultancy, Subcontracting and Training Founded in 1998 by Erik van VeenendaalFounded in 1998 by Erik van Veenendaal
Eindhoven
Improve Quality ServicesWaalre (near Eindhoven)
The Netherlands
www.improveqs.nl
Policy? Strategy?Policy? Strategy?
What does it mean ??What does it mean ??
Do you need this ??Do you need this ??
Case study Case study Critical success factorsCritical success factors
Sounds familiar ?Sounds familiar ?
Definitions Test Policy & StrategyDefinitions Test Policy & Strategy
Test Policy Test Policy – High level description of principles, – High level description of principles, approach and major objectives of the organization approach and major objectives of the organization regarding testing.regarding testing.
Test StrategyTest Strategy – High level description of test levels for – High level description of test levels for an organization or programme, and the testing to be an organization or programme, and the testing to be performed within each level.performed within each level.
Test Policy
Test Strategy
Master Test Plan
Generic risksGeneric risksGeneric risksGeneric risks
Business goalsBusiness goalsBusiness goalsBusiness goals Test goalsTest goalsTest goalsTest goals
BlueprintBlueprintBlueprintBlueprint
ProjectsProjectsProjectsProjects
Organization
Test Policy 20XX
Definition of testing
Mission / Ambition based onbusiness goals
based onbusiness goals
EmployeesTest processCustomers
Test Process Improvement- objectives, performance indicators, model
Test targets
Test StrategyTest Strategy
Based on the test policyBased on the test policy
Addresses the generic product risksAddresses the generic product risks
Contains description of test levels:Contains description of test levels:
� Purpose, responsibility� Level of independence (organizational)� Entry and exit criteria� Test process and deliverables � Approach (top-down, bottom-up)
� Test design techniques� Tools to be used� Standards to be used� Test environment� Typical non-functionals
Test policy & strategy are important … Test policy & strategy are important … Provide common basis for projectsProvide common basis for projects
What is What is reallyreally important for business / important for business / managersmanagers
Leading in models like Leading in models like TMMi, TPI, and TMap NextTMMi, TPI, and TMap Next
… … but also difficult to realize !but also difficult to realize ! ‘‘We have no business goals or policy’We have no business goals or policy’
‘‘We don’t have time / interest / priority’We don’t have time / interest / priority’
The challengeThe challenge
Practical case studyPractical case study
Principal: IT-Solutions Mgr. Solutions ArchitecturePrincipal: IT-Solutions Mgr. Solutions Architecture
Draft Test Strategy guidelines as starting pointDraft Test Strategy guidelines as starting point
My assignment: coaching creation of Test StrategyMy assignment: coaching creation of Test Strategy
Themes: strategy, terminology, organizationThemes: strategy, terminology, organization
Scope: IT Express BeNeLuxScope: IT Express BeNeLux Business, Business-IT and IT-servicesBusiness, Business-IT and IT-services
Roadmap and planningRoadmap and planning
Organize meeting structureOrganize meeting structure
Kick-off with management groupKick-off with management group
Training day for work groupTraining day for work group
Strategy workshops Strategy workshops (4 sessions)(4 sessions)
Elaboration Test Policy & StrategyElaboration Test Policy & Strategy
Reporting to management groupReporting to management group
ApproachApproach
33
55
66
77
11
22
44
Kick-off with managementKick-off with management
Purpose: get feedback and constraintsPurpose: get feedback and constraints
Presentation using provocative argumentsPresentation using provocative arguments
Results:Results: Feedback:Feedback:
How ‘testing’ is perceived in the organizationHow ‘testing’ is perceived in the organization
Effects of insufficient testing (= business case for improvement)Effects of insufficient testing (= business case for improvement)
Important quality characteristicsImportant quality characteristics
ConstraintsConstraints Realistic (acceptable) costs and organization structureRealistic (acceptable) costs and organization structure
33
Strategy workshopsStrategy workshops
1.1. Agree on purpose and scopeAgree on purpose and scope using management feedback and constraints as inputusing management feedback and constraints as input
1.1. Determine policy themesDetermine policy themes reliability, customer facing, usability, reliability, customer facing, usability,
performance, efficient testing, time to marketperformance, efficient testing, time to market
1.1. Definition of testingDefinition of testing
2.2. Business goalsBusiness goals not our responsibility, proactive thinking is (rationale)not our responsibility, proactive thinking is (rationale) 7 values DHL taken as starting point7 values DHL taken as starting point
1.1. Test strategyTest strategy V-model, determine test levelsV-model, determine test levels RACI*, responsibilities, who is ‘main contractor’RACI*, responsibilities, who is ‘main contractor’
55
* = RACI: Responsible, Accountable, Consulted, Informed
Elaboration: resultsElaboration: results
Test Policy documentTest Policy document Quality goals + rationale (2 A4)Quality goals + rationale (2 A4) Test policy (2 A4)Test policy (2 A4)
Test Strategy documentTest Strategy document PrinciplesPrinciples V-modelV-model Characteristics per levelCharacteristics per level Risk based testingRisk based testing
Management presentationManagement presentation Results and recommendationsResults and recommendations
66
Elaboration: Test PolicyElaboration: Test Policy66
Elaboration: V-modelElaboration: V-model66
Full Roll-OUT
Management presentationManagement presentation
RecommendationsRecommendations Agree on common test strategyAgree on common test strategy Apply risk based testingApply risk based testing Install a test competence cellInstall a test competence cell Start a test process improvement projectStart a test process improvement project Agree on common test terminology (e.g. ISTQB)Agree on common test terminology (e.g. ISTQB)
Recommendations have been implementedRecommendations have been implemented Test competence cell: competence manager installedTest competence cell: competence manager installed Interviews / meetings with stakeholdersInterviews / meetings with stakeholders Plan for deployment writtenPlan for deployment written
77
Lessons and success factorsLessons and success factors
It requires time to determine a good test strategy!It requires time to determine a good test strategy!
If there is no policy available, then propose one.If there is no policy available, then propose one.
Involve management (feedback, constraints).Involve management (feedback, constraints).
Broad representation of organization as workgroup.Broad representation of organization as workgroup.
Foundation test knowledge in workgroup (training).Foundation test knowledge in workgroup (training).
Bring sufficient flexibility into your V-model.Bring sufficient flexibility into your V-model.
Who is responsible for what (RACI) ?Who is responsible for what (RACI) ?
Pioneer for implementation from own organization.Pioneer for implementation from own organization.
Epilogue (1)
Follow-up by test competence manager: Further elaboration and deployment of test
strategy (templates and tools)
Organized sessions/training on test awareness
Assistance in projects to apply test strategy
Has set up method to manage test environment
The assignment ended in 2008, what happened afterwards…The assignment ended in 2008, what happened afterwards…
Epilogue (2)
Current situation (mid 2009): Competence cell (0.5 FTE*): central organization
of testing expertise.
Close cooperation helps to get clear requirements
Clearly defined test levels are helpful
Test strategy currently applied in a project for European standardisation of one of DHL’s most important product lines
The assignment ended in 2008, what happened afterwards…The assignment ended in 2008, what happened afterwards…
* = FTE: Full-time equivalent
ConclusionsConclusions
A sound Test Policy and Test Strategy …A sound Test Policy and Test Strategy … are the important starting points for projectsare the important starting points for projects
help to have good discussions …and decisionshelp to have good discussions …and decisions
can be very clear and specificcan be very clear and specific
give answers to relative questionsgive answers to relative questions
are the first step in test process improvementare the first step in test process improvement
… … are worthwhile to give a try in your organization ?!?!are worthwhile to give a try in your organization ?!?!