Upload
stag-software-pvt-ltd
View
2.472
Download
0
Embed Size (px)
DESCRIPTION
Test process from different perspective - Test Maturity Model (TMMSM)TMM goals and symptomsMetrics system architectureGoal focused measurements
Citation preview
TEST PROCESS MATURITY AND RELATED MEASUREMENTS
M C Nagaraj
in.linkedin.com/in/mcnagaraj
Copyright 2011. STAG Software Pvt Ltd. All rights reserved.
Contents
Test process from different perspective
Test Maturity Model (TMMSM )
TMM goals and symptoms
Metrics system architecture
Goal focused measurements
TMM SM is a registered service mark of Illinois Institute of Technology, Chicago
2
Copyright 2011. STAG Software Pvt Ltd. All rights reserved.
Test Process looked from different perspective
Methodology
Technology and tools
People
3
Copyright 2011. STAG Software Pvt Ltd. All rights reserved.
Test Process looked from different perspective
Methodology What are the key testing methods?
When do we use them?
How do we implement them?
How do we prevent defect migration and improve defect-finding capabilities at all levels?
How can we adapt standard terminologies and test standards?
How do we create meaningful test plans and other critical test deliverables?
How do we identify and prioritize potential software test process improvements?
4
Copyright 2011. STAG Software Pvt Ltd. All rights reserved.
Test Process looked from different perspective
Technology and tools• How can we use risk analysis techniques to decide on goals for testing?
• How can we design adequate test cases?
• How can we decide on what types of test required to find what types of bugs?
• What measurements to collect and analyze to control test process?
• How can we select test tools and vendors?
• How to implement an effective tools program?
• What are the most significant opportunities for automating test process?
• How to get better ROI for investment made on tools?
5
Copyright 2011. STAG Software Pvt Ltd. All rights reserved.
Test Process looked from different perspective
People• Do we understand the attitude required in people involved in testing?
• Do we know what training required for people involved in testing?
• Do people in the organization believe that the quality of test process determines the success of overall test process?
• Do we understand the development team also has role in testing?
• Do people in the organization feel that test engineers pursue faults not the people?
• Do we understand the business value of effective testing?
• Does management team support continuous test process improvement?
6
• Is there a way through which we can prove test assets created are indeed complete and effective?
• Is there a defect detection technology one can use at each stage of SDLC to uncover defects effectively?
• How can we assess accurately where we stand with respect to quality goals?
• Can we guarantee clean software to our customers?
• Is it possible to assess how much to retest when code change?
• Which, where, when and how much to automate in test engineering activities?
Have a look at HBT (Hypothesis Based Testing) at www.stagsoftware.com
Effective test process!
Copyright 2011. STAG Software Pvt Ltd. All rights reserved. 7
Copyright 2011. STAG Software Pvt Ltd. All rights reserved. 8
Internal structure of TMM
Testing capability
Levels
Maturity Goals
Maturity sub-goals
Activities/tasks/resp.
Critical views
Manager Dev/tester User/client
Implementation
and organization
adaptation
Indicate Contain
Supported by
Achieved by
Organized by
Address
TMM levels
• Test process optimization
• Quality control
• Application of process data for defect prevention
Level 5: Optimization/Defect Prevention and Quality
Control
• Software quality evaluation
• Establish a test measurement program
• Establish an organization wide review program
Level 4:
Management and Measurement
• Control and monitor the testing process
• Integrate testing into the software life cycle
• Establish a technical training program
• Establish a software test organization
Level 3:
Integration
• Institutionalize basic testing techniques & methods
• Initiate a test planning process
• Develop testing and debugging goals
Level 2:
Phase Definition
Level 1:
Initial
Copyright 2011. STAG Software Pvt Ltd. All rights reserved. #
Copyright 2011. STAG Software Pvt Ltd. All rights reserved. 10
Metrics system architecture
Process-1 Process-2 Process-n Customer
Measurements
Questions
GOALS
Metrics
Basili’s GQM Paradigm is goal centered framework
Copyright 2011. STAG Software Pvt Ltd. All rights reserved.
TMM Level 1
• Testing is a chaotic process
• Not distinguished from debugging
• Documented set of specification not available
• Tests are developed in ad-hoc way after coding is completed
• The objective of testing is to show that software works
11
Copyright 2011. STAG Software Pvt Ltd. All rights reserved.
Measurements at L1
As there is no specific goals at this maturity level we can see whether we have answers for these questions
1. How do you know the size of software tested?
2. How do you categorize projects as big, medium and small projects?
3. Do you know how many defects were found in the project?
4. How much effort spent for activity coming under heading “testing”?
Size:Size of code in KLOCNumber of requirements or featuresNumber of test cases developedNumber of test cases executed
Defects:High, medium, low severity defects countDefects/KLOC
Cost:Total effort spent for testing
12
Copyright 2011. STAG Software Pvt Ltd. All rights reserved.
TMM Level 2
TMM Level 2 goals Characteristics
2.1 Develop Testing and
Debugging Goals and Policies
2.2 Initiate a Test Planning
Process
2.3 Institutionalize basic
testing techniques and
methods
There is a clear separation
between debugging and testing
phase
It is a planned activity in project
plan
Plan starts after coding is
complete
Basic testing techniques in place
Testing is multi-leveled
13
• How much time planned for all levels of testing?
• How much time spent in testing activities?
• How much time spent in debugging and fixing different types of bugs?
• What percentage of code coverage targeted and achieved?
• Did we uncover different types of defects at right stage in the SDLC?
Possible questions to check on goals status
Copyright 2011. STAG Software Pvt Ltd. All rights reserved. 14
Copyright 2011. STAG Software Pvt Ltd. All rights reserved.
Measurements at L2
Time/effort related measurement:
Time/effort spent is test planning
Time/effort spent in unit, integration, system, regression testing
Total time/effort spent in testing activities
(Granularity in above measurements also possible like time/effort spent in test design for unit / integration/ system tests)
Number of planned test cases, unplanned test cases
Planned/actual degree of statement coverage
Defects:
Number of defects in each phase (SDLC Phases)
Number of defects found in each level of testing (UT, IT, ST, UAT)
Number of each type of defects found
Time taken to fix and re-test each defects type
15
Copyright 2011. STAG Software Pvt Ltd. All rights reserved.
TMM Level 3
TMM Level 3 goals Characteristics
3.1 Establish a Test
Organization
3.2 Establish a technical
training program
3.3 Integrate testing into the
software life cycle
3.4 Control and monitor the
testing process
There is an established test
organization
Testing is integrated to SDLC
Test plan is developed, tracked
and controlled (Integrated with
project plan)
Test engineers drive test process
improvement
Users/clients attend milestone
meeting
User/clients support in
developing usability test plans
16
• How many people are in testing team at different levels?
• How many hrs of training the team go through for competency development for different levels?
• How much time spent in all levels of testing against plan?
• How many projects using tools invested effectively?
• Do you know what percentage of defects escaping UT/IT (internal) and ST (External)?
• Are we under control for planned budgets for testing in all projects?
• Is team productivity assumed matching with actual observed?
Possible questions to check on goals status
Copyright 2011. STAG Software Pvt Ltd. All rights reserved. 17
Copyright 2011. STAG Software Pvt Ltd. All rights reserved.
Measurements at L3
• Coverage related measurementRequirement, statement, branch coverage
Types of test planned and completed
• Productivity related measurementNumber of Test cases written / Per unit time
Test cases executed /Per unit time
• Training related measurementNumber of training hrs attended / year (for all test professionals)
• ROI on tools initiativesTotal cost saved by automation/ Total tool program cost
• Defect escapes at different levels of testingUnit Test escape = (Total UT bugs found in ST/ Total ST bugs)
External Test escape = (Total bugs found in UAT / Total bugs found in all levels
• Number of Users/Clients formal interactions
18
Copyright 2011. STAG Software Pvt Ltd. All rights reserved.
TMM Level 4
TMM Level 4 goals Characteristics
4.1 Establish an organization
wide review program
4.2 Establish a test
measurement program
4.3 Software quality
evaluation
Review program is effective
Reviews are planned activity in
project plan
Measurements collection and
analysis process effective
Quality attributes of a product
are well-defined and measured
19
• Is review activities conducted formally?
• How many are trained on inspection process?
• Is formal inspection effective?
• Hope measurements to be collected in all projects are agreed upfront and collected?
• Are measurements collected are analyzed?
• Is there an organization critical metrics defined with baseline value?
• Do we get clear visibility on quality at each review?
Possible questions to check on goals status
Copyright 2011. STAG Software Pvt Ltd. All rights reserved. 20
Copyright 2011. STAG Software Pvt Ltd. All rights reserved.
Measurements at L4
• Number of inspection leaders available
• Number of people trained on inspection
• Size of the item inspected
• Time spent on inspection activities
• Number of defects found per hour of review time
• Effort spent on different types of test
• Quality levels/quality index reached at each milestone review against plan
• Measurement collections against plan
• # of corrective action if required based on analysis of measurements collected
21
Copyright 2011. STAG Software Pvt Ltd. All rights reserved.
TMM Level 5
TMM Level 5 goals Characteristics
5.1 Defect prevention
5.2 Quality control
5.3 Test process optimization
Data from all projects are
collectively analyzed
Critical defects types are
analyzed thoroughly
Quality control concepts are
adopted
Right tools are inserted
progressively in the organization
22
• Is organization critical measurements are coming from all projects?
• Do we understand various types of defects we uncover and where we need to act for prevention?
• Do we have clear control limits defined for critical metrics in the organization for key processes?
• Are you happy with ROI on tools invested?
• Is senior management support for spending effort process improvements evident?
Possible questions to check on goals status
Copyright 2011. STAG Software Pvt Ltd. All rights reserved. 23
Copyright 2011. STAG Software Pvt Ltd. All rights reserved.
Measurements at L5
• Time/Effort spent in defect causal analysis
• Number of actions suggested
• Effort/cost for implementing action plans
• Costs of statistical testing
• Effort/costs of training SEPG team in process control
• Effort/costs on quantitative process analysis
• Number of process changes
• Number of new tools introduced in the organization
24
Copyright 2011. STAG Software Pvt Ltd. All rights reserved.
References
1. Practical Software Testing by Ilene Burnstein
2. Software Metrics by C. R. Pandian
25
Copyright 2011. STAG Software Pvt Ltd. All rights reserved. 26
Thank you!
Web: www.stagsoftware.com
Blog: www.stagsoftware.com/blog
Twitter: @stagsoft