Improving the ROI of Software Quality

Assurance Activities: An Empirical Study

PRESENTED BY KHUSH BAKHAT

Return On Investment

Investment source

Investor benefit

IEEE’s Definition of Software Quality Assurance

“A planned and systematic pattern of all actions necessary

to provide adequate confidence that an item or product

conforms to established technical requirements and a set of

activities designed to evaluate the process by which the

products are developed or manufactured ”

Quality Assurance Activities

Context

Series Of Incremental Projects In

China

3 Types Of QA

Activities

Empirical Findings Of

Effort Distribution

Pattern

Results

The result of the study gives implications on:

How to identify which type of QA activity is insufficient

How to balance the effort allocation and planning for future projects

How to improve the weak part of each QA activity and finally improve

the return on investment (ROI) of QA activities and the whole process

effectiveness under the specific organization context

Related Work

Related Work

Background and Data Source of the Study

The empirical study is based on the process

data from developing a series of incremental

R&D software projects within a medium-

sized software organization in China

Project Quality Goal

Every incremental version of this project shares the same quality goal before releasing:

Rate of requirements coverage by test cases is 100%

All planned test cases executed and passed

No non-trivial defects are detected during at least one day of continuous testing

Satisfy the quality goal of 0.2defects/KLOC when it is released

Objective

Empirical assessment

• Effort distribution patterns of QA activities

Develop understanding

• QA activities during software life cycle

Provide guidelines and process implication

• Planning and balancing the effort distribution

• Maximize the ROI of QA activities

Defects of specification, missing

portions, developer blind spots are

easier to be found by review

Defects of numerical approximations

and program dynamics are harder to

be found by review

Reviewers can’t tell how fast and

reliable the system will be

how user-friendly the system is by

just reviewing

Testing serves as a hands-on

experience of the actual and

operational system that review can’t

achieve by imaging

It is good at finding numerical

approximations and program

dynamics defects but harder to find

defects of specification, missing

portions, and developer blind spots

RE

VIE

WT

ES

TIN

G

Process Audit

Process audit has an indirect impact on the software

product

Adopt and tailor the most effective method that comes

from the best industrial practices to the specific context of

the project

Return on Investment

Investments to the software development life cycle is an important and

challenging issue for project planning

Balance the effort allocation among QA Activities to improve resource

allocation and maximize the return on investment (ROI) of the entire quality

assurance

Methodology

This empirical study follows the Goal/Question/Metric

(GQM) methodology

GQM Model

The GQM model used in this study is shown as follows:

• Goal: Improve the ROI of QA activities

Question 1(Q1): What is the baseline of QA activities effort

allocation under the specific organization context?

Metric1(M1):Statistic description: Mean, Median, Standard

Deviation of Effort Distribution Percentage

Question 1 and Metric

Goal/Question/Metric (GQM)

Question 2(Q2): How to identify which QA activities might be insufficient

while others might be overdone?

Metric2(M2):Regression Diagnostic Model

Metric3(M3):Cost of Finding and Removing Defects in Each Phase

Metric4(M4):Distribution of Types of Defects Found by Testing

Question 2 and Metric 2, 3, 4 Metric 2: Regression Diagnostic Model

Testing Is Inversely Proportional To Review & Audit

Multivariable Linear Regression Analysis

Process Implication

For this case study, testing is the most labor-intensive activities of

the QA activities

For this software organization, improving the process audit activity

might be more cost-effective

Metric 3

Compare the requirement, design and testing phase’s cost of finding and

removing defects

Process Implication

Findings from this analysis support that process audit and

reviewing are more effective than testing

Metric 4: Distribution of Types of Defects Found by Testing

Goal/Question/Metric (GQM)

Question 3(Q3): How to improve reviewing if it is

insufficient?

Metric3(M5): Filter Effectiveness

Question 3 and Metric 5

If put the extra effort on the weakest part would be more cost-

effective

The definition the Filter Effectiveness (FE) of each phase:

Process Implication

GQM

Question 4(Q4): How to improve process audit if it is insufficient?

Metric4(M6): NCs (Not Compatibles) Distribution

Question 5(Q5): How to improve testing if it is insufficient?

Metric5(M7):Value-based Software Testing: Feature Prioritization

Periodical Reports

Main items of periodical reports include task completion ratio,

planned and actual effort, problems or risks they encounter.

Periodical reports are usually used by project manager to monitor

developers’ task progress, identify risks and issues.

Task Estimation

Overestimating the task effort might lead to project delay, shorten

the time for testing and lead to low quality product,

Underestimates might lead to resource waste

For estimation its better to adopt systematic and professional effort

estimation methods or tools, such as COCOMO tool

Process Implication

Analysis help organization to maintain their process maturity on a higher level

Continuous process improvement by putting more effort on process audit and

improving the process of Project Planning and Project Monitoring and Control

In future it would benefit for process improvement and achieve a higher ROI.

Threats to Validity

• Oversimplified assumptions of Regression Model

The assumption of regression equation is that the utility function of each QA

activity is linear

The utility function of QA activities is not simply linear in real life, usually it

comes out as an “S” shape production function

Data Acquisition for other organizations

In this case study, analysis based on a series of maintenance and evolution

projects with the same process maturity level and quality goals.

This assumption limits its adoption by only those organizations that collect and

have the required process data

Data Quantity and Soundness of Statistical Analysis

In this case study, the data is very limited with missing which is a threat to

analysis’s validity

Study assume that all releases are the same

In reality, different releases may involve work of different nature, requiring

different types of QA activities, and different effort distribution patterns

Data Variance Analysis

In this case study, the author’s deal with averages without further

analysis for its variance, e.g.:

Baseline of required QA effort percentage in table 1

Cost of finding and removing defects in table 4

Filter effectiveness of each phase by reviewing in table 5

Conclusion

The results from this study support that early review is more effective than late

testing

This study will effective to prevent the defect introduction; these three types of

QA activities complement each other to achieve the required quality goal

Future Work

Future work would include adopting value-based principles to each

QA activity to improve its effectiveness

Expand the basis of this approach to find a way to balance the effort

distribution of QA activities to maximize its ROI.