Making Numbers Count: Metrics That Matter

Making Numbers Count: Metrics That Matter

©2015 Professional Quality Assurance Ltd. (PQA) --http://www.pqatesting.com 1

6/11/2015

Better Software

Conference West

June 11, 2015

Las Vegas, Nevada

Mike Trites

Making Numbers Count

Metrics that Matter

About Me

Mike Trites

‣ Senior Test Consultant

‣ Over 10 years of experience in software testing, working

with several clients in multiple industry verticals

Certifications

‣ Black Box Software Testing Foundations – AST

‣ Certified Tester, Foundation Level – ISTQB

‣ Certified Tester, Advanced Level Test Manager – ISTQB

2 ©2015 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com



6/11/2015

Introduction


How is testing going?

“We’re on-track.” “95% of test cases so far

have passed.”

“We found 15 new defects yesterday.”

What I’m here for…

Illustrate the important role that metrics play in the software

testing process.

Explain how to give context when communicating metrics.

Describe how to use metrics to improve the testing process

for future projects.

Identify some common pitfalls and how to avoid them.

Identify a selection of metrics to:

‣ Measure progress during a test cycle.

‣ Determine when to stop testing.

‣ Measure the effectiveness of testing.




6/11/2015

What I’m NOT here for…

Provide a “one size fits all” solution for you or your

organization.

Why not?

‣ Metrics are situation / context specific.

‣ There is no silver bullet.

‣ There is no “right” answer.

‣ They will not solve all your problems.


Agenda

Why Metrics are Important

Defining and Capturing Metrics

Reporting on Metrics

Identifying Trends

Misuse of Metrics




6/11/2015





6/11/2015



Metrics play an important role in:

• Estimation

• Evaluating suspension criteria

• Evaluating exit criteria

• Status reporting

• Process improvement

Process Improvement

"Those who cannot remember the past are condemned to

repeat it.”

- George Santayana

What is “better”? How do we know if we’re better now than

we were?

Estimates:

‣ Comparing planned effort and actual effort

‣ Estimate variance and what factors affected it




6/11/2015

Metrics for Process Improvement


Closure Period / Turn-around Time

• Measures how long it takes for a reported defect to be rectified

Defect Ratio

• Useful for assessing the effectiveness of regression or automation suites

Metrics for Process Improvement


Defects by Root Cause

• Allows you to proactively avoid anticipated defects in the future

Closed Defects by Resolution

• Identifies instances of duplicate or invalid defects

Defect Removal Efficiency

• Identifies defects that are uncovered after the application is released



6/11/2015

Defining and Capturing Metrics

Defining Metrics

Every phase of the testing process can be measured.

Schedule / Budget

Test Coverage:

‣ Requirements

‣ Configurations

‣ Code

‣ Automation

Test Design

Test Execution

Defects




6/11/2015

Capturing Metrics


Expensive commercial tools can be

useful, but are not always necessary.

• Examples:

‣ Test Case Management Tool

‣ Defect Management Tool

‣ Time Tracking System

There are less expensive tools that can

be used to achieve the same results:

• Examples:

‣ Bugzilla

‣ JIRA

‣ Excel

Reporting on Metrics



6/11/2015

Metrics Need Context

When metrics are used in isolation, their significance is left

open to interpretation.

Stakeholders can draw the wrong conclusions if no context

is provided and they are instead left to interpret the data on

their own.

Textual summary can provide context, but so can a good

chart.


What makes a good chart?

Accurate – Data is not purposely skewed or omitted.

Clear – Information is presented in a way that is

unambiguous and easy to interpret at a glance.

Concise – Chart is not cluttered with unneeded or

distracting data.

Includes any necessary:

‣ Titles

‣ Labels

‣ Data values

‣ Trend lines

‣ Annotations




6/11/2015


Medium 68%

Low 18%

High 14%

Medium 68%

Low 18%

High 14%

Colour Conventions

Defects by Severity

More meaningful with

appropriate colours

0%10%20%30%40%50%60%70%80%90%

100%

Example: Test Progress


0%10%20%30%40%50%60%70%80%90%

100%



6/11/2015

Example: Defects by Release


= 63 (49%)

= 56 (44%)

= 9 (7%)

Fixed / Planned

v0.1 v0.2 v0.3 v0.4 v0.5 v0.6 v0.7

Dis

co

ve

red

v0.1 1 9 2

v0.2 7 10 15 1 1 3

v0.3 11 17 1 3

v0.4 1 31 10 3

v0.5 1 1

Example: Scorecard


Execution

Progress

Pass

Rate

Blocked

TestsMust-Fix Other

Component 1 100 % 100 % 0 0 0

Component 2 100 % 78 % 0 0 4

Component 3 87 % 80 % 0 1 1

Component 4 95 % 100 % 0 0 4

Component 5 75 % 92 % 1 1 0

Component 6 98 % 79 % 2 1 4

Overall 93 % 88 % 3 3 13

Defects Test Execution



6/11/2015

http://dilbert.com/strips/comic/2007-05-16/

Dashboards


Identifying Trends



6/11/2015

Trends

One measurement captures status at a point in time.

When a series of data points are examined, trends in that

data start to emerge.

Trends allow us to:

‣ Differentiate between systemic behaviour and temporary

anomalies.

‣ Make predictions about the future.

Accuracy increases with more historical data.


0

5

10

15

20

25

30

35

Defects Detected

Trends


Day # of

Defects

1

2

3

4

5 15

6

7

8

9

10

?

?

?



6/11/2015

0

5

10

15

20

25

30

35

Defects Detected

0

5

10

15

20

25

30

35

Defects Detected

Trends


Day # of

Defects

1

2 4

3

4

5 15

6

7

8

9

10

0

5

10

15

20

25

30

35

Defects Detected

0

5

10

15

20

25

30

35

Defects Detected

Trends


Day # of

Defects

1

2 4

3

4

5 15

6

7

8

9

10 1



6/11/2015

0

5

10

15

20

25

30

35

Defects Detected

Trends


Day # of

Defects

1

2 4

3

4

5 15

6 2

7

8 8

9

10 1

0

5

10

15

20

25

30

35

Defects Detected

Trends


Day # of

Defects

1 3

2 4

3 10

4 7

5 15

6 2

7 3

8 8

9 2

10 1



6/11/2015

0

5

10

15

20

25

30

Example: Open vs. Closed Defects


0

5

10

15

20

25

30

0

2

4

6

8

10

12

14

16

18

0

2

4

6

8

10

12

14

16

18

Example: Open Defects




6/11/2015

Misuse of Metrics

Intentional Manipulation

“There are three kinds of lies: lies, damned lies, and

statistics.”

- Benjamin Disraeli

Metrics can be manipulated to make

any situation look good or bad.




6/11/2015

Misleading Graphs

An example of a good graph:


0

100

200

300

400

500

A B C D

430 455 448 465

Misleading Graphs

Inappropriate scale:


0

200

400

600

800

1000

A B C D



6/11/2015

Misleading Graphs

Scale does not start at 0:


420

425

430

435

440

445

450

455

460

465

A B C D

Tester 1 Tester 2

Tester 3

Tester 4


Misleading Graphs

Defects by Component



6/11/2015

11% 42%

5% 42%

Tester 1

Tester 2

Tester 3

Tester 4



Misleading Graphs

Poor Choice of Format


12%

15%

14%

8% 11%

17%

13%

10% Tester 1

Tester 2

Tester 3

Tester 4

Tester 5

Tester 6

Tester 7

Tester 8




6/11/2015

Better Choice of Format


0%2%4%6%8%

10%12%14%16%18%

12%

15% 14%

8%

11%

16%

13%

10%


Overloading

Often, managers are most

interested in defect statistics.

Does defect count measure:

‣ Tester effectiveness?

‣ Developer skill?

‣ Product quality?

‣ Effort remaining?

One metric does not tell the entire story.




6/11/2015

Performance Assessment

People optimize what we measure them against, at the

expense of what we don’t.

For example, if we assess testers based on their defect

counts, how much time will testers spend:

‣ Documenting their tests?

‣ Coaching other testers?

‣ Researching and polishing defect reports to make their

defect easier to understand, assess and replicate?

‣ Logging variants of the same failure to increase defect

count?


Performance Assessment


0

100

200

300

400

500

Tester 1 Tester 2 Tester 3

Test Cases Executed



6/11/2015

Key Points to Remember


Metrics can be captured for just about any testing activity.

Metrics play an important role in:

‣ Estimation

‣ Status reporting

‣ Evaluating suspension and exit criteria

‣ Process improvement

Metrics, like anything, should be planned in advance.

Reporting needs should be identified before setting up

tools.

To avoid misinterpretation, metrics need context.




6/11/2015


Even when properly presented, too much information can

be overwhelming.

It is best to limit the number of metrics and reports that are

presented to stakeholders to only what is essential.

Capturing, analysing and reporting on metrics can be a

drain or waste of time if those metrics are not being used to

provide some value in return.

Metrics influence tester performance, but not always in the

way you might think.

Metrics should be used as a starting point for discussion.

One metric does not tell the entire story.


Questions or Comments?




6/11/2015

Thanks for coming!

Contact Information:

©2015 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com

[email protected]

ca.linkedin.com/in/miketrites

@mike_trites

www.pqatesting.com

49