28
W18 Special Topics 5/7/2014 3:00:00 PM Making Numbers Count: Metrics That Matter Presented by: Mike Trites Professional Quality Assurance, Ltd. Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073 888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com

Making Numbers Count: Metrics That Matter

Embed Size (px)

DESCRIPTION

As testers and test managers, we are frequently asked to report on the progress and results of our testing. The question “How is testing going?” may seem simple enough, but our answer is ultimately based on our ability to extract useful metrics from our work and present them in a meaningful way. This is particularly important in agile environments, where clear, concise, and up-to-date metrics are potentially needed multiple times per day. Mike Trites identifies a number of ways metrics can be used to measure progress during a test cycle and, ultimately, to determine when to consider testing complete. Learn the common pitfalls that metrics misuse can lead to and how you can avoid them by giving proper context when communicating metrics to your stakeholders. Take back key metrics for measuring the effectiveness of your testing and discover how to use what is learned on one project to improve your testing process on future projects.

Citation preview

Page 1: Making Numbers Count: Metrics That Matter

W18

Special Topics

5/7/2014 3:00:00 PM

Making Numbers Count: Metrics

That Matter

Presented by:

Mike Trites

Professional Quality Assurance, Ltd.

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com

Page 2: Making Numbers Count: Metrics That Matter

Mike Trites

Professional Quality Assurance, Ltd.

Senior test consultant with Professional Quality Assurance, Ltd. Mike Trites has nearly a decade of experience in the software testing industry and holds multiple certifications, including an Advanced-Level Test Manager Certification from the ISTQB. As a consultant, Mike has worked with clients, both small and large, in a wide range of environments. He has helped deliver projects from web applications and database management software to financial software and video lottery systems. As a test manager, Mike has managed all aspects of the testing lifecycle and works with clients to help define, implement, and improve their internal testing processes. Find Mike on LinkedIn at ca.linkedin.com/in/miketrites.

Page 3: Making Numbers Count: Metrics That Matter

PQA Company Overview 4/26/2014

©2012 Professional Quality Assurance Ltd. (PQA) --

http://www.pqatesting.com 1

STAREAST 2014

May 4 – 9, 2014

Orlando, Florida

Mike Trites

Making Numbers Count

Metrics that Matter

About Me

� Mike Trites

‣ Senior Test Consultant

‣ Nearly 10 years of experience in software testing, working

with several clients in multiple industry verticals

� Certifications

‣ Black Box Software Testing Foundations – AST

‣ Certified Tester, Foundation Level – ISTQB

‣ Certified Tester, Advanced Level Test Manager – ISTQB

2©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com

Page 4: Making Numbers Count: Metrics That Matter

PQA Company Overview 4/26/2014

©2012 Professional Quality Assurance Ltd. (PQA) --

http://www.pqatesting.com 2

Introduction

3©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com

How is testing going?

“95% of test cases so far

have passed.”

“We found 15 new defects yesterday.”

“We’re on-track.”

What I’m here forA

� Illustrate the important role that metrics play in the software

testing process.

� Explain how to give context when communicating metrics.

� Describe how to use metrics to improve the testing process

for future projects.

� Identify some common pitfalls and how to avoid them.

� Identify a selection of metrics to:

‣ Measure progress during a test cycle.

‣ Determine when to stop testing.

‣ Measure the effectiveness of testing.

4©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com

Page 5: Making Numbers Count: Metrics That Matter

PQA Company Overview 4/26/2014

©2012 Professional Quality Assurance Ltd. (PQA) --

http://www.pqatesting.com 3

What I’m NOT here forA

� Provide a “one size fits all” solution for you or your

organization.

� Why not?

‣ Metrics are situation / context specific.

‣ There is no silver bullet.

‣ There is no “right” answer.

‣ They will not solve all your problems.

5©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com

Agenda

� Why Metrics are Important

� Defining and Capturing Metrics

� Reporting on Metrics

� Using Metrics for Process

Improvement

� Misuse of Metrics

6©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com

Page 6: Making Numbers Count: Metrics That Matter

PQA Company Overview 4/26/2014

©2012 Professional Quality Assurance Ltd. (PQA) --

http://www.pqatesting.com 4

Why Metrics are Important

8©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com

Page 7: Making Numbers Count: Metrics That Matter

PQA Company Overview 4/26/2014

©2012 Professional Quality Assurance Ltd. (PQA) --

http://www.pqatesting.com 5

Why Metrics are Important

9©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com

Metrics play an important role in:

• Estimation

• Evaluating suspension criteria

• Evaluating exit criteria

• Status reporting

• Process improvement

Estimation

� Metrics captured on one project can help us estimate

various activities for future projects.

� With enough historical data, we can begin to estimate:

‣ Number of test cases

‣ Execution rate

‣ Number of defects

‣ Overall effort

10©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com

Page 8: Making Numbers Count: Metrics That Matter

PQA Company Overview 4/26/2014

©2012 Professional Quality Assurance Ltd. (PQA) --

http://www.pqatesting.com 6

Suspension Criteria

� Occasionally, it may be necessary to halt testing of a

particular version / build if the quality is found to be too low.

� How do we make this determination?

‣ Number of defects?

‣ Number of high-severity defects?

‣ Defect detection rate?

‣ Test failure rate?

11©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com

Exit Criteria

� How do we know when to stop testing?

‣ The allotted effort (schedule and / or budget) is expended.

‣ All planned tests have been executed.

‣ All planned tests pass.

‣ The defect detection rate drops below a certain threshold.

� There are pros and cons for each approach.

12©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com

Page 9: Making Numbers Count: Metrics That Matter

PQA Company Overview 4/26/2014

©2012 Professional Quality Assurance Ltd. (PQA) --

http://www.pqatesting.com 7

Exit Criteria

� The best approach is a combination that takes into account

multiple criteria, such as:

‣ Effort (schedule / budget)

‣ Defect detection rate

‣ Number of open defects

‣ Completion percentage

‣ Pass rate

13©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com

Status Reporting

� Simple and clear method for communicating the current

status of testing to stakeholders

� Allows stakeholders to:

‣ Draw conclusions

‣ Make decisions

� Detailed status reports provide a transparent view into the

testing effort and show progress toward defined exit criteria.

14©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com

Page 10: Making Numbers Count: Metrics That Matter

PQA Company Overview 4/26/2014

©2012 Professional Quality Assurance Ltd. (PQA) --

http://www.pqatesting.com 8

Defining and Capturing Metrics

Defining Metrics

Every phase of the testing process can be measured.

� Schedule / Budget

� Test Coverage:

‣ Requirements

‣ Configurations

‣ Code

‣ Automation

� Test Design

� Test Execution

� Defects

16©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com

Page 11: Making Numbers Count: Metrics That Matter

PQA Company Overview 4/26/2014

©2012 Professional Quality Assurance Ltd. (PQA) --

http://www.pqatesting.com 9

Capturing Metrics

17©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com

Expensive commercial tools can be

useful, but are not always necessary.

• Examples:

‣ Test Case Management Tool

‣ Defect Management Tool

‣ Time Tracking System

There are less expensive tools that can

be used to achieve the same results:

• Examples:

‣ Bugzilla

‣ JIRA

‣ Excel

Reporting on Metrics

Page 12: Making Numbers Count: Metrics That Matter

PQA Company Overview 4/26/2014

©2012 Professional Quality Assurance Ltd. (PQA) --

http://www.pqatesting.com 10

Metrics Need Context

� When metrics are used in isolation, their significance is left

open to interpretation.

� Stakeholders can draw the wrong conclusions if no context

is provided and they are instead left to interpret the data on

their own.

� Textual summary can provide context, but so can a good

report.

19©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com

What makes a good report?

� Accurate – Data is not purposely skewed or omitted.

� Clear – Information is presented in a way that is

unambiguous and easy to interpret at a glance.

� Concise – Report is not cluttered with unneeded or

distracting data.

� Includes any necessary:

‣ Titles

‣ Labels

‣ Data values

‣ Trend lines

‣ Annotations

20©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com

Page 13: Making Numbers Count: Metrics That Matter

PQA Company Overview 4/26/2014

©2012 Professional Quality Assurance Ltd. (PQA) --

http://www.pqatesting.com 11

21©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com

Medium68%

Low 18%

High 14%

Medium68%

Low 18%

High 14%

Colour Conventions

Defects by Severity

More meaningful with

appropriate colours

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Example: Test Progress

22©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Page 14: Making Numbers Count: Metrics That Matter

PQA Company Overview 4/26/2014

©2012 Professional Quality Assurance Ltd. (PQA) --

http://www.pqatesting.com 12

Example: Requirements Coverage

23©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com

Example: Configuration Coverage

24©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com

WindowsXP

WindowsVista

Windows7

Windows8

MacOSX

IE 7 ✓✓✓✓ ✓✓✓✓

IE 8 ✓✓✓✓ ✓✓✓✓

IE 9 ✓✓✓✓ ✓✓✓✓

IE 10 ✓✓✓✓

Chrome ✓✓✓✓ ✓✓✓✓ ✓✓✓✓ ✓✓✓✓ ✓✓✓✓

Firefox ✓✓✓✓ ✓✓✓✓ ✓✓✓✓ ✓✓✓✓ ✓✓✓✓

Page 15: Making Numbers Count: Metrics That Matter

PQA Company Overview 4/26/2014

©2012 Professional Quality Assurance Ltd. (PQA) --

http://www.pqatesting.com 13

Example: Automation Coverage

25©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com

Total Number

of Test Cases

Cycle 2

(40%)

Cycle 1

(15%)

0

5

10

15

20

25

30

Example: Open vs. Closed Defects

26©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com

0

5

10

15

20

25

30

Page 16: Making Numbers Count: Metrics That Matter

PQA Company Overview 4/26/2014

©2012 Professional Quality Assurance Ltd. (PQA) --

http://www.pqatesting.com 14

0

2

4

6

8

10

12

14

16

18

0

2

4

6

8

10

12

14

16

18

Example: Open Defects

27©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com

Example: Defects by Release

28©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com

= 63 (49%)

= 56 (44%)

= 9 (7%)

Fixed / Planned

v0.1 v0.2 v0.3 v0.4 v0.5 v0.6 v0.7

Discovered

v0.1 1 9 2

v0.2 7 10 15 1 1 3

v0.3 11 17 1 3

v0.4 1 31 10 3

v0.5 1 1

Page 17: Making Numbers Count: Metrics That Matter

PQA Company Overview 4/26/2014

©2012 Professional Quality Assurance Ltd. (PQA) --

http://www.pqatesting.com 15

Example: Scorecard

29©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com

Execution

Progress

Pass

Rate

Blocked

TestsMust-Fix Other

Component 1 100 % 100 % 0 0 0

Component 2 100 % 78 % 0 0 4

Component 3 87 % 80 % 0 1 1

Component 4 95 % 100 % 0 0 4

Component 5 75 % 92 % 1 1 0

Component 6 98 % 79 % 2 1 4

Overall 93 % 88 % 3 3 13

Defects Test Execution

Execution

Progress

Pass

Rate

Blocked

TestsMust-Fix Other

Component 1 100 % 100 % 0 0 0

Component 2 100 % 78 % 0 0 4

Component 3 87 % 80 % 0 1 1

Component 4 95 % 100 % 0 0 4

Component 5 75 % 92 % 1 1 0

Component 6 98 % 79 % 2 1 4

Overall 93 % 88 % 3 3 13

Defects Test Execution

http://dilbert.com/strips/comic/2007-05-16/

Dashboards

30©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com

Page 18: Making Numbers Count: Metrics That Matter

PQA Company Overview 4/26/2014

©2012 Professional Quality Assurance Ltd. (PQA) --

http://www.pqatesting.com 16

Process Improvement

Process Improvement

"Those who cannot remember the past are condemned to

repeat it.”

- George Santayana

� What is “better”? How do we know if we’re better now than

we were?

� Estimates:

‣ Comparing planned effort and actual effort

‣ Estimate variance and what factors affected it

32©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com

Page 19: Making Numbers Count: Metrics That Matter

PQA Company Overview 4/26/2014

©2012 Professional Quality Assurance Ltd. (PQA) --

http://www.pqatesting.com 17

Metrics for Process Improvement

33©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com

Closure Period / Turn-around Time

• Measures how long it takes for a reported defect to be rectified

Defect Ratio

• Can be used to predict how many defects may be found based on the number of test cases that remain to be executed

Metrics for Process Improvement

34©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com

Root Cause Analysis

• Allows you to proactively avoid anticipated defects in the future

Closed Defects by Resolution

• Identifies instances of duplicate or invalid defects

Defect Removal Efficiency

• Identifies defects that are uncovered after the application is released

Page 20: Making Numbers Count: Metrics That Matter

PQA Company Overview 4/26/2014

©2012 Professional Quality Assurance Ltd. (PQA) --

http://www.pqatesting.com 18

Trends

� One measurement captures status at a point in time.

� When a series of data points are examined, trends in that

data start to emerge.

� Trends allow us to

‣ Differentiate between systemic behaviour and temporary

anomalies.

‣ Make predictions about the future.

� Accuracy increases with more historical data.

35©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com

Misuse of Metrics

Page 21: Making Numbers Count: Metrics That Matter

PQA Company Overview 4/26/2014

©2012 Professional Quality Assurance Ltd. (PQA) --

http://www.pqatesting.com 19

Intentional Manipulation

“There are three kinds of lies: lies, damned lies, and

statistics.”

- Benjamin Disraeli

� Metrics can be manipulated to make

any situation look good or bad.

37©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com

Misleading Graphs

An example of a good graph:

38©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com

0

100

200

300

400

500

A B C D

430455 448 465

Page 22: Making Numbers Count: Metrics That Matter

PQA Company Overview 4/26/2014

©2012 Professional Quality Assurance Ltd. (PQA) --

http://www.pqatesting.com 20

Misleading Graphs

Inappropriate scale:

39©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com

0

200

400

600

800

1000

A B C D

Misleading Graphs

Scale does not start at 0:

40©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com

420

425

430

435

440

445

450

455

460

465

A B C D

Page 23: Making Numbers Count: Metrics That Matter

PQA Company Overview 4/26/2014

©2012 Professional Quality Assurance Ltd. (PQA) --

http://www.pqatesting.com 21

Tester 1Tester 2

Tester 3

Tester 4

41©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com

Misleading Graphs

Defects by Tester

11%42%

5%42%

Tester 1

Tester 2

Tester 3

Tester 4

Defects by

Tester

42©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com

Misleading Graphs

Page 24: Making Numbers Count: Metrics That Matter

PQA Company Overview 4/26/2014

©2012 Professional Quality Assurance Ltd. (PQA) --

http://www.pqatesting.com 22

Poor Choice of Format

43©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com

12%

15%

14%

8%11%

17%

13%

10% Tester 1

Tester 2

Tester 3

Tester 4

Tester 5

Tester 6

Tester 7

Tester 8

Defects by Tester

Better Choice of Format

44©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com

0%

2%

4%

6%

8%

10%

12%

14%

16%

18%

12%

15%14%

8%

11%

16%

13%

10%

Defects by Tester

Page 25: Making Numbers Count: Metrics That Matter

PQA Company Overview 4/26/2014

©2012 Professional Quality Assurance Ltd. (PQA) --

http://www.pqatesting.com 23

Overloading

� Most managers are interested in

defect statistics.

� Does defect count measure:

‣ Tester effectiveness?

‣ Developer skill?

‣ Product quality?

‣ Effort remaining?

� One metric does not tell the entire story.

45©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com

Surrogate Measures

� Sometimes, we don’t have a way to easily measure the

attribute we are really interested in.

� Instead, we use another, more convenient metric in its

place.

� But, does the metric we have chosen actually give us the

information we are looking for?

‣ To find out, we can use a “five-why” analysis

� Examples:

‣ Code Coverage

‣ Velocity

46©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com

Page 26: Making Numbers Count: Metrics That Matter

PQA Company Overview 4/26/2014

©2012 Professional Quality Assurance Ltd. (PQA) --

http://www.pqatesting.com 24

Performance Assessment

� People optimize what we measure them against, at the

expense of what we don’t.

� For example, if we assess testers based on their defect

counts, how much time will testers spend:

‣ Documenting their tests?

‣ Coaching other testers?

‣ Researching and polishing defect reports to make their

defect easier to understand, assess and replicate?

‣ Logging variants of the same failure to increase defect

count?

47©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com

Page 27: Making Numbers Count: Metrics That Matter

PQA Company Overview 4/26/2014

©2012 Professional Quality Assurance Ltd. (PQA) --

http://www.pqatesting.com 25

Key Points to Remember

� Metrics can be captured for just about any testing activity.

� Metrics play an important role in:

‣ Estimation

‣ Status reporting

‣ Evaluating suspension and exit criteria

‣ Process improvement

� Metrics, like anything, should be planned in advance.

� Reporting needs should be identified before setting up

tools.

� To avoid misinterpretation, metrics need context.

49©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com

Key Points to Remember

� Even when properly presented, too much information can

be overwhelming.

� It is best to limit the number of metrics and reports that are

presented to stakeholders to only what is essential.

� Capturing, analysing and reporting on metrics can be a

drain or waste of time if those metrics are not being used to

provide some value in return.

� Metrics influence tester performance, but not always in the

way you might think.

� Metrics should be used as a starting point for discussion.

� One metric does not tell the entire story.

50©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com

Page 28: Making Numbers Count: Metrics That Matter

PQA Company Overview 4/26/2014

©2012 Professional Quality Assurance Ltd. (PQA) --

http://www.pqatesting.com 26

Questions or Comments?

51©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com

Thanks for coming!

Contact Information:

52©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com

[email protected]

ca.linkedin.com/in/miketrites

@mike_trites

www.pqatesting.com