Making Sense Of Metrics

Graham Freeburn

Newell & Budge Ltd.

IMPROVING YOUR BUSINESS AGILITY

Making Sense of Metrics

Graham Freeburn

Principal Consultant graham.freeburn@newellandbudge.com

Sopra Newell & Budge Ltd. Tel: +44(0)1698 464 239

Main Course

• Demo

• Risk Spider

Desert

The Menu

• Summary

• Some caveats

Starter• Mediated Metrics

• Metrics

Aperitif • Introduction

• Principal Testing Consultant

• 25 years in IT - fully paid up testing evangelist!

• Regular speaker at Conferences

(EuroSTAR, SQAM and BCS SIGIST)

• EuroSTAR

Member of 2002 Programme Committee

Country co-ordinator for Scotland

Track Chair

• Chairman and co-founder of the Scottish Testing Group (STG)

• Been involved in large development and testing projects within Finance, Telecommunications and Utilities industries

• Author of the advanced automation case study on Advanced Approach in Software Test Automation by Mark Fewster and Dorothy Graham

(ISBN: 0-201-33140-3 )

Graham Freeburn

• UK & Ireland arm of Sopra Group

• Provide full range of consultancy, IT delivery and support services

• Workforce of 800 based across UK, Ireland and India

• Testing Division

100+ staff

Outsourcing, Management, Improvement, Optimisation

£12M revenue in 2004/2005

• Sopra Group

Leader in European consultancy and IT services market

Total workforce of 8,500

Nearly 40 years experience across wide range of sectors

Sopra Newell & Budge

Making Sense of Metrics

We used to think that if we knew one, we knew two, because one and one

are two. We are finding that we must learn a great deal more about 'and'.

Sir Arthur Eddington, English astronomer (1882 - 1944)

Metrics

• ‘The art of using poetic metre’

OED

• „Measurements, collections of data about activities, resources and/or

deliverables”.

retailindustry.about.com/od/abouttheretailindustry/l/blterm_m.htm’

• „Measurements taken over time that monitor, assess, and

communicate vital information about the results of a program or

activity. Metrics are generally quantitative, but can be qualitative‟

fast.faa.gov/archive/v999/ams/app-c.htm

There are three types of lies : lies, damned lies and statistics

Benjamin Disraeli, British Politician (1804 - 1881)

Mediated Metrics – Are…

• What they are not:

A way to hide the facts or lies!

Something to be used without agreement

Applicable in all circumstances

• What they are:

Measurements that are visual presented using measures and scales

preset by the metric owner to reflect variations in their significance

according to the project context and status

Variable RAG ratings on a chosen set of metrics that can be successfully

subjected to justification and analysis.

Mediated Metrics - Principles

• Dashboard

Visual

Aim to rapidly provide visual information on which decisions can be taken

• Small Number

Best used where a limited set of metrics are chosen to monitor progress / status

• Scales

Vary according to scope / time etc.

• 100 faults in week1 – no problem

• 100 faults in last week of project – major problem

Mediated Metrics - Benefits

• Simplify Presentation

• Deliver Clear Message

• Match the changing risk profile of a project over time

• Are flexible

• Have high impact

Where it all started..

Graphics….

• Minard's Graphic of Napoleon's Moscow Campaign of

1812 [Tufte, 1983, p. 41, © Graphics Press]

Kiviat / Radar Chart

Axis: The measure of interest

Value: Plotted on relevant axis

Real value or Scaled

Points connected to create pattern

Rating Ranges

Code Turmoil(loc) 6 1 = 0-50, 2 = 51-100, 3 = 101-150, 4 = 151-200, 5 = 201-250, 6 > 250

Defects found this week 6 1 = 0-2, 2 = 3-4, 3 = 5-6, 4 = 7-8, 5 = 9-10, 6 > 10

Total Open Defects 2 1 = 0-10, 2 = 11-20, 3 = 21-30, 4 = 31-40, 5 = 41-50, 6 > 50

Test Success Rate 1 1 = 100%-98%, 2 = 97%-95%, 3 = 94%-92%, 4 = 91%-89%, 5 = 88%-86%, 6 < 86%

Test Completion Rate 5 1 = 100%-95%, 2 = 94%-90%, 3 = 89%-85%, 4 = 84%-80%, 5 = 79%-75%, 6 < 75%

TOTAL RATING 20 5 to 10 11 to 15 >15

Raw Data 10-Sep-01 17-Sep-01 24-Sep-01 01-Oct-01

Code Turmoil 1127 1409 1020 799

Defects found this week 23 16 6 13

Total Open Defects 24 32 7 12

Test Success Rate 95.50% 98.70% 99.50% 99.80%

Test Completion Rate 44.80% 70.00% 71.80% 78.90%

0123456

Code Turmoil(loc)

Defects found this week

Total Open DefectsTest Success Rate

Test Completion Rate

Source: Mike Ennis, Managing the End Game of a Software

Project, StarEAST 2001

Managing the end game…

The Risk Spider

Setup - Scope

• Scope

Time Period

• Months / Weeks / Sessions

Project

• Single Project or Multiple Projects

Test Run

• E.g. cycles in end-to-end testing

Function / Component

Setup - Mediation

• Metrics

Product Quality, Coverage,

Bug Discovery, Bug Turnaround

Volatility, Resource Availability

• Scale ranges

Single, Variable, Specific

• Scale values

Reflect the scope of the chosen metric

Conversion of actual metric value to „ranking‟ of 1 – 6 scale

Use

• Setup

Needs consensus

Documented by baselining

Driven by the project(s) context

• Input

Direct

Import from external sources

• Single / Multiple View

Function 10 : Orders

0

1

2

3

4

5

6Bug Turnaround

Bug Discovery

Product QualityResource Availability

Risk Coverage

Examples

Period 6 : 07/11/03

0

1

2

3

4

5

6

Defects Found

Total Open Defects

Test SuccessTest Completion

Productive Time

Project 5 : ProjectID5

0

1

2

3

4

5

6Defects Found

Total Open Defects

Test SuccessTest Completion

Risk Coverage

Demo

• Excel spreadsheet

• Can be linked to Test Management tools

Next Steps

• Dynamic metric selection

• Improved integration with test management

tools

How they help Test Managers

• Mediated Metrics enable

simpler reporting

clearer message to project / business managers

• Leading to

higher perceived value of testing

greater influence

increased reliance on management skills

Mediated Metrics

• Caveats

You could be accused of cooking the books!

Preparation, Preparation

• Understand context, chose scales carefully, get agreement

Management

• Remove unnecessary detail but need care for precisely that reason

Justification

• Drill down information must always be available

• Any changes during the project must be auditable

Statistician:

A man who believes figures don't lie, but admits that under analysis

some of them won't stand up either.

Evan Esar, American Humorist (1899 - 1995)

Summary

• Consider using mediated metrics in your Test Managers

tool kit

• Remember they require intelligent use

Be open and honest

Be careful

Be Pro-active

• Like all management aids are only valuable if you always

remember to actually MANAGE !

Questions

Acknowledgements /Resources

• Acknowledgements

Mike Ennis, Managing the End Game of a Software Project, StarEAST 2001

Grove Consultants, Basic Risk Spider spreadsheet, ISEB Practitioner materials

Isobel Evans, Get Your Message Across: Reporting for Maximum Impact, EuroSTAR 2002

David Tegarden, Business Information Visualization, CAIS, 1999

• Resources

Tuffe, “Visual Display of Quantitative Information”, 1983

Kolence, K. and Kiviat, P., "Software Unit Profiles and Kiviat Figures." Performance Evaluation Review (2:3) September 1973, pp. 2-12.

More details of risk spider (slides / mindmaps on mediated metrics)

• www.newellandbudge.com/services/testing