Upload
others
View
0
Download
0
Embed Size (px)
Citation preview
Making Sense Of Metrics
Graham Freeburn
Newell & Budge Ltd.
IMPROVING YOUR BUSINESS AGILITY
Making Sense of Metrics
Graham Freeburn
Principal Consultant [email protected]
Sopra Newell & Budge Ltd. Tel: +44(0)1698 464 239
Main Course
• Demo
• Risk Spider
Desert
The Menu
• Summary
• Some caveats
Starter• Mediated Metrics
• Metrics
Aperitif • Introduction
• Principal Testing Consultant
• 25 years in IT - fully paid up testing evangelist!
• Regular speaker at Conferences
(EuroSTAR, SQAM and BCS SIGIST)
• EuroSTAR
Member of 2002 Programme Committee
Country co-ordinator for Scotland
Track Chair
• Chairman and co-founder of the Scottish Testing Group (STG)
• Been involved in large development and testing projects within Finance, Telecommunications and Utilities industries
• Author of the advanced automation case study on Advanced Approach in Software Test Automation by Mark Fewster and Dorothy Graham
(ISBN: 0-201-33140-3 )
Graham Freeburn
• UK & Ireland arm of Sopra Group
• Provide full range of consultancy, IT delivery and support services
• Workforce of 800 based across UK, Ireland and India
• Testing Division
100+ staff
Outsourcing, Management, Improvement, Optimisation
£12M revenue in 2004/2005
• Sopra Group
Leader in European consultancy and IT services market
Total workforce of 8,500
Nearly 40 years experience across wide range of sectors
Sopra Newell & Budge
Making Sense of Metrics
We used to think that if we knew one, we knew two, because one and one
are two. We are finding that we must learn a great deal more about 'and'.
Sir Arthur Eddington, English astronomer (1882 - 1944)
Metrics
• ‘The art of using poetic metre’
OED
• „Measurements, collections of data about activities, resources and/or
deliverables”.
retailindustry.about.com/od/abouttheretailindustry/l/blterm_m.htm’
• „Measurements taken over time that monitor, assess, and
communicate vital information about the results of a program or
activity. Metrics are generally quantitative, but can be qualitative‟
fast.faa.gov/archive/v999/ams/app-c.htm
There are three types of lies : lies, damned lies and statistics
Benjamin Disraeli, British Politician (1804 - 1881)
Mediated Metrics – Are…
• What they are not:
A way to hide the facts or lies!
Something to be used without agreement
Applicable in all circumstances
• What they are:
Measurements that are visual presented using measures and scales
preset by the metric owner to reflect variations in their significance
according to the project context and status
Variable RAG ratings on a chosen set of metrics that can be successfully
subjected to justification and analysis.
Mediated Metrics - Principles
• Dashboard
Visual
Aim to rapidly provide visual information on which decisions can be taken
• Small Number
Best used where a limited set of metrics are chosen to monitor progress / status
• Scales
Vary according to scope / time etc.
• 100 faults in week1 – no problem
• 100 faults in last week of project – major problem
Mediated Metrics - Benefits
• Simplify Presentation
• Deliver Clear Message
• Match the changing risk profile of a project over time
• Are flexible
• Have high impact
Where it all started..
Graphics….
• Minard's Graphic of Napoleon's Moscow Campaign of
1812 [Tufte, 1983, p. 41, © Graphics Press]
Kiviat / Radar Chart
Axis: The measure of interest
Value: Plotted on relevant axis
Real value or Scaled
Points connected to create pattern
Rating Ranges
Code Turmoil(loc) 6 1 = 0-50, 2 = 51-100, 3 = 101-150, 4 = 151-200, 5 = 201-250, 6 > 250
Defects found this week 6 1 = 0-2, 2 = 3-4, 3 = 5-6, 4 = 7-8, 5 = 9-10, 6 > 10
Total Open Defects 2 1 = 0-10, 2 = 11-20, 3 = 21-30, 4 = 31-40, 5 = 41-50, 6 > 50
Test Success Rate 1 1 = 100%-98%, 2 = 97%-95%, 3 = 94%-92%, 4 = 91%-89%, 5 = 88%-86%, 6 < 86%
Test Completion Rate 5 1 = 100%-95%, 2 = 94%-90%, 3 = 89%-85%, 4 = 84%-80%, 5 = 79%-75%, 6 < 75%
TOTAL RATING 20 5 to 10 11 to 15 >15
Raw Data 10-Sep-01 17-Sep-01 24-Sep-01 01-Oct-01
Code Turmoil 1127 1409 1020 799
Defects found this week 23 16 6 13
Total Open Defects 24 32 7 12
Test Success Rate 95.50% 98.70% 99.50% 99.80%
Test Completion Rate 44.80% 70.00% 71.80% 78.90%
0123456
Code Turmoil(loc)
Defects found this week
Total Open DefectsTest Success Rate
Test Completion Rate
Source: Mike Ennis, Managing the End Game of a Software
Project, StarEAST 2001
Managing the end game…
The Risk Spider
Setup - Scope
• Scope
Time Period
• Months / Weeks / Sessions
Project
• Single Project or Multiple Projects
Test Run
• E.g. cycles in end-to-end testing
Function / Component
Setup - Mediation
• Metrics
Product Quality, Coverage,
Bug Discovery, Bug Turnaround
Volatility, Resource Availability
• Scale ranges
Single, Variable, Specific
• Scale values
Reflect the scope of the chosen metric
Conversion of actual metric value to „ranking‟ of 1 – 6 scale
Use
• Setup
Needs consensus
Documented by baselining
Driven by the project(s) context
• Input
Direct
Import from external sources
• Single / Multiple View
Function 10 : Orders
0
1
2
3
4
5
6Bug Turnaround
Bug Discovery
Product QualityResource Availability
Risk Coverage
Examples
Period 6 : 07/11/03
0
1
2
3
4
5
6
Defects Found
Total Open Defects
Test SuccessTest Completion
Productive Time
Project 5 : ProjectID5
0
1
2
3
4
5
6Defects Found
Total Open Defects
Test SuccessTest Completion
Risk Coverage
Demo
• Excel spreadsheet
• Can be linked to Test Management tools
Next Steps
• Dynamic metric selection
• Improved integration with test management
tools
How they help Test Managers
• Mediated Metrics enable
simpler reporting
clearer message to project / business managers
• Leading to
higher perceived value of testing
greater influence
increased reliance on management skills
Mediated Metrics
• Caveats
You could be accused of cooking the books!
Preparation, Preparation
• Understand context, chose scales carefully, get agreement
Management
• Remove unnecessary detail but need care for precisely that reason
Justification
• Drill down information must always be available
• Any changes during the project must be auditable
Statistician:
A man who believes figures don't lie, but admits that under analysis
some of them won't stand up either.
Evan Esar, American Humorist (1899 - 1995)
Summary
• Consider using mediated metrics in your Test Managers
tool kit
• Remember they require intelligent use
Be open and honest
Be careful
Be Pro-active
• Like all management aids are only valuable if you always
remember to actually MANAGE !
Questions
Acknowledgements /Resources
• Acknowledgements
Mike Ennis, Managing the End Game of a Software Project, StarEAST 2001
Grove Consultants, Basic Risk Spider spreadsheet, ISEB Practitioner materials
Isobel Evans, Get Your Message Across: Reporting for Maximum Impact, EuroSTAR 2002
David Tegarden, Business Information Visualization, CAIS, 1999
• Resources
Tuffe, “Visual Display of Quantitative Information”, 1983
Kolence, K. and Kiviat, P., "Software Unit Profiles and Kiviat Figures." Performance Evaluation Review (2:3) September 1973, pp. 2-12.
More details of risk spider (slides / mindmaps on mediated metrics)
• www.newellandbudge.com/services/testing