Measuring for Results: Metrics and Myths · Metrics and Myths Peter Hundermark Certiﬁed Scrum Coach and Trainer 1 ScrumSense Tuesday 01 September 2009. Project Success Rates 32%

Measuring for Results:

Metrics and Myths

Peter Hundermark

Certified Scrum Coach and Trainer

1

ScrumSense

Tuesday 01 September 2009

Project Success Rates

32%

44%

24%

2

Source: Standish Group CHAOS Summary 2009

Succeeded

Challenged

Failed

On time

On budget

Required features

Late

Over budget

Missing features

Cancelled

Never used


Agile is Value-Driven

Cost

Constraints

Estimates

Requirements

Cost Schedule

Value Driven

Features

Cost Schedule

Predictive Adaptive

Plan Driven

3


What is Valued?

45%

19%

16%

13%

7%

Never

Rarely

Sometimes

Often

Always

4

Source: Standish Group (2002)

20% frequently used

64% rarely or never used


Real V

alu

e D

eliv

ere

d /

Earn

ed

Time

Delivery 4

Value of Time

Incremental delivery

Traditional single delivery

Value gap

5

Delivery

1 - 5

Delivery 5

Delivery 3

Delivery 2Delivery 2


knowledge

workers

measure themselves

6


Eli Goldratt “The Goal” (2004)7

Tell me how you

will measure me

and I’ll tell you

how I will behave.


On the Folly...

Adapted from Steven Kerr “On the folly of rewarding A, while hoping for B”, Academy of Management Executive (1995 vol 9 no 1)

We hope for…

• Long-term growth

• Commitment to quality

• Teamwork and collaboration

• Innovative thinking and risk taking

• Employee involvement and empowerment

• Candour: surfacing bad news early

But we often reward…

• Quarterly earnings

• Shipping with defects

• Individual effort

• Proven methods and not making mistakes

• Tight control of operations and resources

• Reporting good news, whether it’s true or not

Agreeing with the boss, whether or not (s)he’s

right!

8


Metrics and Myths

Leve

l of p

erf

orm

ance

Time

Measurement indicators

True performance

Source: Robert Austin, “Measuring and Managing Performance in Organisations” (1996)

9


Good Agile Metrics

10


Good Agile Metrics

Reinforce desired behaviour

10


Good Agile Metrics


Measure results not output

10


Good Agile Metrics



Measure trends

10


Good Agile Metrics



Measure trends

Vital few

10


Good Agile Metrics



Measure trends

Vital few

Easy to collect

10


Good Agile Metrics



Measure trends

Vital few

Easy to collect

Reveal their context

10


Good Agile Metrics



Measure trends

Vital few

Easy to collect


Fuel meaningful conversation

10


Good Agile Metrics



Measure trends

Vital few

Easy to collect



Amplify learning

10


Good Agile Metrics



Measure trends

Vital few

Easy to collect



Amplify learning

May measure value or process

10


Good Agile Metrics



Measure trends

Vital few

Easy to collect



Amplify learning

May measure value or process

Good enough quality

10


Leading and Lagging

11


Leading and Lagging

Leading indicator

Signals future events

Changes before the event

E.g. amber traffic light

11


Leading and Lagging

Leading indicator

Signals future events

Changes before the event

E.g. amber traffic light

Lagging indicator

Reports past events

Changes as an outcome

E.g. unemployment

11


Measurement Dimensions

12

Value(to Customer)

Collaboration(Process)

Quality(Product)

Predictability(Schedule)

Source: Pete Behrens, “Measuring Agility - Top 5 Metrics and Myths” (2009)


Customer Surveys

Many opportunities over time

Baseline and measure quarterly

Qualitative and quantitative

Responsiveness

On-time delivery

Value

Quality

Support

13

Source: Pete Behrens, “Measuring Agility -

Top 5 Metrics and Myths” (2009)

Value

Lagging

Metric


Customer Surveys

Ask: “How likely are you to recommend

this product or service to a friend?”

Scale 0 - 10

9 - 10 ! Promoters

7 - 8 !Passive

0 - 6 !Detractors

Net promoters score

(promoters - detractors) ÷ sample size

0 1 2 3 4 5 6 7 8 9 10

12

15

10

8

5

23

2111

Adapted from Sanjiv Augustine and Roland Cuellar, “Agile Metrics for Senior Managers & Executives” (2009)

Value

Lagging

14


Customer Surveys

Ask: “How likely are you to recommend

this product or service to a friend?”

Scale 0 - 10

9 - 10 ! Promoters

7 - 8 !Passive

0 - 6 !Detractors

Net promoters score

(promoters - detractors) ÷ sample size

0 1 2 3 4 5 6 7 8 9 10

12

15

10

8

5

23

2111

27

14

Net promoters score:

(27-14) / 57 = 23%

Adapted from Sanjiv Augustine and Roland Cuellar, “Agile Metrics for Senior Managers & Executives” (2009)

18

Value

Lagging

14


Velocity

0

10

20

30

40

1 2 3 4 5 6 7 8

Velo

city

Sprints

15

Predict

Leading


Cum

ula

tive C

ost

Time

Optimal Cost

Actual Cost

16

Myth

Higher velocity is always a good thing

Truth

Risk of incurring technical debt

Do

Add technical debt to product backlog

Monitor and reduce over time

Myth


Source: Pete Behrens, “Measuring Agility - Top 5 Metrics and Myths” (2009)

17

Velocity measures productivity

or value

Truth

Story points are relative

Cannot compare velocity

Teams are different

Size measures complexity

Do

Use the metric as designed

Myth


18

0

10

20

30

40

1 2 3 4 5

Velo

city

Sprints

Committed Actual

100% Committed vs. actual

drives estimation accuracy

Truth

Teams learn to be safe

Leads to lower productivity

Better

What features did we deliver?

Why did we miss one?

What is the impact?

! Learning

Myth


Source: Donald Reinertsen “Managing the Design Factory: A

Product Developer’s Toolkit (1997)

19

Sprints fail

Truth

Less probable events ! more information

Maximum information at 50% probability

Knowledge from low-probability events

Myth


Rate of Features Delivered

Product or Release

Burnup / Burndown Chart

0

50

100

150

200

250

300

0 1 2 3 4 5 6 7 8

Sto

ry P

oin

ts

Sprints

Story points delivered Story points remainingPredict

Leading

Metric

20


Running Tested Features / Running Automated Tests

Source: Ron Jeffries, http://xprogramming.com/xpmag/jatRtsMetric (2004)

21

QualityLeading Predict Metric


Technical Debt

Quality

Leading

Metric

22

What is

technical debt?

Why does it

occur?

Add stories to

backlog

Team is

responsible!


Work-in-process

Collab.

Leading

Metric

23

Stories in-process

Aim for " 2

No silos or blockages!

Story ToDo WIP Done


Story ToDo WIP Done

Story Cycle Time

0

10

20

30

1 2 3 4 5 6 7 8 9 10

Sto

ry P

oin

ts

Days

Collab.

Leading

Metric

Average time to complete

Aim for around 3 days

No lagging tests!

24


Basic Agile Metrics

Value

Collaboration Product Quality

Predicability

25

Velocity

RTF / running

automated tests

Work-in-process Technical debt

Burnup /

Burndown

Story cycle time

Customer

surveys


Extending the Metrics

Value


Predicability

26

Velocity

RTF / running

automated tests


Burnup /

Burndown

Story cycle time

Customer

surveys



Value


Predicability

26

Velocity

RTF / running

automated tests


Burnup /

Burndown

Story cycle time

Customer

surveys



Value


Predicability

26

Velocity

RTF / running

automated tests


Burnup /

Burndown

Story cycle time

Customer

surveys

Team

surveys



Value


Predicability

26

Velocity

RTF / running

automated tests


Burnup /

Burndown

Story cycle time

Customer

surveys

Team

surveys

Cost per

sprint / point



Value


Predicability

26

Velocity

RTF / running

automated tests


Burnup /

Burndown

Story cycle time

Customer

surveys

Team

surveys

Cost per

sprint / point

Real value

delivered



Value


Predicability

26

Velocity

RTF / running

automated tests


Burnup /

Burndown

Story cycle time

Customer

surveys

Team

surveys

Cost per

sprint / point

ROI / NPV

Real value

delivered


Metrics and MythsPuzzles?27


Peter Hundermark

28

Copyright in these slides is owned by Peter Hundermark and Scrum Sense CC. Some content and images may the

copyright of others.

Further information on Scrum training and coaching is available from:

[email protected]

www.scrumsense.com

ScrumSense


http://www.scrumsense.com

mailto:[email protected]

mailto:[email protected]

http://www.scrumsense.com

Documents

Measuring for Results: Metrics and Myths · Metrics and Myths Peter Hundermark Certiﬁed Scrum Coach and Trainer 1 ScrumSense Tuesday 01 September 2009. Project Success Rates 32%