No estimates - 10 new principles for testing

#NoEstimates

10 New Principles for

Software Development

(and testing!)

Vasco Duarte@duarte_vasco

Tweet at:#TAHelsinki

Vasco Duarte

@duarte_vasco

http://SoftwareDevelopmentToday.com

http://bit.ly/vasco_slideshare

Vasco.Duarte@oikosofy.com

http://NoEstimatesBook.com

#NoEstimates is for you if…

Principle #1Trust your process, or change your

process

Nu

mb

er

of

Bu

gs

Timeline

Bug evolution in a non-agile project

Open

Closed

Submit

Development phase Desperately testing and fixing phase

Waterfall

Principle #2Shorten the feedback cycle

Can estimates be accurate?

Accidental complication

Cost/Duration =

Essential Complication

x

Accidental Complication

Yet, some people still argue we can be good at estimating… Let’s

look at the evidence

80% Late or Failed

Source: Software Estimation by Steve McConnell

Reality: 80% is late or failed

Lets break that down

Chaos Report 1995For the same combined challenged and impaired projects, over one-third also experienced time overruns of 200 to 300%. The average overrun is 222% of the original time estimate. For large companies, the average is 230%; for medium companies, the average is 202%; and for small companies, the average is 239%.

Time Overruns % of responses

Under 20% 13.9%

21 – 50% 18.3%

51 – 100% 20.0%

101 – 200% 35.5%

201 – 400% 11.2%

Over 400% 1.1%

~68% of projects 51% or more late!

Let’s continue to break that down

Chaos Report 2009

Average cost overrun: 45%

Average time overrun: 63%

Chaos Report 2011

Average time overrun: 63%

Just in case you don’t like the CHAOS report

Source Gartner survey of project failure, 2012

Failure, means total failure, not just late

Of the large systems that are completed, about 66% experience schedule

delays & cost overrun

Source: “Project Management Tools and Software Failures and Successes” by Capers JonesCrosstalk, the Journal of Defense Software Engineering

Traditional projects: 53% failed or challenged

Source: Project success survey by Scott Ambler

Agile project: 40% failed of challenged

Source: Project success survey by Scott Ambler

17 percent of large IT projects go so badly that they can threaten the very existence of the company

Source : McKinsey & Company in conjunction with the University of OxfordType of survey : Study on large scale IT Projects

Date : 2012

More data coming, sign-up at NoEstimatesBook.com to receive

this report when available

Principle #3Believe the data, not the estimates

In 1986, Profs. S.D. Conte, H.E. Dunsmoir, andV.Y. Shen proposed that a good estimationapproach should provide estimates that arewithin 25% of the actual results 75% of the time

--Steve McConnel, Software Estimation: Demystifying the Black Art

Principle #4Use alternatives to Estimate-driven

decision making

Comparison of 17 projects ending between 2001 and 2003. (Average: 62%)

Testing for value, a story…

Principle #5Test for value first, then test for

functionality

Can #NoEstimates work in RealCompanies?

http://j.mp/NoEstimates-Book

Principle #6Estimation is waste, reduce it’s

impact on your business

Principle #7Measure progress only with validated, running software

In 1986, Profs. S.D. Conte, H.E. Dunsmoir, andV.Y. Shen proposed that a good estimationapproach should provide estimates that arewithin 25% of the actual results 75% of the time

--Steve McConnel, Software Estimation: Demystifying the Black Art

Source: Software Estimation by Steve McConnell

“Good” estimates: 25% of estimated

I AM GOING TO GO AHEAD AND

ASK YOU TO DELIVER 10

STORIES NEXT SPRINT...

WTF!!!!!!#%&!

0

1

2

3

4

5

6

7

8

9

10

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

velocity

Average=

LCL

UCL

Target

Actual, measured throughput over 21 sprints

The 4 Estimates Dysfunctions

1. Estimate Bargaining

2. Internal Politics

3. Blame Shifting

4. Late Changes

5. Sunk Cost fallacy

Principle #8The system where you work has

predictable outputs, learn to understand the system

Wow! But I have a business to run!

Is there a way do better than that?

#NoEstimates delivers!

Counting Stories vs. Estimated Story Points

Q: Which ”metric” is more accurate when compared to

what actually happened in the project?

A long project

24Sprints

Which metric predicted most accurately the output of the

whole project?

a) After only the first 3 Sprints

b) After only the first 5 Sprints

Disclaimer...This is only one project!

Find 21 more at: http://bit.ly/NoEstimatesProjectsDB

After just 3 sprints

# of Stories predictive powerStory Points predictive power

The true output: 349,5 SPs

completed

The predictedoutput: 418 SPs

completed

+20%

The true output: 228 Stories

The predictedoutput: 220

Stories

-4%!

After just 5 sprints

# of Stories predictive powerStory Points predictive power

The true output: 349,5 SPs

completed

The predictedoutput: 396 SPs

completed

+13%

The true output: 228 Stories

The predictedoutput: 220

Stories

-4%!

Q: Which ”metric” is more accurate when compared to

what actually happened in the project?

But there is more...

What difference does a Story Point make in a project that used both Story Points and

#NoEstimates?

Next you will see the forecasted release date when

using Story Points (values 1:3:5)

6871 71 71 71 71 71 72 72 72 73 73

0 3 7 7 9 11 12 13

20 20 22 23

0

10

20

30

40

50

60

70

80

90

100

1/3 1/17 1/31 2/14 2/28 3/14 3/28 4/11 4/25 5/9 5/23 6/6 6/20 7/4 7/18 8/1 8/15 8/29 9/12 9/26 10/10 10/24 11/7

Product Backlog Cumulative Flow Diagram

Remaining

Done

Linear (Remaining)

Linear (Done)

Release on 20th October

2014

Next you will see the forecasted release date when

using Story Points (values 1:2:3)

48

51 51 51 51 51 51 52 52 52 53 53

0 2 5 5 7 8 9 1015 15 17 18

0

10

20

30

40

50

60

70

80

1/3 1/17 1/31 2/14 2/28 3/14 3/28 4/11 4/25 5/9 5/23 6/6 6/20 7/4 7/18 8/1 8/15 8/29 9/12 9/26 10/10 10/24 11/7

Product Backlog Cumulative Flow Diagram

Remaining

Done

Linear (Remaining)

Linear (Done)Release on

14th October 2014

Next you will see the forecasted release date when

#NoEstimates (or, all stories = 1 story point)

28

31 31 31 31 31 3132 32 32

33 33

0 1 3 35 5 6 7

10 1012 13

0

10

20

30

40

50

60

1/3 1/17 1/31 2/14 2/28 3/14 3/28 4/11 4/25 5/9 5/23 6/6 6/20 7/4 7/18 8/1 8/15 8/29 9/12 9/26 10/10 10/24 11/7

Product Backlog Cumulative Flow Diagram

Remaining

Done

Linear (Remaining)

Linear (Done)

Release on 29thSeptember 2014

Conclusion...

All dates within 3 weeks of each other in a 38 to 42 week

project (a margin of ~8%)

Data used with permission from Bill Hanlon at Microsoft

”At that point, I stopped thinking that estimating

was important.”

Bill Hanlon: http://bit.ly/BHanlon

In 1986, Profs. S.D. Conte, H.E. Dunsmoir, andV.Y. Shen proposed that a good estimationapproach should provide estimates that arewithin 25% of the actual results 75% of the time

--Steve McConnel, Software Estimation: Demystifying the Black Art

In this presentation you have seen examples that far outperform what estimation specialists consider a ”good estimation”. In common they have one way to look at work: #NoEstimates

#NoEstimates testimonial

The deviation between estimated and actual velocity would have been approximately 15% lower if we would have used #NoEstimates.

We have analyzed data from 50 Sprints…

…at no time the story point based estimation was better than #NoEstimates.

Principle #9Don’t bet your company on poor

track record methods, use methods with a proven track record

aka: Hope is a bad management strategy

Carmen faces a very difficult situation…

The 7 Step Journey To NoEstimates

1. Start using Story Points 2. Stop estimating tasks3. Limit the calendar duration for

Stories and Features4. Reduce the number of

allowed estimates (say, 1,2,3 and 5 only)

5. Track your data6. Use your data7. Simply count the number of

stories

http://j.mp/NoEstimates-Book

http://j.mp/NoEstimates-Book

http://j.mp/NoEstimates-Book

Principle #10The transformation starts with you…

Vasco DuarteVasco.Duarte@oikosofy.com