Combining Estimates with Planning Poker - An Empirical Study

Kjetil Moløkken-ØstvoldNils-Christian Haugen April 2007

2

Agenda

• Combining expert estimates

• Planning Poker

• Study results

• Summary

• Q&A

3

Background

• Warning! Much of the software engineering literature misinterprets psychological research

• Combination of expert estimates might reduce over-optimism

• Lack of empirical research

– Properties of various methods used to combine estimates

– Project issues (customer, priorities, management)

– Types of experts involved

4

An overview of some methods for combining estimatesMethod Structure Anonymity Interaction Overhead

Delphi Heavy Yes No Major

Wideband Delphi

Moderate Limited Limited Moderate

PlanningPoker

Light No Yes Limited

Unstructured group

None No Yes Limited

5

Planning Poker

• James W. Grenning (www.objectmentor.com)

• Process1. Task presented

2. Task is discussed (defines the tasks before estimating)

3. Individual estimates derived (combines knowledge from several sources)

4. Estimates revealed simultaneously (reduces social comparison concerns)

5. High/low estimator justifies

6. Consensus sought

6

Study background

• Twofold purpose of study– Investigate possible reduction of optimism in the PP

tasks

– Compare PP tasks with individual estimates

• Planning poker estimates performed in sprint planning– 14-day sprints

– Data from 4 sprints

– 50% of tasks re-estimated using planning poker

• Most likely estimates in hours (not predefined units)

7

Team and methodology

• Consultants from same company– 4-6 people total

– All developers

• Scrum– Solo programming

– Tasks tested and QAed by another person on the team

– Tasks kept in issue tracking system (Jira)

– Daily scrum

8

Results from the tasks estimated with planning poker• Compared with the mechanical combination of

individual estimates, the consensus estimates were

– less optimistic after discussion

– more accurate after discussion

• This is opposite of what is found in most studies from other areas

• Can be caused by different perspectives on a task and/or identification of sub-problems

9

PP Ind. Comment

Median 0,00 0,00 Typical case on target for both groups

Mean 0,33 -0,04 Some PP tasks were underestimated

Results PP vs. Individual

Measure:BRE-bias = (actual - estimate) / min(actual, estimate)

10

Results from analysis of code

• Planning poker tasks had on average:– Twice as many deleted control statements

(indicates that effort was spent to reduce complexity)

– Twice as many out-of-class references deleted (indicates that effort was spent to reduce coupling)

• Extra time spent on restructuring and simplifying code can explain why there were some overruns in the planning poker tasks

• Question: Did planning poker (group discussion) lead to increased focus on quality?

11

Possible benefits of Planning Poker

• Includes several perspectives

• Participants gets more ownership of estimates

• Uncertainty related to the implementation can be discussed and handled at an early stage

• Cost-efficient

12

Possible hazards of Planning Poker

• Decision may be vulnerable to peer-pressure

• Often not easy to deviate from original estimate(s)

• An unstructured discussion might have side-effects, such as increasing number of sub-tasks

13

Summary

• Combination of estimates may increase accuracy, but can have certain side-effects

– Does planning poker lead to more focus on quality?

• Planning poker was popular among developers

– Found in Norway and previously in the UK

– Rated as fun and easy to implement

• Choice of combination method should depend on project characteristics

14

Questions?