Estimation Bias and Mitigation With Agile Estimate ... · Harvard Business Review explains this Nobel Prize Winning Phenomenon: • Humans seem hardwired to be optimists • Routinely

Estimation Bias and MitigationWith Agile Estimate Guidance

2017 Edition China November 2017

Dan Galorath

[email protected]

+1 310 414-3222 x614

© 2017 Copyright Galorath Incorporated

mailto:[email protected]

Key Points

Every forecast is subject to estimation

bias…a major cause of

program failure and corporate mis-spending

Agile is a good

thing.

Estimates are still

important

Experts Bias is probably costing

in cost, schedule, and

less than hoped for benefits

2

Human Nature: YOUR PEOPLE Are Optimism Biased

Harvard Business Review explains this Nobel Prize Winning Phenomenon:

• Humans seem hardwired to be optimists

• Routinely exaggerate benefits and discount costs

• Bias permeates opinions & decisions & causes waste & failure

Delusions of Success: How Optimism Undermines Executives' Decisions (Source: HBR Articles | Dan Lovallo, Daniel Kahneman | Jul 01, 2003)

3

Solution - Temper with “outside view”:

Past Measurement Results, traditional forecasting, risk

analysis and statistical parametrics can help

Don’t remove optimism, but balance optimism and

realism

http://hbr.org/search/Dan Lovallo/

http://hbr.org/search/Daniel Kahneman/

Mitigate Bias With Reference Class Forecasting (Source SEER-SEM)

Anchoring Experiment: Anchoring Biases Estimates (Source: myweb.liu.edu/~uroy/eco23psy23/ppt/04-anchoring.pptx)

1. Subject witnesses the number that comes up when a wheel of fortune is spun

2. Is asked whether the number of African countries in the U.N. is greater than or less than the number on the wheel of fortune

3. Is asked to guess the number of African countries in the U.N.

Result: those who got higher numbers on the

wheel of fortune guessed bigger numbers

in Step 3If given a number that biases estimates

Flaw of Averages Case Studies (Source: HBR)

• Example: $2 billion property damage in North Dakota

• U.S. Weather Service forecast that North Dakota’s rising Red River would crest at 49 feet.

• Made flood management plans based on this average figure

• In fact, the river crested above 50 feet, breaching the dikes, and unleashing a flood that forced 50,000 people from their homes.

© 2016 Copyright Galorath Incorporated 6

Average productivity estimates or average velocity can be very inaccurate

Beware business decisions with such data

Agile: Business Leaders Need Estimates for Decision MakingAnd Bias Against Consistent Measurement Is Hurting Industry

© 2017 Copyright Galorath Incorporated7

Agile: Detailed Software Development Life Cycle Management (Scrum Example)

• Focus is on what features can be delivered per iteration

• Not fully defined what functionality will be delivered at the end?

• Iterations are often called “Sprints”


Agile Is Not a Silver Bullet http://www.jamasoftware.com/blog/rethink-agile-manifesto-projects-still-fail/

• Projects still fail at roughly the same rate as 2001

• Dr. Dobbs: Agile is not a productivity revolution


0%

20%

40%

60%

80%

Agile (72%) Traditional (64%)8% Improvementnot Revolution

Agile 8% More Success

Some think agile is more successful because they remove cost & schedule

goals from the evaluation

Agile Software Project Performance Still Disappointing (McKinsey)

-80%

-60%

-40%

-20%

0%

20%

40%

60%

45% Over Budget 56% less Functionality thanplanned

McKinsey Study: Projects over $15m


So does skipping estimating solve this?Well: If commitments aren’t made

there can be no disappointment

Story Points: Good For Sprint Planning, Lacking for Project Decisions(Source I Brown)

Story Point Benefits

• Excellent for sprint planning works well for planning sprints

• Immediacy of feedback from previous sprint

• Encourages communication & expectation management

Challenges

• Generation of estimates to establish project budgets

• New development team with no history

• No consistent portfolio management metrics across organization


Story Points Ineffective for Absolute Effort Estimation (Comyene-Abran)


Relative Error% - (realEffort –Estimated Effort / RealEffort

Agilons: Standardized Sizing For Agile Projects (Source: SEER, I Brown)

• As a customer I would like to have the ability to search for and reserve a hotel room in order to spend the night in another city

• Using Agilon

• User story seems to have multiple Agilontypesthat need to be decomposed

• Hotel data (10 Agilons)

• Search for hotel room (4 Agilons)

• Reserve hotel room (4 Agilons)

• Total of 18 Agilons

• •If my team’s velocity is around 18 Agilons per sprint, we’re good to go…


Agile Trying To Kill Estimates Without Considering Business Needs

• #noestimates

There Are Times When #Noestimate Is Appropriate

• A promise of 5 sprints with 4 people is good enough

• Projects are small and time is not concerning

Unsubstantiated Promise Good

Enough

•Willing to spend in whatever time

•No Business impact from software

•No concern with failure and contingency

•Resource optimization not concerning

•No system testing on top of development

Cost / Schedule not a

consideration

• You don’t need to consider total ownership costs

• If lack of documentation is ok for maintenance

No Maintenance COncern


Manual Estimates: Human Reasons For Error EVEN IN AGILE SPRINTS (Goldratt)

• Manual Task estimates yield SIGNIFICANT error

• Desire for “credibility” motivates overestimate behavior (80% probability?)

• Then must spend all the time to be “reliable”

• Better approach: force 50% probability & have “buffer” for overruns

• Technical pride sometimes causes underestimates

16

Viable up-front estimates provide agile teams with fair terms

Agile Large Systems Back To Waterfall (Estimation & Planning Should Consider Hybrid)

• BACK TO WATERFALL or HYBRID

• UK’s Universal Credit Welfare System , “a complex IT project that involves switching off multiple benefits and reworking them into a new tax credit system” was, by most accounts, the most ambitious agile software development project in history. Suppliers include Accenture, Cap Gemini, HP, and IBM

• Quoting Computer Weekly “ DWP drops agile from flagship government software project”

• Cast Software research found that applications produced using traditional Agile or Waterfall methods alone have more security vulnerabilities, more reliability and performance issues, and a higher cost to maintain than those produced with a mixed method.


https://www.gov.uk/government/policies/simplifying-the-welfare-system-and-making-sure-work-pays/supporting-pages/introducing-universal-credit

http://www.computerweekly.com/blogs/public-sector/2013/05/dwp-drops-agile-from-flagship.html

Hybrid Agile Development (Source IPPS-A)

Ag

ile T

errit

ory

Develop

Detailed Design

UnitTesting

Build

PDR -> CDR -> PLT

Requirements Analysis

Fit/Gap

Integration Testing

Operational Testing

Plan Analyze Deploy Run

Waterline

SRR -> SFR -> IBR TRR -> DIT -> GAT -> LUT -> OPT -> DEP

• Hybrid Agile - Establishes a waterline below for Agile development

• From Planning and Requirements Analysis down to Agile Territory for the Build and then back up to Testing for final Deployment

#noestimates Viable For Detailed Development -Should Not Abdicate In Substantial Developments

Business CaseEvaluation of alternatives

Agile or Hybrid Agile

Software Development

System Test (when

appropriate)

Maintenance & Support


Agile development = root level software development management…

Story point estimating is short term productivity management

It is not a business decision making process

How Much? How long?Ownership

CostGo / no go

Hybrid Agile: Requirements

& Design

For substantial systems

Agile Developments Should Consider Risk and Probability

• For Estimation the Cone is the same – AND Loss Aversion may keep projects alive that should be killed

ROM Detailed Estimate


with Data& SEER

EstimationBias

Strategic Mis-

estimation

When Making Management Decisions Remember: Software Often Less Than 10% Total Ownership Cost

SoftwareDevelopment

SoftwareMaintenance

IT Infrastructure

IT Services


Development =Biggest Risk

Software Development is about 6-10% of total ownership cost…But much more of the risk

Assume $10m development could be over $100m total ownership

IT Services & Infrastructure Are Situational

but Generally 60% of TOC

Initial Backlog is Fraught with Uncertainties

Uncertainty Is Best Answered By Parametric Models

Known & Unknown Constraints

People SkillsProcess KnowledgeNew Technology

Customer AvailabilityMarket Uncertainty

Agile Can Only Help Mitigate This


SEER-SEM Predictive Analytics Provide Outside View: Agile Estimates & Risk Analysis

Using Agile Artifacts: Story Point and Epoch Sizing

Estimate Using Project History:

• Development

• Defects

• Total Ownership Cost

• Estimate Backlog

• Estimate Maintenance Effort

• Single or parallel Agile teams

Agile Risk Still Needs Consideration

Least, likely, and mostinputs provide a range of

cost and schedule outcomes

Confidence (probability)can be set and displayedfor any estimated item

• SEER predictsoutcomes

• SEER uses inputs to develop probability distributions

• The result is a probabilistic estimate

• SEER will predict a likely range of outcomes

• Monte Carlo provides project-level assessments of risk


Bia

s M

itig

ation:

Ris

k

SEER-SEM As Crosscheck to Supplier Quotes

• Example:

• 300 Functions 4GL

• Web, BI

• Team Size 3 FTE x 1 Month

• Reference is without Schedule Adjustment

• Current Estimate with Schedule Adjustment

Earned Value Management Process

INTEGRITY INNOVATION EXCELLENCE 26

How Galorath Applies Earned Value Management To Large Agile Systems

• As long as there is a plan and product, which can be measured EVM can be used

• Work planned and status rolled up at the EPIC Level (Often Control Account (CA))

• Features at the Work Package (WP) level breaks EPIC into functional packages

• Sprints statused by Stories and Story Points(statused in Agile Program Management Tool)

• Sprints are the process to the completing the Story(s) and Stories are Steps to completing the Feature, which rolls into the EPIC

• As Features are completed the percent complete is rolled up to the EPIC level

© 2017 Copyright Galorath Federal Incorporated 27

Agile Earned Value Management Building Blocks*

Sprints

Epic 1Feature 1

User Story 1

Epic 2Feature 2

User Story 2

Epic 2 Feature 3

User Story 3

Theme/Increment 1

Release 1 (made up of multiple Themes/Increments

Cost Estimating done at the Sprint Level

EVM work Packages

identified at Epic or

Theme level

* These “building blocks” are program specific and may be called by different names

Feature Point

values applied to each Feature

INTEGRITY INNOVATION EXCELLENCE 28

Back to this ProblemHow Much? vs. Show Me The Money

Product Owner

• Has a need and time to market expectations

• Can describe problem in large generalities

• May or may not be able to be proactive in development

• Needs to set budget for project approval

Development Staff

• Agile is perfect for everything mindset

• Tell us what you want

• Tell us what priority it is needed

• We’ll make a guess at how much it will cost

• Now show me the money


Key Points

Every forecast is subject to estimation

bias…a major cause of

program failure and corporate mis-spending

Agile is a good

thing.

Estimates are still

important

Experts Bias is probably costing

in cost, schedule, and

less than hoped for benefits

30

Bibliography (Partial)

• FLYVBJERG, BENT, Curbing Optimism Bias and Strategic Misrepresentation in Planning: Reference Class Forecasting in Practice, European Planning Studies Vol 16 No 1, Jan 2008,

• Kahneman, Daniel, and Amos Tversky. “Prospect Theory: An Analysis of Decision under Risk.” Econometrica 47, no. 2, March 1979.

• Johnson, H. Thomas. Relevance Regained. Free Press.

• Mislick, Gregory K.; Nussbaum, Daniel A.. Cost Estimation: Methods and Tools (Wiley Series in Operations Research and Management Science) (p. 143). Wiley.

• Rose, Todd. The End of Average: How We Succeed in a World That Values Sameness, HarperCollins.

• Taleb, Nassim Nicholas. Fooled by Randomness: The Hidden Role of Chance in Life and in the Markets (Incerto). Random House Publishing Group.

• Glen, Paul; McManus, Maria. The Geek Leader's Handbook: Essential Leadership Insight for People with Technical Backgrounds. Leading Geeks Press.

• Kogon, Kory; Merrill, Adam; Rinne, Leena. The 5 Choices: The Path to Extraordinary Productivity. Simon & Schuster.

• Patterson. Crucial Conversations Tools for Talking When Stakes Are High, Second Edition. McGraw-Hill.

• Kahneman, Daniel. Thinking, Fast and Slow. Farrar, Straus and Giroux.

• Hubbard, Douglas W.. The Failure of Risk Management: Why It's Broken and How to Fix It . Wiley.

• Hubbard, Douglas W.. How to Measure Anything: Finding the Value of Intangibles in Business . Wiley.


Backup Slides


IT Failure Can Impact Business Dramatically (Source: HBR)

• Case Study: Levi Strauss

• $5M ERP deployment contracted

• Risks seemed small

• Difficulty interfacing with customer’s systems

• Had to shut down production

• Unable to fill orders for 3 weeks

• $192.5M charge against earnings on a $5M IT project failure

“IT projects touch so many aspects of organization they pose a new singular risk”

http://hbr.org/2011/09/why-your-it-project-may-be-riskier-than-you-think/ar/1


Cognitive Bias: Bias Guesses and Forecasts (Source BeingHuman.org)

Cognitive bias

• Tendency to make systematic decisions based on PERCEPTIONS rather than evidence

• “Perception has more to do with our desires—with how we want to view ourselves—than with reality." Behavioral economist Dan Ariely

Researchers theorize in the past, biases helped survival

• Our brains using shortcuts (heuristics) that sometimes provide irrational conclusions

Bias affects everything:

• from deciding how to handle our money

• to relating to other people

• to how we form memories


Observations

• Agile is an excellent software development life cycle approach

• For substantial developments management decisions and commitments are still critical

• For some managers at least Agile keeps them from having to think about software (problem and sort of benefit)


Working software maturing

COUNTERPOINT: #noestimate: No Problem?

Agile can sometimes remove software worries from the C level

Every Decision is a Forecast (L. Maccherone, AgileCraft)

• You are forecasting that your choice will have better outcomes than other alternatives

• So quality of decisions depends on

1. Alternatives considered

2. Processes and models used to forecast the outcome of these alternatives…

• Probabilistic models are superior

• http://www.slideshare.net/lmaccherone/you-want-it-when-probabilistic-forecasting-and-decision-making


An Agile Approach to Planning (Source: Cohn)1. Projects rapidly generating useful new capabilities and

new knowledge

2. Flow of new capabilities & knowledge to guide work

3. Plan for what you want to learn – not what the product will be in the end

4. Traditional projects are like a 10K race – you know where the finish line is

• Get there as fast as possible

5. Agile projects are likea timed race

• See how far you can runin sixty minutes!

• Iterate until product owneris satisfied


Management Agile Manifesto #EstimatesSupportBusiness• We know you are technically capable… please help:

• Stop overpromising and under delivering

• Support management ROI, schedule needs and investment strategy

• Solve our problem

• Ensure it works, is usable, and secure

• Consider total cost to the business, not just initial costs

• Don’t waste too much resource

• “Base choices on those providing the maximum business value to the organization” Eli Goldratt


Give Management The Whole Picture: Total Ownership Costs Includes On-Going Support

Sprint or ReleaseBacklog

Iteration to Build

CodeTest

Refactor

Delivered Functionality

Working System

n-Iterations per Release Warranty & MaintenancePlanning

User Stories, Use Cases, Business

Requirements, Etc.

Fixes, Enhancements, Sustainment

Defects & Unfinished Work

Collection of Functionality

Many programs need total ownership cost evaluation... Estimation is essential

Development + Maintenance

Identify Total Ownership Costs for the Software

Allows Independent Maintenance Team

Assumptions

Estimate Cost of: Corrective, Adaptive,

Perfective, and Enhancement support

The Planning Fallacy (Kahneman & Tversky, 1979)

•Manifesting bias rather than confusion

•Judgment errors made by experts and laypeople alike

•Errors continue when estimators aware of their nature

Judgment errors are systematic & predictable,

not random

•Underestimate costs, schedule, risks

•Overestimate benefits of the same actionsOptimistic due to overconfidence

ignoring uncertainty

•“inside view” focusing on components rather than outcomes of similar completed actions

•FACT: Typically past more similar assumed

•even ventures may appear entirely different

Root cause: Each new venture

viewed as unique


Bias Mitigation Reference Class Forecasting

Attempt to force the outside view and

eliminate optimism and

misrepresentation

Choose relevant “reference class”

completed analogous projects

Compute probability distribution

Compare range of new projects to

completed projects

Provide an “outside view” focus on

outcomes of analogous projects


Best predictor of performance is actual performanceof implemented comparable projects (Nobel Prize Economics 2002)

Predicts outcome of planned action basedon actual outcomes in a reference class:similar actions to those being forecast.

Documents

Estimation Bias and Mitigation With Agile Estimate ... · Harvard Business Review explains this Nobel Prize Winning Phenomenon: • Humans seem hardwired to be optimists • Routinely