Primary funding is provided by

The SPE Foundation through member donations and a contribution from Offshore Europe

The Society is grateful to those companies that allow their professionals to serve as lecturers

Additional support provided by AIME

Society of Petroleum Engineers Distinguished Lecturer Programwww.spe.org/dl

Society of Petroleum Engineers Distinguished Lecturer Programwww.spe.org/dl 2

Duane A. McVayTexas A&M University

The Value of Assessing Uncertainty(What You Don’t Know Can Hurt You)

Overview

• You don’t know as much as you think you know

• Consequences of underestimating uncertainty are substantial

• Industry performs below expectations• Look-backs can improve assessments

and financial performance

3

E&P Has Underperformed Due toPoor Uncertainty Assessment

4

Author Industry Performance Reasons Cited

Capen (1976)

Low industry returns Underestimation of uncertainty

Brashear et al. (2001)

In 1990s, large U.S. E&P returned less than cost of capital

Evaluations did not account for “full” uncertainty

Merrow (2011)

Success of large E&P projects declined from 50% to 22% since 2003

Overconfident and optimistic project scheduling

Why Does Underestimation of Uncertainty Persist?

• We don’t fully understand uncertainty• We don’t appreciate the value of

assessing uncertainty• We don’t know how to reliably assess

uncertainty

5

Humans Are Poor at Assessing Uncertainty

• Almost universal tendency for overconfidence– Tversky and Kahneman (1974), “Judgment under

Uncertainty: Heuristics and Biases”– Capen (1976, SPE 5579), “The Difficulty of Assessing

Uncertainty”

• General tendency for optimism – Weinstein (1980), “Unrealistic Optimism About Future

Life Events”– Merrow (2011, OTC 21858), “Oil Industry

Megaprojects: Our Recent Track Record”

6

Result: Decisions Made With Wrong Distributions

7

true

estimated

k, Φ, qi, reserves, price

Experts Are No Better at Assessing Uncertainty

California Energy Commission Delphi Oil Price Forecasts

Polled international panel of financial, academic, industry, consulting, and government experts

8

$0

$20

$40

$60

$80

$100

$120

1970 1980 1990 2000 2010 2020 2030

Oil

Pric

e (1

997

$/bb

l)

Year

Pessimistic

Optimistic

Actual

9

Also Overconfident, Optimistic in Production Forecasting

10

Expectation

81% Mean

80% 100% 120%40% 60%

Percent Attainment of Planned Production

Modified from SPE 145437 (IPA, 2011)

147 Projects, 1992-20082nd 6-months production

Underestimating Uncertainty Has Economic Consequences

• Poor decisions– Incorrect selections– Incorrect rejections

• Reduced value– McVay and Dossary (2014, SPE 160189)– Ran portfolio selection experiments – Determined quantitatively the value of

assessing uncertainty 11

Modeling Overconfidence

12

Modeling Directional Bias (Pessimism or Optimism)

OC = 0 0.5 0.9

trueestimated

DB = -1 -0.5 0.50 1

All with confidence of 0.5trueestimated

0.1 1

True and Estimated Project Distributions

13

• Confidence = 0.5 and Directional Bias = 0.5• PVOCF = Present Value of Operating Cash Flow (NPV+CapEx)• CapEx = Capital Expenditures

Assessment of Portfolio PerformanceWith Overconfidence and Optimism

Estimated NPV

14

EstimationError(+)

Decision Error(+)

Disappointment(+)

Best Possible NPV

Actual NPV

Select: EstimatedValue: Estimated

Select: TrueValue: True

Select: EstimatedValue: True

Distributions used to select projects and value portfolio

From SPE 116525 (2008)

Confidence

Expected Disappointment is Large for Moderate Overconfidence and Optimism

15Unconstrained Budget Case

Complete Overconfidence

No Overconfidence

Complete PessimismComplete Optimism

Est NPV – Actual NPVEst NPV

Maximize Value Only When No Overconfidence and No Directional Bias

16Unconstrained Budget Case

Actual NPVBest Possible NPV

Pleasant Surprise Comes at the Cost of Large Decision Error

Is Actual Industry Performance Consistent With Simulations?

• Brashear et al. 2001 (SPE 73141)– Large U.S. E&P in the 1990s– Returned 7% with 9-12% capital costs– Disappointment > 150%

• Nandurdikar 2014 (October JPT)– Average E&P development over last 15 years– Delivered only 60% of value promised at sanction– Disappointment = 40%

17

Low oil prices

High oil prices

What Should We Do?

• Eliminate overconfidence (and optimism)

• Have to measure them first• Conduct look-backs and calibration• Account for “unknown unknowns”

18

Calibration is Key to ReliableProbabilistic Estimation

Calibration Score = measure of distance from unit-slope line

19

Underconfidence

Overconfidence

Actual

We want perfect calibration

Benefits of Calibration

• Awareness of problem

20

Can Measure Overconfidence and Directional Bias

21

Confidence = 1 – m = 0.5DB = 1CompleteOptimism

DB = -1CompletePessimism

DB = 0Neutral

All Confidence = 0.5

Calibration of IPA Analysis Shows High Overconfidence and Optimism

22

OC ≈ 0.72DB ≈ 0.92

Modified from SPE 145437 (2011)

Benefits of Calibration

• Awareness of problem• Improve forecasts with internal

correction

23

Calibration and Feedback Improve Internal Assessments

Student predictions of football game scores (P10, P50, P90 for Texas A&M – opponent)

24

Time Period Calibration Score

1st half of season 0.039

2nd half of season 0.015

2012 Season

Benefits of Calibration

• Awareness of problem• Improve forecasts with internal

correction• Improve forecasts with external

correction

25

Application of ExternalForecast Correction

• Problem: production forecasts are long-term• Solution: calibrate short-term forecasts to adjust long-term• 187 Barnett shale wells, median 82 months production

26Time: 3 yrs

External Adjustment Improves Forecast Reliability

• Externally adjust long-term forecasts using short-term calibration results

27

Time: 6-7 yrs

10% 200 44% 200 10% 9850% 250 56% 250 50% 22390% 350 72% 350 90% 623

AdjustedFittedOriginal

Production Values in MMscf

Method proposed by Capen 1976

Recommendations: Train Individually

• Train to become well calibrated• Formal or self training• Capen, 1976, “The Difficulty of

Assessing Uncertainty”• Hubbard, 2014, How to Measure

Anything28

Forecast Probabilistically

• Do not have to use complex, expensive methods

• Calibration, not complexity, is key• Simple methods with calibrated

uncertainties better than complex methods with biased uncertainties

29

Record Frequently

• Record as many forecasts as possible

• Record forecast when make it• Start tracking forecasts now to

reap benefits later

30

Track Centrally

31Fondren et al., 2013 (SPE 166422)

Calibrate Periodically

• Look back and calibrate at predetermined frequency

• Do not calibrate on an individual project basis

32

Incentivize Properly• Incentivized to be overconfident and

optimistic• No accountability for unreliable forecasts• Recipe for chronic disappointment• To change behavior, change incentives• Track and calibrate throughout company• Make factor in employee compensation

33

Summary

• Industry is overconfident and optimistic• Poor decisions; large disappointment• Best decisions made when uncertainty is

reliably quantified• Reliable uncertainty assessment can add

value to the bottom line• Calibration key to reliable uncertainties• If don’t change, problems will continue

34

Acknowledgements• Texas A&M students involved in uncertainty research

• Mubarak Dossary, Mark Fondren, Raul Gonzalez, Xinglai Gong, Alejandro Valdes, Houda Hdadou, Carlos Morales, Hisham Almohammadi, Zhenzhen Dong, Yao Tian, Chang Liu, Yueming Cheng, Chih-Ming Tien, Chile Ogele, Ahmed Daoud, Jesus Salazar, Gulcan Turkarslan, Rubiel Ortiz, Sara Martin, Kun Cheng, Wenyan Wu, Yuhong Wang, Jay Holmes, Ashish Mendjoge, Grant Olsen, Festus Fariyibi, Nasir Akilu, Martin Alvarado, Ozgur Senel

• Other collaborators in uncertainty research• John Lee, Walter Ayers, Steve Holditch, Sam Noynaert, Eric

Bickel, Rick Gibson, Jeffrey Hart, Stephen Pickering

35

Society of Petroleum Engineers Distinguished Lecturer Programwww.spe.org/dl 36

Your Feedback is ImportantEnter your section in the DL Evaluation Contest by

completing the evaluation form for this presentation

Visit spe.org/dl/contest.phpClick on “Section Attendees’ Tour Evaluation form”