34
Past performance is no guide to future returns: Why we can’t accurately forecast the future Jonathan Koomey, Ph.D. Research Fellow, Stanford University http://www.koomey.com Presented on a webinar for US EPA and US DOE May 18, 2016 1 Copyright Jonathan G. Koomey 2016

Past performance is no guide to future returns: Why we can't accurately forecast the future

Embed Size (px)

Citation preview

Page 1: Past performance is no guide to future returns:  Why we can't accurately forecast the future

Past performance is no guide to future returns: Why we can’t accurately

forecast the future Jonathan Koomey, Ph.D.

Research Fellow, Stanford University http://www.koomey.com

Presented on a webinar for US EPA and US DOE May 18, 2016

1  Copyright  Jonathan  G.  Koomey  2016  

Page 2: Past performance is no guide to future returns:  Why we can't accurately forecast the future

My background

•  Founded LBNL’s End-Use Forecasting group and led that group for more than 11 years.

•  Peer reviewed articles and books on – Forecasting methodology – Economics of greenhouse gas mitigation – Critical thinking skills –  Information technology and resource use

2  Copyright  Jonathan  G.  Koomey  2016  

Page 3: Past performance is no guide to future returns:  Why we can't accurately forecast the future

3  

Cost-benefit analysis: the standard approach

Copyright  Jonathan  G.  Koomey  2016  

Page 4: Past performance is no guide to future returns:  Why we can't accurately forecast the future

True or False?: If only we had enough…

•  Time •  Money •  Graduate Students •  Coffee

we could accurately predict the cost of energy technologies in

2050 4  Copyright  Jonathan  G.  Koomey  2016  

Page 5: Past performance is no guide to future returns:  Why we can't accurately forecast the future

Widespread modeling practice implies that the answer is “True”

5  Copyright  Jonathan  G.  Koomey  2016  

Page 6: Past performance is no guide to future returns:  Why we can't accurately forecast the future

Based on my experience and reviews of historical

retrospectives on forecasting, I say “No way”

6  Copyright  Jonathan  G.  Koomey  2016  

Page 7: Past performance is no guide to future returns:  Why we can't accurately forecast the future

Aside: Many of the best modelers acknowledge the difficulties in the pursuit of accurate forecasts, but in their heart of hearts they still

believe they can predict accurately with greater effort

7  Copyright  Jonathan  G.  Koomey  2016  

Page 8: Past performance is no guide to future returns:  Why we can't accurately forecast the future

Uncertainty affects even physical systems

Es=mates  of  Planck’s  constant  "h"  over  =me.  In  this  physical  system  researchers  repeatedly  underes=mated  the  error  in  their  determina=ons.  At  each  stage  uncertain=es  existed  of  which  the  researchers  were  unaware.    The  problem  of  error  es=ma=on  is  far  greater  in  long-­‐range  energy  forecas=ng.      Taken  from  Koomey  et  al.  2003.  

8  Copyright  Jonathan  G.  Koomey  2016  

Page 9: Past performance is no guide to future returns:  Why we can't accurately forecast the future

Forecasting Accuracy: The Models Have Done Badly

•  Energy forecasting models have little or no ability to accurately predict future energy prices and demand (Craig et al. 2002)

•  Even the sign of the impacts of proposed policies is a function of key assumptions (Repetto and Austin 1997)

•  The dismal accuracy and inherent limitations of these models should make modelers modest in the conclusions they draw (Decanio 2003) Craig, P., A. Gadgil, and J. Koomey (2002). “What Can History Teach Us? A Retrospective Analysis of Long-term Energy Forecasts for the U.S.” Annual Review of Energy and the Environment 2002. R. H. Socolow, D. Anderson and J. Harte. Palo Alto, CA, Annual Reviews, Inc. (also LBNL-50498). 27: 83-118.

Repetto, R. and D. Austin (1997). The Costs of Climate Protection: A Guide for the Perplexed. Washington, DC, World Resources Institute. DeCanio, S. J. (2003). Economic Models of Climate Change: A Critique. Basingstoke, UK, Palgrave-Macmillan.

9  Copyright  Jonathan  G.  Koomey  2016  

Page 10: Past performance is no guide to future returns:  Why we can't accurately forecast the future

One example: 1970s projections of year 2000 U.S. primary energy

Source: Craig, Paul, Ashok Gadgil, and Jonathan Koomey. 2002. "What Can History Teach Us?: A Retrospective Analysis of Long-term Energy Forecasts for the U.S." In Annual Review of Energy and the Environment 2002. Edited by R. H. Socolow, D. Anderson and J. Harte. Palo Alto, CA: Annual Reviews, Inc. (also LBNL-50498). pp. 83-118.

10  Copyright  Jonathan  G.  Koomey  2016  

Page 11: Past performance is no guide to future returns:  Why we can't accurately forecast the future

What drove errors in US primary energy forecasts?

Copyright  Jonathan  G.  Koomey  2016   11  

Graph  from  Hirsh  and  Koomey  2015  

Page 12: Past performance is no guide to future returns:  Why we can't accurately forecast the future

Another  example:  Oil  

price  projec3ons  by  U.S.  DOE,  AEO  1982  

through  AEO  2000  

12  Copyright  Jonathan  G.  Koomey  2016  

Page 13: Past performance is no guide to future returns:  Why we can't accurately forecast the future

Not  any  beEer  aFer  2000:  Oil  price  

projec3ons  by  U.S.  

DOE,  AEO  2000  

through  AEO  2007  

13  Copyright  Jonathan  G.  Koomey  2016  

Page 14: Past performance is no guide to future returns:  Why we can't accurately forecast the future

Yet another example: NERC fan

Copyright  Jonathan  G.  Koomey  2016   14  

US  electricity  genera=on  BkWh/year  

Page 15: Past performance is no guide to future returns:  Why we can't accurately forecast the future

Why Are Long-term Energy Forecasts Almost Always Wrong?

•  Core data and assumptions, which drive results, are based on historical experience, which can be misleading if structural conditions change

•  The exact timing and character of pivotal events and technology changes cannot be predicted

Laitner, J.A., S.J. DeCanio, J.G. Koomey, A.H. Sanstad. (2003) “Room for Improvement: Increasing the Value of Energy Modeling for Policy Analysis.” Utilities Policy, 11, pp. 87-94.

15  Copyright  Jonathan  G.  Koomey  2016  

Page 16: Past performance is no guide to future returns:  Why we can't accurately forecast the future

Conditions for Model Accuracy •  Hodges and Dewar: models can be

accurate when they describe systems that – are observable and permit collection of

ample and accurate data – exhibit constancy of structure over time – exhibit constancy across variations in

conditions not specified in the model

Source: Hodges, James S., and James A. Dewar. 1992. Is it you or your model talking? A framework for model validation. Santa Monica, CA: RAND. ISBN 0-8330-1223-1.

16  Copyright  Jonathan  G.  Koomey  2016  

Page 17: Past performance is no guide to future returns:  Why we can't accurately forecast the future

∑: Accurate forecasts require structural constancy and no

surprises

17  Copyright  Jonathan  G.  Koomey  2016  

Page 18: Past performance is no guide to future returns:  Why we can't accurately forecast the future

Market structure can change fast

Source:    Scher  and  Koomey  2010.  

18  Copyright  Jonathan  G.  Koomey  2016  

Page 19: Past performance is no guide to future returns:  Why we can't accurately forecast the future

Fast changing markets #2: US electricity consumption

Copyright  Jonathan  G.  Koomey  2016   19  

Graph  from  Hirsh  and  Koomey  2015  

Page 20: Past performance is no guide to future returns:  Why we can't accurately forecast the future

Surprises can be big: U.S. nuclear busbar costs

Source: Koomey and Hultman 2007. Assumes 7% real discount rate.

Projected cost range from Tybout 1957

20  Copyright  Jonathan  G.  Koomey  2016  

Page 21: Past performance is no guide to future returns:  Why we can't accurately forecast the future

Implications for long-term energy forecasting

•  Forecasting models describing well-defined physical systems using correct parameters can be accurate because physical laws are geographically and temporally invariant (as long as there are no surprises)

•  Economic, social, and technological systems do not exhibit the required structural constancy, so models forecasting the future of these systems are doomed to be inaccurate. Four big sources of inconstancy –  Pivotal events (like Sept. 11th or the 1970s oil shocks) –  Technological innovation –  Institutional change –  Policy choices

21  Copyright  Jonathan  G.  Koomey  2016  

Page 22: Past performance is no guide to future returns:  Why we can't accurately forecast the future

∑: Economics ≠ Physics

22  Copyright  Jonathan  G.  Koomey  2016  

Page 23: Past performance is no guide to future returns:  Why we can't accurately forecast the future

So no matter how many $, coffee cups, months, or graduate

students you have, accurate long-run forecasting of technology

costs is impossible

23  Copyright  Jonathan  G.  Koomey  2016  

Page 24: Past performance is no guide to future returns:  Why we can't accurately forecast the future

Two senses of the word “impossible”:

Practically and

Theoretically

Either way, the net result is the same: inaccurate forecasts

24  Copyright  Jonathan  G.  Koomey  2016  

Page 25: Past performance is no guide to future returns:  Why we can't accurately forecast the future

So what does this result imply for predictions of the costs of

energy technologies?

25  Copyright  Jonathan  G.  Koomey  2016  

Page 26: Past performance is no guide to future returns:  Why we can't accurately forecast the future

Some lessons •  The world is evolutionary and path dependent

–  Increasing returns, transaction costs, information asymmetries, bounded rationality, prospect theory

–  Our actions now affect our options later (so do surprises!)

•  Experimentation is the order of the day •  Use real data to prove results

–  For nuclear power, we’re in the “show me” stage. Cost projections are no longer enough

•  Prefer technologies that –  are mass produced vs. site-built –  have short lead times vs. longer lead times

26  Copyright  Jonathan  G.  Koomey  2016  

Page 27: Past performance is no guide to future returns:  Why we can't accurately forecast the future

Nuke costs: here we go again?

Source: Koomey and Hultman 2007. 27  Copyright  Jonathan  G.  Koomey  2016  

Page 28: Past performance is no guide to future returns:  Why we can't accurately forecast the future

“No battle plan survives contact with the enemy.” –Helmuth von Moltke the elder

Copyright  Jonathan  G.  Koomey  2016   28  

Page 29: Past performance is no guide to future returns:  Why we can't accurately forecast the future

More lessons •  Use physical and technological constraints to

define bounding cases. Examples: –  2 degrees Celsius warming limit implies a carbon

budget, which implies a certain rate of implementation of non-fossil energy sources to avoid worst effects of climate change.

– Certain technologies use materials that are in limited supply. Working backwards from a goal can help identify resource constraints.

–  Lifetime of power generation technologies and buildings limits penetration of new technologies unless we scrap existing capital

29  Copyright  Jonathan  G.  Koomey  2016  

Page 30: Past performance is no guide to future returns:  Why we can't accurately forecast the future

Reconsidering benefit-cost analysis for climate

•  "A corollary is that it is fruitless to attempt to determine the "optimal" carbon tax. If neither the costs nor the benefits can be known with any precision, just about the only thing that can be said with certainty about the welfare maximizing price of carbon emissions is that it is greater than zero. Economists have a great deal to say about how to implement such a tax efficiently and effectively, about the similarities and differences between a tax and a system of tradable carbon emissions permits, about about the best way to recycle the revenue from such a tax or permit system. And, as we have seen above, the distributional consequences of such a tax or permit auction plan will affect other economic variables through system-wide feedbacks. However, any attempt to specify the exact level of the "optimal" tax is less an exercise in scientific calculation than a manifestation of the analyst’s willingness to step beyond the limits of established economic knowledge."

•  –DeCanio, Stephen J. 2003. Economic Models of Climate Change: A Critique. Basingstoke, UK: Palgrave-Macmillan. p.157.

Copyright  Jonathan  G.  Koomey  2016   30  

Page 31: Past performance is no guide to future returns:  Why we can't accurately forecast the future

Conclusions •  It is impossible to accurately forecast energy

technology characteristics because of –  structural inconstancy and –  pivotal events

•  Forecasting community has yet to absorb the implications of this insight

•  To cope we need new ways to think about the future –  Experimental approach to implementation (try many things,

fail fast, learn quickly, try again) –  Rely on physical and technological constraints to create

bounding cases –  Embrace path dependence (there is no optimal solution,

just lots of possible pathways of roughly similar costs)

31  Copyright  Jonathan  G.  Koomey  2016  

Page 32: Past performance is no guide to future returns:  Why we can't accurately forecast the future

“The best way to predict the future is to invent it.” –Alan Kay

Copyright  Jonathan  G.  Koomey  2016   32  

Page 33: Past performance is no guide to future returns:  Why we can't accurately forecast the future

Some Key References •  Craig, Paul, Ashok Gadgil, and Jonathan Koomey. 2002. "What Can History Teach Us?: A

Retrospective Analysis of Long-term Energy Forecasts for the U.S." In Annual Review of Energy and the Environment 2002. Edited by R. H. Socolow, D. Anderson and J. Harte. Palo Alto, CA: Annual Reviews, Inc. pp. 83-118.

•  Ghanadan, Rebecca, and Jonathan Koomey. 2005. "Using Energy Scenarios to Explore Alternative Energy Pathways in California." Energy Policy. vol. 33, no. 9. June. pp. 1117-1142.

•  Hirsh, Richard F., and Jonathan G. Koomey. 2015. "Electricity Consumption and Economic Growth: A New Relationship with Significant Consequences?" The Electricity Journal. vol. 28, no. 9. November. pp. 72-84. [http://www.sciencedirect.com/science/article/pii/S1040619015002067]

•  Koomey, Jonathan. 2008. Turning Numbers into Knowledge: Mastering the Art of Problem Solving. Oakland, CA: Analytics Press. 2nd edition. <http://www.analyticspress.com>

•  Koomey, Jonathan. 2002. "From My Perspective: Avoiding "The Big Mistake" in Forecasting Technology Adoption." Technological Forecasting and Social Change. vol. 69, no. 5. June. pp. 511-518.

•  Koomey, Jonathan G., Paul Craig, Ashok Gadgil, and David Lorenzetti. 2003. "Improving long-range energy modeling: A plea for historical retrospectives." The Energy Journal (also LBNL-52448). vol. 24, no. 4. October. pp. 75-92.

•  Chapter 4: “Why we can’t accurately forecast the future”, in Koomey, Jonathan G. 2012. Cold Cash, Cool Climate: Science-Based Advice for Ecological Entrepreneurs. Burlingame, CA: Analytics Press. [http://www.analyticspress.com/cccc.html]

•  Koomey, Jonathan. 2013. "Moving Beyond Benefit-Cost Analysis of Climate Change." Environmental Research Letters. vol. 8, no. 041005. December 2. [http://iopscience.iop.org/1748-9326/8/4/041005/]

•  Laitner, J.A., S.J. DeCanio, J.G. Koomey, A.H. Sanstad. (2003) “Room for Improvement: Increasing the Value of Energy Modeling for Policy Analysis.” Utilities Policy, vol. 11, no. 2. June. pp. 87-94.

•  Scher, Irene, and Jonathan G. Koomey. 2011. "Is Accurate Forecasting of Economic Systems Possible?" Climatic Change. Vol 104, No. 3-4, pp.473-479.

33  Copyright  Jonathan  G.  Koomey  2016  

Page 34: Past performance is no guide to future returns:  Why we can't accurately forecast the future

More Key References •  Armstrong, J. Scott, ed. 2001. Principles of Forecasting: A Handbook for Researchers and

Practitioners. Norwell, MA: Kluwer Academic Publishers. •  Ascher, William. 1978. Forecasting: An Appraisal for Policy Makers and Planners. Baltimore,

MD: Johns Hopkins University Press. •  Cohn, Steve. 1991. "Paradigm Debates in Nuclear Cost Forecasting." Technological

Forecasting and Social Change. vol. 40, no. 2. September. pp. 103-130. •  Grubler, Arnulf, Nebojsa Nakicenovic, and David G. Victor. 1999. "Dynamics of energy

technologies and global change." Energy Policy. vol. 27, no. 5. May. pp. 247-280. •  Hodges, James S., and James A. Dewar. 1992. Is it you or your model talking? A framework for

model validation. Santa Monica, CA: RAND. ISBN 0-8330-1223-1. •  Huntington, Hillard G. 1994. "Oil Price Forecasting in the 1980s: What Went Wrong?" The

Energy Journal. vol. 15, no. 2. pp. 1-22. •  Huss, William R. 1985. "Can Electric Utilities Improve Their Forecast Accuracy? The Historical

Perspective." In Public Utilities Fortnightly. December 26, 1985. pp. 3-8. •  Landsberg, Hans H. 1985. "Energy in Transition: A View from 1960." The Energy Journal. vol. 6,

pp. 1-18. •  O'Neill, Brian C., and Mausami Desai. 2005. "Accuracy of past projections of U.S. energy

consumption." Energy Policy. vol. 33, no. 8. May. pp. 979-993. •  Tetlock, Philip E. 2005. Expert Political Judgment: How Good Is It? How Can We Know?

Princeton, NJ: Princeton University Press.•  Tybout, Richard A. 1957. "The Economics of Nuclear Power." American Economic Review. vol.

47, no. 2. May. pp. 351-360.

34  Copyright  Jonathan  G.  Koomey  2016