2. Shallow versus deep uncertainties Why many predictions / forecasts fail

2. Shallow versus deep uncertaintiesWhy many predictions / forecasts fail

QuickTime™ and a decompressor

are needed to see this picture. QuickTime™ and a decompressor

are needed to see this picture.

Society is playing a high-stakes game of chance against

nature

We want to

- assess the hazard - how often dangerous events happen - mitigate or reduce the risk - the resulting losses.

Often nature surprises us, when an earthquake, hurricane, or flood is bigger or has greater effects than expected from hazard assessments.

In other cases, nature outsmarts us, doing great damage despite expensive mitigation measures, or causing us to waste resources on what proves a minor hazard.

Hazard assessment failed

2010 map predicts probability of strong shaking in next 30 years

But: 2011 M 9.1 Tohoku, 1995 Kobe M 7.3 & others in areas mapped as low hazard

In contrast: map assumed high hazard in Tokai “gap”

Geller 2011

Hazard model divided trench into segments

Expected Earthquake Sources 50 to 150 km segments M7.5 to 8.2(Headquarters for Earthquake Research Promotion)

Giant earthquake broke many segments

2011 Tohoku Earthquake 450 km long fault, M 9.1 (Aftershock map from USGS)

J. Mori

Expected Earthquake Sources 50 to 150 km segments M7.5 to 8.2(Headquarters for Earthquake Research Promotion)

Tsunami runup approximately twice

fault slip (Plafker, Okal &

Synolakis 2004)

M9 generates much larger tsunami

Planning assumed maximum magnitude 8 Seawalls 5-10 m high

CNN

NYTStein & Okal, 2011

NY Times 3/31/2011

Mitigation failed

Expensive seawalls - longer than Great Wall of China -proved ineffective

Tsunami overtopped 10m high sea walls, causing more than 15,000 deaths and $210 billion damage.

What’s going wrong?

Shallow uncertainty - we don’t know what will happen, but know the odds (probability

density function). The past is a good predictor of the future. We can make math

models that work well.

Deep uncertainty - we don’t know the odds. The past is a poor predictor of the future. We can make math models, but

they generally won’t work well.

Shallow uncertainty is like estimating the chance that a batter will get a hit. His batting average is a good predictor.

Deep uncertainty is like trying to predict the winner of the World Series five years from now. Teams' past performance give only limited insight into the future.

Due to deep uncertainty

Predicted natural or other disaster probabilities are hard to estimate and thus often very inaccurate

The world is more complicated than we think or admit

Prob(sinking) = 0

1986 – Loss of shuttle Challenger

NASA claimed probability of loss = 1/100,000

Richard Feynman argued for 1/100 – 1000 times higher.

As he pointed out in his report dissenting from the government investigation commission, because this rate implies that "one could put up a shuttle every day for 300 years expecting to lose only one, we could properly ask what is the cause of management's fantastic faith in the machinery... "

1986 – Loss of shuttle Challenger

NASA claimed probability of loss = 1/100,000

Richard Feynman argued for 1/100 – 1000 times higher.

As he pointed out in his report dissenting from the government investigation commission, because this rate implies that "one could put up a shuttle every day for 300 years expecting to lose only one, we could properly ask what is the cause of management's fantastic faith in the machinery... "

In 2003, shuttle Columbia was lost on the 107th shuttle mission

2 lost in 107 missions ≈ 1/50

Activity 2.1:

Given that the shuttle was a new spacecraft, before flights started, how could you assess NASA’s estimated probability of loss = 1/100,000?

Activity 2.1:

Given that the shuttle was a new spacecraft, before flights started, how could you assess NASA’s estimated probability of loss = 1/100,000?

One way: of 11 Apollo missions, one (Apollo 13) suffered near disaster. In addition, Apollo 1 was lost in a launch pad fire.

QuickTime™ and a decompressor

are needed to see this picture.

QuickTime™ and a decompressor

are needed to see this picture.

Boeing 787 Dreamliner batteries

Boeing “concluded that they were likely to emit smoke less than once in every 10 million flight hours.

Once the planes were placed in service, the batteries overheated and emitted smoke twice, and caused one fire, after about 50,000 hours of commercial flights.” (NYT, 2/7/13)

From 1975 to 2007, U.S. housing prices grew steadily. Neither Washington nor Wall Street recognized that this could not go on forever, or worried that trillions of dollars of risky

mortgages were embedded throughout the financial system

QuickTime™ and a decompressor

are needed to see this picture.

As housing prices and subprime lenders collapsed, Wall Street & government weren’t concerned based

on computer models

The models were wrong

QuickTime™ and a decompressor

are needed to see this picture.

NY Times 3/21/11

Hazard maps are hard to get right: successfully predicting future shaking depends on accuracy of four

assumptions over 500-2500 years

Where will large earthquakes occur?

When will they occur?

How large will they be?

How strong will their shaking be?

Uncertainty & map failure result because these are often hard to assess, given that the earthquakes are much more variable in space and time than the short

earthquake history shows

2008 Wenchuan earthquake (Mw 7.9) was not expected: map showed low hazard based on lack of recent

earthquakes

Didn’t use GPS data showing 1-2 mm/yr (~Wasatch)

Earthquakes prior to the 2008 Wenchuan event

Aftershocks of the Wenchuan event delineating the rupture zone

QuickTime™ and a decompressor

are needed to see this picture.

GSHAP 1999

NUVEL-1Argus, Gordon, DeMets & Stein, 1989

Swafford & Stein, 2007

Slow plate boundary

Africa-Eurasia convergence rate varies smoothly (5 mm/yr)

2004

2003

Swafford & Stein, 2007

GSHAP 1999

NUVEL-1Argus, Gordon, DeMets & Stein, 1989

M 6.4

M 6.3

Slow plate boundary

Africa-Eurasia convergence rate varies smoothly (5 mm/yr)

Familiar pattern

2001 hazard map

http://www.oas.org/cdmp/document/seismap/haiti_dr.htm

2010 M7 earthquake shaking much greater than predicted for next 500 years

Italian hazard maps, which predicted the expected shaking in the next 500 years, forecast some earthquake locations well and others poorly, and so required updating within a decade.

A posteriori changes to a model are "Texas sharpshooting:” shoot

at the barn and then draw

circles around the bullet holes.

Overfitting: a subtle trap

We can fit very complicated models to data like earthquake histories, but we are partly fitting noise

Flipping a coin gives lots of complicated patterns

H H T T H T T T H T H H T H

We could fit a model to those data, but it would do no better than 50% at predicting the next flip

Overfitting: a subtle trap

We can fit very complicated models to data like earthquake histories, but we are partly fitting noise

In such cases, a more complicated model can give worse predictions

QuickTime™ and a decompressor

are needed to see this picture.

Activity 2.2: Time between large earthquakes from paleoseismic record on

southern San Andreas

What’s the mean time between large earthquakes here? When would you expect the next one?

Is one due soon, overdue, or…

DD 9.8

Time dependent predicts lower until ~2/3 mean recurrence

Results depend on both model choice & assumed mean recurrence

Hebden & Stein, 2008

We don’t know whether to assume that probability of a major earthquake is

- constant with time (time-independent) or

- small after a large earthquake and then increases (time-dependent ).

Activity 2.3 Deep uncertainty in earthquake recurrence

Imagine an urn containing e balls labeled "E" for earthquake, and n balls labeled "N" for no earthquake.

We can draw balls in two ways.

Activity 2.3 Deep uncertainty in earthquake recurrence

Option 1: after drawing a ball, we replace it. In successive draws, the probability of an event is constant or time-independent.

Because one event happening does not change the probability of another happening, an event is never overdue.

Activity 2.3 Deep uncertainty in earthquake recurrence

Option 2: We can add a number a of E-balls after a draw when an event does not occur, and remove r E-balls when an event occurs. This makes the probability of an event increase with time until one happens, after which it decreases and then grows again. Events are not independent, because one happening changes the probability of another.

Activity 2.3 Deep uncertainty in earthquake recurrence

Problem: Given a sequence of results, it’s hard or impossible to tell how the urn was sampled. Thus it’s hard to assess the probability of an “earthquake” in the next draw.

Italian flag graphic - one way to illustrate uncertainty we can’t

quantify well

50% 30% 20%