2. Shallow versus deep uncertaintiesWhy many predictions / forecasts fail
QuickTime™ and a decompressor
are needed to see this picture. QuickTime™ and a decompressor
are needed to see this picture.
Society is playing a high-stakes game of chance against
nature
We want to
- assess the hazard - how often dangerous events happen - mitigate or reduce the risk - the resulting losses.
Often nature surprises us, when an earthquake, hurricane, or flood is bigger or has greater effects than expected from hazard assessments.
In other cases, nature outsmarts us, doing great damage despite expensive mitigation measures, or causing us to waste resources on what proves a minor hazard.
Hazard assessment failed
2010 map predicts probability of strong shaking in next 30 years
But: 2011 M 9.1 Tohoku, 1995 Kobe M 7.3 & others in areas mapped as low hazard
In contrast: map assumed high hazard in Tokai “gap”
Geller 2011
Hazard model divided trench into segments
Expected Earthquake Sources 50 to 150 km segments M7.5 to 8.2(Headquarters for Earthquake Research Promotion)
Giant earthquake broke many segments
2011 Tohoku Earthquake 450 km long fault, M 9.1 (Aftershock map from USGS)
J. Mori
Expected Earthquake Sources 50 to 150 km segments M7.5 to 8.2(Headquarters for Earthquake Research Promotion)
Tsunami runup approximately twice
fault slip (Plafker, Okal &
Synolakis 2004)
M9 generates much larger tsunami
Planning assumed maximum magnitude 8 Seawalls 5-10 m high
CNN
NYTStein & Okal, 2011
NY Times 3/31/2011
Mitigation failed
Expensive seawalls - longer than Great Wall of China -proved ineffective
Tsunami overtopped 10m high sea walls, causing more than 15,000 deaths and $210 billion damage.
What’s going wrong?
Shallow uncertainty - we don’t know what will happen, but know the odds (probability
density function). The past is a good predictor of the future. We can make math
models that work well.
Deep uncertainty - we don’t know the odds. The past is a poor predictor of the future. We can make math models, but
they generally won’t work well.
Shallow uncertainty is like estimating the chance that a batter will get a hit. His batting average is a good predictor.
Deep uncertainty is like trying to predict the winner of the World Series five years from now. Teams' past performance give only limited insight into the future.
Due to deep uncertainty
Predicted natural or other disaster probabilities are hard to estimate and thus often very inaccurate
The world is more complicated than we think or admit
Prob(sinking) = 0
1986 – Loss of shuttle Challenger
NASA claimed probability of loss = 1/100,000
Richard Feynman argued for 1/100 – 1000 times higher.
As he pointed out in his report dissenting from the government investigation commission, because this rate implies that "one could put up a shuttle every day for 300 years expecting to lose only one, we could properly ask what is the cause of management's fantastic faith in the machinery... "
1986 – Loss of shuttle Challenger
NASA claimed probability of loss = 1/100,000
Richard Feynman argued for 1/100 – 1000 times higher.
As he pointed out in his report dissenting from the government investigation commission, because this rate implies that "one could put up a shuttle every day for 300 years expecting to lose only one, we could properly ask what is the cause of management's fantastic faith in the machinery... "
In 2003, shuttle Columbia was lost on the 107th shuttle mission
2 lost in 107 missions ≈ 1/50
Activity 2.1:
Given that the shuttle was a new spacecraft, before flights started, how could you assess NASA’s estimated probability of loss = 1/100,000?
Activity 2.1:
Given that the shuttle was a new spacecraft, before flights started, how could you assess NASA’s estimated probability of loss = 1/100,000?
One way: of 11 Apollo missions, one (Apollo 13) suffered near disaster. In addition, Apollo 1 was lost in a launch pad fire.
QuickTime™ and a decompressor
are needed to see this picture.
QuickTime™ and a decompressor
are needed to see this picture.
Boeing 787 Dreamliner batteries
Boeing “concluded that they were likely to emit smoke less than once in every 10 million flight hours.
Once the planes were placed in service, the batteries overheated and emitted smoke twice, and caused one fire, after about 50,000 hours of commercial flights.” (NYT, 2/7/13)
From 1975 to 2007, U.S. housing prices grew steadily. Neither Washington nor Wall Street recognized that this could not go on forever, or worried that trillions of dollars of risky
mortgages were embedded throughout the financial system
QuickTime™ and a decompressor
are needed to see this picture.
As housing prices and subprime lenders collapsed, Wall Street & government weren’t concerned based
on computer models
The models were wrong
QuickTime™ and a decompressor
are needed to see this picture.
NY Times 3/21/11
Hazard maps are hard to get right: successfully predicting future shaking depends on accuracy of four
assumptions over 500-2500 years
Where will large earthquakes occur?
When will they occur?
How large will they be?
How strong will their shaking be?
Uncertainty & map failure result because these are often hard to assess, given that the earthquakes are much more variable in space and time than the short
earthquake history shows
2008 Wenchuan earthquake (Mw 7.9) was not expected: map showed low hazard based on lack of recent
earthquakes
Didn’t use GPS data showing 1-2 mm/yr (~Wasatch)
Earthquakes prior to the 2008 Wenchuan event
Aftershocks of the Wenchuan event delineating the rupture zone
QuickTime™ and a decompressor
are needed to see this picture.
GSHAP 1999
NUVEL-1Argus, Gordon, DeMets & Stein, 1989
Swafford & Stein, 2007
Slow plate boundary
Africa-Eurasia convergence rate varies smoothly (5 mm/yr)
2004
2003
Swafford & Stein, 2007
GSHAP 1999
NUVEL-1Argus, Gordon, DeMets & Stein, 1989
M 6.4
M 6.3
Slow plate boundary
Africa-Eurasia convergence rate varies smoothly (5 mm/yr)
Familiar pattern
2001 hazard map
http://www.oas.org/cdmp/document/seismap/haiti_dr.htm
2010 M7 earthquake shaking much greater than predicted for next 500 years
Italian hazard maps, which predicted the expected shaking in the next 500 years, forecast some earthquake locations well and others poorly, and so required updating within a decade.
A posteriori changes to a model are "Texas sharpshooting:” shoot
at the barn and then draw
circles around the bullet holes.
Overfitting: a subtle trap
We can fit very complicated models to data like earthquake histories, but we are partly fitting noise
Flipping a coin gives lots of complicated patterns
H H T T H T T T H T H H T H
We could fit a model to those data, but it would do no better than 50% at predicting the next flip
Overfitting: a subtle trap
We can fit very complicated models to data like earthquake histories, but we are partly fitting noise
In such cases, a more complicated model can give worse predictions
QuickTime™ and a decompressor
are needed to see this picture.
Activity 2.2: Time between large earthquakes from paleoseismic record on
southern San Andreas
What’s the mean time between large earthquakes here? When would you expect the next one?
Is one due soon, overdue, or…
DD 9.8
Time dependent predicts lower until ~2/3 mean recurrence
Results depend on both model choice & assumed mean recurrence
Hebden & Stein, 2008
We don’t know whether to assume that probability of a major earthquake is
- constant with time (time-independent) or
- small after a large earthquake and then increases (time-dependent ).
Activity 2.3 Deep uncertainty in earthquake recurrence
Imagine an urn containing e balls labeled "E" for earthquake, and n balls labeled "N" for no earthquake.
We can draw balls in two ways.
Activity 2.3 Deep uncertainty in earthquake recurrence
Option 1: after drawing a ball, we replace it. In successive draws, the probability of an event is constant or time-independent.
Because one event happening does not change the probability of another happening, an event is never overdue.
Activity 2.3 Deep uncertainty in earthquake recurrence
Option 2: We can add a number a of E-balls after a draw when an event does not occur, and remove r E-balls when an event occurs. This makes the probability of an event increase with time until one happens, after which it decreases and then grows again. Events are not independent, because one happening changes the probability of another.
Activity 2.3 Deep uncertainty in earthquake recurrence
Problem: Given a sequence of results, it’s hard or impossible to tell how the urn was sampled. Thus it’s hard to assess the probability of an “earthquake” in the next draw.
Italian flag graphic - one way to illustrate uncertainty we can’t
quantify well
50% 30% 20%