Upload
hagen
View
32
Download
0
Embed Size (px)
DESCRIPTION
Validating uncertain predictions. Tony O’Hagan, Leo Bastos , Jeremy Oakley, University of Sheffield. Why am I here?. I probably know less about finite elements modelling than anyone else at this meeting But I have been working with mechanistic models of all kinds for almost 20 years - PowerPoint PPT Presentation
Citation preview
Validating uncertain predictions
Tony O’Hagan, Leo Bastos, Jeremy Oakley,
University of Sheffield
Why am I here? I probably know less about finite elements
modelling than anyone else at this meeting But I have been working with mechanistic models of all
kinds for almost 20 years Models of climate, oil reservoirs, rainfall runoff, aero-
engines, sewer systems, vegetation growth, disease progression, ...
What I do know about is uncertainty I’m a statistician My field is Bayesian statistics One of my principal research areas is to understand,
quantify and reduce uncertainty in the predictions made by models
I bring a different perspective on model validation6/9/20092 mucm.group.shef.ac.uk
Some background Models are often highly computer intensive
Long run times FE models on fine grid
Oil reservoir simulator runs can take days
Things we want to do with them may require many runs Uncertainty analysis
Exploring output uncertainty induced by uncertainty in model inputs
Calibration Searching for parameter values to match observational data
Optimisation Searching for input settings to optimise output
We need efficient methods requiring minimal run sets
6/9/20093 mucm.group.shef.ac.uk
Emulation We use Bayesian statistics Based on a training sample of model runs, we
estimate what the model output would be at all untried input configurations
The result is a statistical representation of the model In the form of a stochastic process over input space The process mean is our best estimate of what the
output would be at any input configuration Uncertainty is captured by variances and covariances
It correctly returns what we know At any training sample point, the mean is the observed
value With zero variance
6/9/20094 mucm.group.shef.ac.uk
2 code runs Consider one input and one output Emulator estimate interpolates data Emulator uncertainty grows between data points
mucm.group.shef.ac.uk 6/9/20095
3 code runs Adding another point changes estimate and
reduces uncertainty
mucm.group.shef.ac.uk 6/9/20096
5 code runs And so on
mucm.group.shef.ac.uk 6/9/20097
MUCM The emulator is a fast meta-model but with a
full statistical representation of uncertainty We can build the emulator and use it for tasks
such as calibration with far fewer model runs than other methods Typically 10 or 100 times fewer
The RCUK Basic Technology grant Managing Uncertainty in Complex Models is developing this approach http://mucm.group.shef.ac.uk See in particular the MUCM toolkit
6/9/20098 mucm.group.shef.ac.uk
Validation What does it mean to validate a simulation
model? Compare model predictions with reality But the model is always wrong How can something which is always wrong ever be
called valid? Conventionally, a model is said to be valid if its
predictions are close enough to reality How close is close enough? Depends on purpose Conventional approaches to validation confuse the
absolute (valid) with the relative (fit for this purpose) Let’s look at an analogous validation problem
6/9/20099 mucm.group.shef.ac.uk
Validating an emulator
6/9/2009mucm.group.shef.ac.uk10
What does it mean to validate an emulator? Compare the emulator’s predictions with the reality of
model output Make a validation sample of runs at new input
configurations The emulator mean is the best prediction and is always
wrong But the emulator predicts uncertainty around that mean
The emulator is valid if its expressions of uncertainty are correct Actual outputs should fall in 95% intervals 95% of the time
No less and no more than 95% of the time Standardised residuals should have zero mean and unit
variance See Bastos and O’Hagan preprint on MUCM website
Validation diagnostics
6/9/2009mucm.group.shef.ac.uk11
Validating the model
6/9/2009mucm.group.shef.ac.uk12
Let’s accept that there is uncertainty around model predictions
We need to be able to make statistical predictions Then if we compare with observations we can see
whether reality falls within the prediction bounds correctly
The difference between model output and reality is called model discrepancy It’s also a function of the inputs Like the model output, it’s typically a smooth function Like the model output, we can emulate this function We can validate this
Model discrepancy
6/9/2009mucm.group.shef.ac.uk13
Model discrepancy was first introduced within the MUCM framework in the context of model calibration Ignoring discrepancy leads to over-fitting and over-
confidence in the calibrated parameters Understanding that it is a smooth error term rather
than just noise is also crucial To learn about discrepancy we need a training
sample of observations of the real process Then we can validate our emulation of reality
using further observations This is one ongoing strand of the MUCM project
Beyond validation
6/9/2009mucm.group.shef.ac.uk14
An emulator (of a model or of reality) can be valid and yet useless in practice Given a sample of real-process observations, we can
predict the output at any input to be the sample mean plus or minus two sample standard deviations
This will validate OK Assuming the sample is representative
But it ignores the model and makes poor use of the sample!
Two valid emulators can be compared on the basis of the variance of their predictions
And declared fit for purpose if the variance is small enough
In conclusion
6/9/2009mucm.group.shef.ac.uk15
I think it is useful to separate the absolute property of validity from the relative property of fitness for purpose
Model predictions alone are useless without some idea of how accurate they are
Quantifying uncertainty in the predictions by building an emulator allows us to talk about validity
Only valid statistical predictions of reality should be accepted Model predictions with a false measure of their accuracy are
also useless! We can choose between valid predictions on the basis
of how accurate they are And ask if they are sufficiently accurate for purpose
Advertisement
6/9/2009mucm.group.shef.ac.uk16
Workshop on emulators and MUCM methods
“Uncertainty in Simulation Models”
Friday 10th July 2009
10.30am - 4pm
National Oceanography Centre Southampton
http://mucm.group.shef.ac.uk/Pages/Project_News.htm
Please register with Katherine Jeays-Ward
([email protected]) by 3rd July 2009
Registration is free, and lunch/refreshments will be
provided