Upload
lonato
View
40
Download
0
Tags:
Embed Size (px)
DESCRIPTION
VIII: Methods for Evaluating Model Predictions. 1. Define predictive quantity and calculate sensitivities and standard deviations (Ex8.1a) 2. Assess data needs for the predictions Which parameters are important to predictions? - PowerPoint PPT Presentation
Citation preview
VIII: Methods for Evaluating Model Predictions
1. Define predictive quantity and calculate sensitivities and standard deviations (Ex8.1a)
2. Assess data needs for the predictions Which parameters are important to predictions?
Use composite and prediction scaled sensitivities (css and pss) and parameter correlation coefficients (pcc) (Ex8.1b)Use parameter-prediction (ppr) statistic (Ex8.1c)
What existing observations are important to predictions?
Use observation-prediction (opr) statistic (Ex8.1d)
How important are the proposed new observations?Use dss, css, pcc (Ex8.1e)Use opr (Ex8.1f)
3. Quantify prediction uncertaintyLinear confidence and prediction intervalsNonlinear confidence and prediction intervalsMonte Carlo – no exercise
Exercise filesMFI2005 does not support predictionsAll the files have been constructed for you, and are in initial\ex8.i
Copy the files to exer\ex8Execute batch files as described in the input instructions.Look at output files
For other problems, you can use the class files as a beginning point.
Define predictive quantityWe typically make predictions to assess the state of the simulated system at a future time.
Often, predictions occur under different conditions than those under which the model was calibrated – addition of pumping, change in boundary conditions, etc.
The predictive quantity might differ from the types of quantities used as observations.
You as the modeler need to help define what model-calculated values are best to use as predictions in the problem of concern
The predictions available can depend on the software used
MODFLOW-2005: any quantity that can be an observation can also be a prediction. Heads, temporal changes in head, some flows, advective transport.
UCODE_2005 or PEST: any quantity that can be calculated using values in the application model output files.
Prediction sensitivities (Book, p. 159)
Used to calculate measures for assessing:Parameters important to predictionsObservations important to predictionsPrediction uncertaintyData needs
Prediction sensitivities:
Usually need to scale these sensitivities to produce useful measures.
jb
z
Prediction scaled sensitivities (pss) (Book, p. 160-161)
Scaling depends on prediction and purpose. Four useful scalings are:1. pss is percent change in prediction caused by a 1-percent change in the parameter value bj (in _sppp file of UCODE_2005):
pssj = (z/ bj) (bj/100) (100/z)
2. pss is percent change in prediction caused by a change in bj equal to 1 percent of its standard deviation sbj (in _spsp file):
pssj = (z/ bj) (sbj/100) (100/z)
3,4. For both these scalings, (100/z) can be replaced by
(100/a), where a’ is some other meaningful quantity. On p.
161, using a reference value is suggested. (in _sppr and _spsr file of UCODE_2005)
In _sp** , third letter identifies pparameter value or sstandard deviation on the parameter; fourth letter identifies pprediction or rreference value)
Parameter Correlation Coefficients (pcc) in the Context of Predictions (Book, p. 162-166)
Calculate pcc using different combinations of observations, prior, and predictions to evaluate the existence and need for unique parameter estimates
For example, two parameters are extremely correlated given a set of observations. This is only a problem if the predictions need unique values.
Next: Use pss, css, and pcc together to assess the importance of parameters to the predictions.
Use pss and css to identify insensitivity problems
Are imprecise parameter estimates important to the predictions?
AcceptableAcceptableNot important: Small pss
Importance of the parameter to predictions of interest
Precise: Large cssImprecise: Small css
Acceptable – consider representing system feature(s)
with more parameters
Improve estimation of this parameter and representation
of system feature(s)
Important: Large pss
Precision of parameter estimate
AcceptableAcceptableNot important: Small pss
Importance of the parameter to predictions of interest
Precise: Large cssImprecise: Small css
Acceptable – consider representing system feature(s)
with more parameters
Improve estimation of this parameter and representation
of system feature(s)
Important: Large pss
Precision of parameter estimate
(from Hill & Tiedeman, Figure 8.2a, p. 166)
Use pcc to identify uniqueness problems
Are non-unique parameter estimates important to the predictions?
AcceptableAcceptableNot important: |pcc| with preds ~ 1
Importance of unique parameter estimates to predictions of interest
Unique: |pcc| < ~0.95Nonunique: |pcc| ~ 1
Acceptable – consider representing system feature(s)
with more parameters
Improve estimation of one or both parameters and
improve representation of system feature(s)
Important: |pcc| with preds < ~0.95
Uniqueness of the estimates for a parameter pair
AcceptableAcceptableNot important: |pcc| with preds ~ 1
Importance of unique parameter estimates to predictions of interest
Unique: |pcc| < ~0.95Nonunique: |pcc| ~ 1
Acceptable – consider representing system feature(s)
with more parameters
Improve estimation of one or both parameters and
improve representation of system feature(s)
Important: |pcc| with preds < ~0.95
Uniqueness of the estimates for a parameter pair
(from Hill & Tiedeman, Figure 8.2b, p. 166)
• Developer claims landfill effluent will go to river, not to wells.
• Water supply wells are being completed and a pump test could be used to collect more data on the system.
• Developer claims the model is inadequate for two reasons: 1. Model was calibrated using heads and flows and no
pumping, but is being used to predict advective transport under pumping conditions.
2. Need for prior suggests the observations are inadequate
• County government officials want to know:• Is existing model adequate? • Wait for new data?
• We, as the modelers, suggest using sensitivity analysis to:• Evaluate the developer’s claims. • Plan new data collection
Prediction Exercises – Book, p. 193-212
Prediction Exercises
Will effluent go to well or river?
Thorough analysis requires full transport model (advection, dispersion, etc.).
We will do preliminary analysis using Advective Transport. Use MODPATH.
?
?
Predicted transport from landfill
Predictions simulated using MODPATH (Pollock 1996)
Calculates motion in the x, y and z directions.When used for observations:
Observed advective transport inferred from concentrations.Three entries are added to the objective function for each advective-transport observation.
When used for predictions:Three predictions for each advective transport prediction – transport along rows (y), columns (x), and vertically (z).
Calculates sensitivities; these are used here to calculate pcc, parameter standard deviations, and linear confidence intervals.
Start of pathParticle path simulated using MODPATH.
The horizontal and vertical bars show one standard deviation at selected travel times.
End of path
Predictive transport: Questions to address (p. 194)
1. Transport destination and time? Ex. 8.1a. Forward model with MODPATH
2. Consider the parameters important to predictions. Are they precisely and uniquely estimated?
Prediction and composite scaled sensitivities and parameter correlation coefficients (pss, css, pcc). Ex. 8.1b. PPR statistic. Ex. 8.1c
3. What existing observations are important? OPR statistic. Ex. 8.1d
4. Are potential new observations worth waiting for?Dimensionless and prediction scaled sensitivities, parameter correlation coefficients (dss, pss, pcc). Ex. 8.1eOPR statistic. Ex. 8.1f
5. What is the prediction uncertainty? Linear confidence intervals. Ex. 8.2a. Nonlinear confidence intervals. Ex 8.2b.
Exercise 8.1a: Predict Advective Transport
Read general Exercises description on p. 193-195.Do Exercise 8.1a on p. 195-196
In the simulation, where does the landfill effluent go, and how long does it take to get there?
MODPATH Output (Equivalent to ADV output on Fig. 8.7a, p. 196
@ [ MODPATH Version 4.00 (V4, Release 3, 7-2003) (TREF= 0.000000E+00 ) ] 1 1.55000E+04 1.65000E+04 9.99900E-01 9.99950E+01 0.00000E+00 16 2 1 1 1 1.51565E+04 1.63903E+04 7.87957E-01 8.93978E+01 -3.15000E+08 16 2 1 1 1 1.51565E+04 1.63903E+04 7.87957E-01 8.93978E+01 3.15000E+08 16 2 1 1 1 1.50000E+04 1.63422E+04 7.10433E-01 8.55216E+01 4.46763E+08 15 2 1 1 1 1.40917E+04 1.60000E+04 4.02401E-01 7.01200E+01 1.11285E+09 15 3 1 1 1 1.40000E+04 1.59693E+04 3.82440E-01 6.91220E+01 1.16709E+09 14 3 1 1 1 1.32700E+04 1.56553E+04 2.56172E-01 6.28086E+01 -1.57000E+09 14 3 1 1 1 1.32700E+04 1.56553E+04 2.56172E-01 6.28086E+01 1.57000E+09 14 3 1 1 1 1.30000E+04 1.55379E+04 2.19734E-01 6.09867E+01 1.70786E+09 13 3 1 1 1 1.20897E+04 1.50000E+04 1.26071E-01 5.63035E+01 2.14586E+09 13 4 1 1 1 1.20000E+04 1.49508E+04 1.19582E-01 5.59791E+01 2.18187E+09 12 4 1 1 1 1.10000E+04 1.41688E+04 5.97149E-02 5.29857E+01 2.58234E+09 11 4 1 1 1 1.08476E+04 1.40000E+04 5.20098E-02 5.26005E+01 2.64385E+09 11 5 1 1 1 1.00382E+04 1.30000E+04 1.70220E-02 5.08511E+01 2.94240E+09 11 6 1 1 1 1.00000E+04 1.29501E+04 1.52036E-02 5.07602E+01 2.95493E+09 10 6 1 1 1 9.76378E+03 1.25238E+04 0.00000E+00 5.00000E+01 3.03896E+09 10 6 1 1 1 9.76378E+03 1.25238E+04 -3.23767E-01 4.67623E+01 -3.15000E+09 10 6 1 1 1 9.76378E+03 1.25238E+04 -3.23767E-01 4.67623E+01 3.15000E+09 10 6 1 1 1 9.76378E+03 1.25238E+04 1.00000E+00 4.00000E+01 3.38193E+09 10 6 2 1 1 9.56571E+03 1.20000E+04 9.22715E-01 3.61357E+01 3.83712E+09 10 7 2 1 1 9.38455E+03 1.10000E+04 6.69144E-01 2.34572E+01 4.32990E+09 10 8 2 1 1 9.38417E+03 1.00000E+04 3.71525E-01 8.57624E+00 4.49415E+09 10 9 2 1
Distance along rows from left
Distance along columns from bottom
Level above
bottom of model
Row, column, layer
Level above
bottom of model layer
Time, in
seconds
Exercise 8.1b: Determine problematic parameters using css and pss
1. Compare prediction and composite scaled sensitivities: Are pss large for any parameters with small css?
• Use pss scaling so that pss represent the percent change in distance traveled caused by a 1-percent change in a parameter value:
For UCODE_2005, these pss are in ex8.1b \ ex8.1b_sppp.
pssj = (z/ bj) (bj/100) |100/z| One-percent scaled sens
Exercise 8.1b: Determine problematic parameters using pcc
2. Compare two different sets of parameter correlation coefficients:
• pcc calculated using only calibration observations• pcc calculated with calibration observations AND
predictions.
• Use parameter correlation coeffeicients in ex8.1b \ ex8.1b._mc or _pcc
ex8.1b-obs-pred\
Weights for the advective travel predictions are calculated from specified standard deviations, in meters (Table 8.3, p. 198). Weights in this analysis can reflect desired prediction precision.
Time of advective travel
Direction 10 years 50 years 100 years
X 200 m 600 m 1000 m
Y 200 m 600 m 1000 m
Z 10 m 15 m 25 m
Exercise 8.1b: Prediction weights needed for
pcc with predictions
Do Exercise 8.1b (p. 196-199) and the Problems.
Question 2: Are parameters important to predictions
precisely and uniquely estimated?
Results of Exercise 8.1b
css-pss analysis for Question 2: Are parameters important to predictions precisely
estimated?
0
1
2
HK_1 K_RB VK_CB HK_2 RCH_1 RCH_2 POR_1&2 POR_CB
Parameter Name
0
10
20
30
40
Com
posi
te s
cale
d se
nsit
ivit
y ( css
)AD10x AD10y AD10z
AD50x AD50y AD50zA100x A100y A100z
css
Abs
olut
e va
lue
of p
redi
ctio
nsc
aled
sen
siti
vity
( pss
)
Figure 8.8, p. 198
Results of Exercise 8.1b
css-pss analysis for Question 2: Are parameters important to predictions precisely
estimated?
0
1
2
HK_1 K_RB VK_CB HK_2 RCH_1 RCH_2 POR_1&2 POR_CB
Parameter Name
0
10
20
30
40
Com
posi
te s
cale
d se
nsit
ivit
y ( css
)AD10x AD10y AD10z
AD50x AD50y AD50zA100x A100y A100z
css
Abs
olut
e va
lue
of p
redi
ctio
nsc
aled
sen
siti
vity
( pss
)
Figure 8.8, p. 198
Not all of them
HK_1 K_RB VK_CB HK_2 RCH_1 RCH_2
HK_1 1.00 -0.40 -0.90 -0.93 0.96 -0.90
K_RB 1.00 0.20 0.34 -0.32 0.32
VK_CB 1.00 0.97 -0.97 0.97
HK_2 symmetric 1.00 -0.99 0.996
RCH_1 1.00 -0.98
RCH_2 1.00
HK_1 K_RB VK_CB HK_2 RCH_1 RCH_2
HK_1 1.00 -0.16 0.078 -0.22 0.71 0.26
K_RB 1.00 -0.61 0.013 0.25 -0.070
VK_CB 1.00 0.33 -0.19 0.28
HK_2 symmetric 1.00 -0.52 0.83
RCH_1 1.00 -0.30
RCH_2 1.00
Observations only
With predictions
pcc analysis
for Question
2: Are parameter
s important
to predictions uniquely
estimated
?
Results of Exercise 8.1b
Tables 8.4 & 8.5,p. 199
No correlations larger than 0.85
Not all of them
Exercise 8.1c: PPR Individual Parameters
Which parameters rank as most important to the predictions by the ppr statistic?0
2
4
6
8
10
HK_1 K_RB VK_CB HK_2 RCH_1 RCH_2 POR_1&2
Parameter Name
Ave
rage
ppr
sta
tist
ic
(a) Average ppr statistic for all predictionsFigure 8.9a, p. 201
0
1
2
HK_1 K_RB VK_CB HK_2 RCH_1 RCH_2 POR_1&2 POR_CB
Parameter Name
0
10
20
30
40
Com
posi
te s
cale
d se
nsit
ivit
y ( css
)AD10x AD10y AD10z
AD50x AD50y AD50zA100x A100y A100z
css
Abs
olut
e va
lue
of p
redi
ctio
nsc
aled
sen
siti
vity
( pss
)
How does this differ from the rankings by the pss?
What causes these differences?
Response to developer concerns
Developer claims the model is inadequate for two reasons:
Model was calibrated using heads and flows and no pumping, but is being used to predict advective transport under pumping conditions.Need for prior suggests the observations are inadequate
Our response based on sensitivity analysisIndeed, the model is inadequate. The calibration data does not provide information on some parameters important to predictions
Also, we stress that we need to consider the results of an uncertainty analysis