Upload
others
View
4
Download
0
Embed Size (px)
Citation preview
Advanced uncertainty evaluation of climatemodels by Monte Carlo methods
Marko [email protected]
Pirkka Ollinaho, Janne Hakkarainen, Johanna Tamminen, HeikkiJärvinen (FMI)
Antti Solonen, Heikki Haario (LUT)Alexander Ilin, Erkki Oja (Aalto)
FMI – Finnish Meteorological InstituteLUT – Lappeenranta University of Technology
SIAM UQ 2012
Contents
Adaptive Markov chain Monte Carlo methodEfficient MCMC for short chainsParallel chainsEarly rejection
MCMC for ECHAM5 climate modelFormulating the cost functionStochastic Lorenz 95 test case
EPPES – ensemble prediction and parameter estimationLorenz 95 experiment with EPPESECHAM5 experiment with EPPES
M.Laine: Advanced uncertainty evaluation of climate models by Monte Carlo methods 2/27
Adaptive MCMC
The current work is based on our success on applying anddeveloping adaptive Markov chain Monte Carlo (MCMC) forchemical kinetics, ecological models, and satellite retrieval. In2010 we started a project on applying MCMC for climate modelclosure parameter uncertainty evaluation.
• Haario H., Saksman E., Tamminen J.: An adaptive Metropolis algorithm.Bernoulli 7(2), pp. 223–242, 2001.
• Haario H., M. Laine, M. Lehtinen, E. Saksman, J. Tamminen: Markovchain Monte Carlo methods for high dimensional inversion in remote sensing,with discussion, J.R. Statist. Soc. B, 66, part 3 pp. 591–607, 2004.
• Haario H., M. Laine, A. Mira, E. Saksman: DRAM: Efficient adaptiveMCMC. Stat. Comput. 16, pp. 339–354, ISSN 0960-3174, 2006.
• Laine, M., J. Tamminen: Aerosol model selection and uncertainty modellingby adaptive MCMC technique. Atmos. Chem. Phys., 8, 2008.
M.Laine: Advanced uncertainty evaluation of climate models by Monte Carlo methods 3/27
Terminology for modeling with MCMC methodsThe observation model in general form is
y = f (x|θ) + ε,observations = model + error.
Likelihood function for Gaussian errors corresponds to aquadratic cost function, with
p(y|θ) ∝ exp
{−1
2∑n
i (yi − f (xi|θ))2
σ2
}
= exp{−1
2SS(θ)
σ2
},
where SS(θ) = −2 log(p(y|θ)), the log-likelihood in"sum-of-squares" cost function format. For calculating theposterior, we also need to account SSpri(θ) = −2 log(p(θ)), theprior "sum-of-squares".
M.Laine: Advanced uncertainty evaluation of climate models by Monte Carlo methods 4/27
Metropolis-Hastings algorithm
Random walk Metropolis-Hastings algorithm with Gaussianproposal distribution (and Gaussian likelihood).
• Propose new parameter value θprop = θcurr + ξ, whereξ ∼ N(0, Σprop) is drawn from the proposal distribution.
• Accept θprop with probability α,
α(θcurr, θprop) = 1∧ exp{− 1
2
(SS(θprop)− SS(θcurr)
σ2
)
− 12
(SSpri(θprop)− SSpri(θcurr)
)}
• Efficient proposal distribution⇒ adaptive tuning of Σprop,AM, DRAM algorithms.
M.Laine: Advanced uncertainty evaluation of climate models by Monte Carlo methods 5/27
Short chains and adaptation• It is important to make short chains as efficient as possible.• Efficient: produce estimates with small Monte Carlo error.
0 1000 2000 3000 4000 50000
0.1
0.2
0.3
0.4
0.5
simulation index
mean of the 1. parameter
mhamdramram
0 1000 2000 3000 4000 50000.94
0.95
0.96
0.97
0.98
0.99
195% quantile
simulation index
mhamdramram
Short MCMC chain repeated 1000 times with differentalgorithms.Gaussian 10 dimensional target, too large initial covariance.
M.Laine: Advanced uncertainty evaluation of climate models by Monte Carlo methods 6/27
Short chains and adaptation
• But, adaptation might slow the convergence.
0 1000 2000 3000 4000 50000
0.1
0.2
0.3
0.4
0.5
0.6
0.7
simulation index
mean of the 1. parameter
mhamdramram
0 1000 2000 3000 4000 50000.94
0.95
0.96
0.97
0.98
0.99
195% quantile
simulation index
mhamdramram
Same as in the previous slide, but now with more optimalinitial proposal.Gaussian 10 dimensional target, near optimal initial covariance.
M.Laine: Advanced uncertainty evaluation of climate models by Monte Carlo methods 7/27
Faster MCMC: parallel chains
• Random walk MCMC isby nature sequential, andit is generally moreefficient to run one longchain than many shortindependent chains.
• In parallel adaptiveMCMC, the adaptation isdone over the points in allchains and they share onecommon adapted proposalcovariance.
• Communication betweenthe chains can beasynchronous.
θi+1 ∼ N(θi, Σi)
α = min(1, π(θi+1)/π(θi))
θi+1 = θi +1
i + 1(θi − θi)
Σi+1 =i− 1
iΣi +
1i(θi − θi)(θi − θi)T
θi+1 ∼ N(θi, Σi)
α = min(1, π(θi+1)/π(θi))
θi+1 ∼ N(θi, Σi)
α = min(1, π(θi+1)/π(θi))
π(θi+1) ∝ p(θi+1)L(y|θi+1) π(θi+1) ∝ p(θi+1)L(y|θi+1) π(θi+1) ∝ p(θi+1)L(y|θi+1)
M.Laine: Advanced uncertainty evaluation of climate models by Monte Carlo methods 8/27
Faster MCMC: early rejectionIdea: evaluate the likelihood in parts and check after each partif the proposed parameter value can be rejected.
Cumulative cost function evaluated after each monthduring one year climate model simulation
0
10
20
30
40
50
60
2 4 6 8 10 12
COST FUNCTION VALUE
MONTH1 2 3 4 5 6 7 8 9 10 11 12
0
200
400
600
800
1000
1200
Early rejection month
time to stop the simulation proportion of stopped runs by month
This simple trick saved 10%–80% of CPU time in different testcases.
M.Laine: Advanced uncertainty evaluation of climate models by Monte Carlo methods 9/27
Early rejection
• In many cases SS(θ) is a monotonically increasing functionwrt. adding new observations or simulating the modelfurther in time.
• With the acceptance probability α(θcurr, θprop) defined inprevious slides, we draw u ∼ U(0, 1), and accept if
−2 log(u) <SS(θprop)− SS(θcurr)
σ2 + SSpri(θprop)−SSpri(θcurr).
• If we write this as
SScrit = −2 log(u) + SS(θcurr)/σ2 + SSpri(θcurr)
< SS(θprop)/σ2 + SSpri(θprop),
then we can stop evaluating the model whenSS(θprop) ≥ (SScrit − SSpri(θprop))σ2.
M.Laine: Advanced uncertainty evaluation of climate models by Monte Carlo methods 10/27
MCMC for ECHAM5 climate model
Studies on feasibility of MCMC for large scale climate models,on the formulation of the likelihood and on the problem ofchaotic behaviour of the models.
• H. Järvinen, P. Räisänen, M. Laine, J. Tamminen, A. Ilin, E. Oja, A.Solonen, H. Haario: Estimation of ECHAM5 climate model closureparameters with adaptive MCMC, Atmospheric Chemistry and Physics,10(1), pages 9993–10002, 2010. doi:10.5194/acp-10-9993-2010
• J. Hakkarainen, A. Ilin, A. Solonen, M. Laine, H. Haario, J. Tamminen,E. Oja, H. Järvinen: On closure parameter estimation in chaotic systems,Nonlinear Processes in Geophysics, 19(1), pages 127–143, 2012.doi:10.5194/npg-19-127-2012
• A. Solonen, P. Ollinaho, M. Laine, H. Haario, J. Tamminen, H. Järvinen:Efficient MCMC for climate model parameter estimation: paralleladaptive chains and early rejection, Bayesian Analysis, 7(2), 1–22 2012.
M.Laine: Advanced uncertainty evaluation of climate models by Monte Carlo methods 11/27
Climate model parametrization• In climate models, the atmosphere is divided into cells.• The scale of many important processes is smaller than a
single cell (e.g. clouds and rain).• These processes are parametrized: for example clouds and
rain are calculated based on the knowledge of thetemperature and humidity in the cell.2.3. Performance Metrics for Climate Models Chapter 2. Modeling Climate
Figure 2.2: The grid points that ECHAM5 uses in horizontal resolution T21.
Roeckner et al. (2006) have tested the sensitivity of the model as to the choice ofspatial resolution and concluded that the convergence of the model to a realisticclimate depends both on the choice of the horizontal and the vertical resolution.Intuitively, the denser the grid, the better the model performs in producing a realisticclimate. However, according to the sensitivity analysis, this is not entirely true. Forinstance, when vertical resolution is L19, it does not matter if the horizontal resolutionis increased above T42. On the other hand, with the vertical resolution of L31, themodel performance becomes monotonically better when the horizontal resolution isincreased.
Running ECHAM5 is very demanding for today’s computers. For instance, Max-Planck Institute for Meteorology lists on their website computer resources needed tointegrate ECHAM5 at specified resolutions1. In their experiment, they have used thesupercomputer NEC SX-6 with four CPUs for the resolution T21L19, with which ittook about 16 minutes to simulate the climate for one year. When the resolution wasincreased to T31L19, the time needed was doubled.
2.3 Performance Metrics for Climate Models
2.3.1 Key Concepts and Notation
This section introduces some basic methodology that can be employed to assess thesuccess of climate simulations. Determining the accuracy of short-term weather fore-casts or long-term climate projections is broadly referred to as forecast verification.2
1http://www.mpimet.mpg.de/en/wissenschaft/modelle/echam/echam5/runcontrol/echam5-computer-resources.html
2This thesis makes distinction only between short-term weather forecasts and long-term climateprojections. Here the distinction is that short-term weather forecasts are comparable to observations
10
M.Laine: Advanced uncertainty evaluation of climate models by Monte Carlo methods 12/27
Stochastic Lorenz 95 test case
We have 40 slow state variables -•- and 320 fast state variables-•-, whose effect is parametrized in the forecast model.
Good test case to study:• Estimation
methodologies.• Different
parameterizationsg(xk, θ).
• Modeling error.• Filtering and
ensemble methods.
3 2 1 40 39
dxk
dt= −xk−1 (xk−2 − xk+1)− xk + F−hc
b
Jk�
j=J(k−1)+1
yj
dyj
dt= −cbyj+1 (yj+2 − yj−1)− cyj +
c
bFy +
hc
bx1+� j−1
J �
NATURE:
FORECAST MODEL: dxk
dt= −xk−1 (xk−2 − xk+1)− xk + F−g(xk, θ)
Wilks, D.: Effects of stochastic parametrizations in the Lorenz ’96 system, Quart. J. Roy. Meteor. Soc., 131(606),389–407, 2005. doi:10.1256/qj.04.03
M.Laine: Advanced uncertainty evaluation of climate models by Monte Carlo methods 13/27
Lorenz 95 – true forcing vs. parameterization
10 5 0 5 10 153
2
1
0
1
2
3
4
5
6
7
slow variables
forc
ing
due
to fa
st v
aria
bles
The true effect of the fast variables wrt. the values of the slowvariables in full model simulation. Red lines give estimatedoptimal linear parameterization and its uncertainty.
M.Laine: Advanced uncertainty evaluation of climate models by Monte Carlo methods 14/27
Climate model parametrization
• Currently, best expert knowledge is used to define theoptimal closure parameter values, based on observations,process studies, etc.
• Closure parameters act as ’tuning handles’ of thesimulated climate.
• Our goal is to come up with an objective, algorithmic wayto determine the closure parameters.
ECHAM5 parameters estimated by MCMC and ensemble methods:Parameter DescriptionCAULOC A parameter influencing the accretion of cloud droplets
by precipitation (rain formation in stratiform clouds).CMFCTOP Relative cloud mass flux at the level above non-buoyancy
(in cumulus mass flux scheme).CPRCON A coefficient for determining conversion from cloud water to rain
(in convective clouds).ENTRSCV Entrainment rate for shallow convection.
M.Laine: Advanced uncertainty evaluation of climate models by Monte Carlo methods 15/27
Cost function - likelihood
• For climate model parameter estimation, it is essential todefine a metric with which we can measure the ’goodness’of the parameters.
• In our MCMC experiments, we have specified differentcost functions based on the net radiation at the top of theatmosphere.
• Example: global F and zonal Ft,x modeled mean fluxes arecompared to the observed ones, Fo and Fo
t,x.
J(θ) =(F− Fo)2
σ2 +12
∑t=1
∑y
(Ft,x − Fot,x)
2
σ2t,x
M.Laine: Advanced uncertainty evaluation of climate models by Monte Carlo methods 16/27
ECHAM5 – MCMC results
chain histograms pairwise chains
0 5 10 15 20 25 30
CAULOC
0 0.05 0.1 0.15 0.2
CMFCTOP
0 0.005 0.01 0.015
CPRCON
0 1 2 3 4 5
x 10−3
ENTRSCV
4 4.5 5 5.5 6
sqrt(costf.)
0
0.05
0.1
CM
FC
TO
P
CAULOC
5
10
15x 10
−3
CP
RC
ON
CMFCTOP
2
4
x 10−3
EN
TR
SC
V
CPRCON
0 10 20 30
4.5
5
5.5
sqrt
(cos
tf.)
0 0.05 0.1 0 5 10 15
x 10−3
0 2 4
x 10−3
ENTRSCV
Conducting a MCMC experiment of about 3000 one yearclimate model runs (ECHAM5 in T21 resolution), required 3months worth of super computer time.
M.Laine: Advanced uncertainty evaluation of climate models by Monte Carlo methods 17/27
EPPES – ensemble prediction and parameterestimation system
• Estimating closure parameters using ensemble runs.• Initially aimed for estimating numerical weather
prediction (NWP) model closure parameters using existingoperational ensemble prediction (EPS) infra-structure.
• But usable for climate model uncertainty analysis, too.
• M. Laine, A. Solonen, H. Haario, H. Järvinen: Ensemble prediction andparameter estimation system: the method, Quarterly Journal of the RoyalMeteorological Society, 138(663), 2012. doi:10.1002/qj.922
• H. Järvinen, M. Laine, A. Solonen, H. Haario: Ensemble prediction andparameter estimation system: the concept, Quarterly Journal of the RoyalMeteorological Society, 138(663), 2012. doi:10.1002/qj.923
• P. Ollinaho, H. Järvinen, M. Laine, A. Solonen, H. Haario: NWP modelforecast skill optimization via closure parameter variations, QuarterlyJournal of the Royal Meteorological Society, in revision, 2012.
M.Laine: Advanced uncertainty evaluation of climate models by Monte Carlo methods 18/27
The EPPES concept for NWP
EPPES = EPS + parameter estimation.
• In addition to initial valueperturbations, modelparameters θ are sampledfrom a proposaldistribution.
• The parameters areweighted according to acost function that dependson forecast skill.
• The backgrounduncertainty in θ is modeledas Gaussian θ ∼ N(µ, Σ)with µ and Σ estimatedsequentially.
model space
* * *
parameter space
1.2 1.4 1.6 1.8 2 2.20.2
0.1
0
0.1
0.2
0.3
0.4
1
2
M.Laine: Advanced uncertainty evaluation of climate models by Monte Carlo methods 19/27
Cost function and likelihoodCost function for model F, observations yk, initial values xk,and parameter θk for forecast time window k:
(yk − F(xk; θk))′ Σ−1
obs (yk − F(xk; θk))
+ (θk − µ)′ Σ−1 (θk − µ)= Jobs + Jpri,
interpreted as a statistical inverse problem:
p(θk|yk) ∝ exp(−12
Jobs) exp(−12
Jpri)
∝ p(yk|θk)p(θk)= likelihood · prior.
In EPPES we are estimating local parameters θk by importancesampling for p(θk|yk), and also the optimal global hyperparameters µ and Σ that describe the variability in θ betweenthe time windows by sequential updates.
M.Laine: Advanced uncertainty evaluation of climate models by Monte Carlo methods 20/27
Random effect model
• The true value of theparameter θ istreated as random.
• The distributiongeneratingindividual θk foreach forecast timewindow is assumedGaussian and theparameters of this"meta" distributionare estimatedsequentially.
The model:
θk ∼ N(µk, Σk)µk ∼ N(µk−1, Wk−1)Σk ∼ iWish(Σk−1, nk−1).
with the following update formulas:
Wk =(
W−1k−1 + Σ−1
k−1
)−1
µk = Wk
(W−1
k−1µk−1 + Σ−1k−1θk
)
nk = nk−1 + 1Σk =
(nk−1Σk−1 + (θi − µk)(θk − µk)′
)/nk.
M.Laine: Advanced uncertainty evaluation of climate models by Monte Carlo methods 21/27
Stochastic Lorenz 95 test case• We parameterize the effect of the fast states -•- using linear
model with two closure parameters in g(xk, θ) of theforecast model.
• We run 50 member ensembles sequentially by perturbingboth the initial values and the closure parameters.
• We observe some ofthe slow variablesevery 2nd "day" ofthe truth run withadded observationerror.
• Cost function isconstructed using6 day forecast skill.
3 2 1 40 39
dxk
dt= −xk−1 (xk−2 − xk+1)− xk + F−hc
b
Jk�
j=J(k−1)+1
yj
dyj
dt= −cbyj+1 (yj+2 − yj−1)− cyj +
c
bFy +
hc
bx1+� j−1
J �
NATURE:
FORECAST MODEL: dxk
dt= −xk−1 (xk−2 − xk+1)− xk + F−g(xk, θ)
M.Laine: Advanced uncertainty evaluation of climate models by Monte Carlo methods 22/27
Stochastic Lorenz 95 experiment
Evolution of two parameters in the L95 experiment. On left,each column of points corresponds to proposed parametervalues in one time window of the sequential estimationprocedure. On right, forecast skill is calculated over a grid,with an ellipse and dot showing the final estimated Σ and µ.
parameter evolution "6 day" forecast skill
50 100 150 200 250
0.5
1
1.5
θ 1
50 100 150 200 250
−0.2
0
0.2
0.4
0.6
θ 2
Ensemble number
M.Laine: Advanced uncertainty evaluation of climate models by Monte Carlo methods 23/27
ECHAM5 climate model experiment
ECHAM5 atmospheric general circulation model was run inensemble prediction (EPS) mode to estimate of 4 closureparameters related to cloud formation.
• Control run and 50 perturbed members.• Initial states from ECMWF EPS system, with data from Jan
2011 to March 2011, 00 and 12UTC daily.• Model resolution: T42 truncation with 31 levels.• Total of 2× 90× 51 = 9180 parameter sample points.• Cost function: RMS of 500 hPa geopotential height 10 days
forecast error.
M.Laine: Advanced uncertainty evaluation of climate models by Monte Carlo methods 24/27
ECHAM5 experiment
First iteration vs. last iteration as 2 dimensional plots showingthe proposed parameter values, the weights they havereceived, and the matrices Σ and W of the EPPES algorithm.
0
0.2
0.4
0.6
0.8
11: 2011−01−01 00:00:00
CM
FC
TO
P
0
0.005
0.01
0.015
CP
RC
ON
0 20 400
1
2
3
4x 10
−3
EN
TR
SC
V
CAULOC0 0.5 1
CMFCTOP0 0.01 0.02
CPRCON
0
0.2
0.4
0.6
0.8
11: 2011−03−31 12:00:00
CM
FC
TO
P
0
0.005
0.01
0.015
CP
RC
ON
0 20 400
1
2
3
4x 10
−3E
NT
RS
CV
CAULOC0 0.5 1
CMFCTOP0 0.01 0.02
CPRCON
M.Laine: Advanced uncertainty evaluation of climate models by Monte Carlo methods 25/27
ECHAM5 validationRMSE variability using default parameters in blue and usingthe EPPES optimized parameter values in red. Independentdata but with the same cost function as in the optimization.
ECHAM5 validation
RMS variability using default parameters in blue and using theEPPES optimised parameter values in red. Independent data butwith the same cost function as in optimization.
0.2
0.3
0.4
0.5
0.6
0.7
0.8
5 6 7 8 9 10 05060708090
100110120
5 6 7 8 9 10 0
2010
jfm
Forecast length (days)
0.20.30.40.50.60.70.8
5 6 7 8 9 10 05060708090
100110120
5 6 7 8 9 10 0
2011
a0.20.30.40.50.60.70.8
5 6 7 8 9 10 0
500hPa geopotential height ACC
5060708090
100110120
5 6 7 8 9 10 0
2011
jfm
Expreriment: l2l3-def 500hPa geopotential height RMSE
M.Laine: Numerical weather prediction model tuning via ensemble prediction system 10/14
500hPa geopotential height RMSE
forecast length (days)
2011, April
optimizedparameters
defaultparameters
about 6 hour increase in the forecast skill
M.Laine: Advanced uncertainty evaluation of climate models by Monte Carlo methods 26/27
Conclusions and ongoing research• MCMC can be applied to climate models with the help of
adaptation and other speed up tricks, but the problem liesin the formulation of the cost function.
• Existing ensemble run infrastructures can be used to inferabout model parameters in NWP and this methodologycan be used with climate models, also.
• MCMC runs, even short ones, are useful to pinpointproblems in parametrizations, formulation of the costfunction, etc.
• Which climate model fields to include in the cost functionand how to scale and weight multi-criteria cost functionterms?
• Can filtering method be used in defining climate modelcost function?
• How well do locally tuned parameter work in longersimulations?
M.Laine: Advanced uncertainty evaluation of climate models by Monte Carlo methods 27/27