Training Course 2009 – NWP-PR: Calibration of EPSs 1/42 Calibration of EPSs Renate Hagedorn European Centre for Medium-Range Weather Forecasts

Training Course 2009 – NWP-PR: Calibration of EPS’s 1/42

Calibration of EPS’s

Renate Hagedorn European Centre for Medium-Range Weather Forecasts


Outline

• Motivation

• Methods

• Training data sets

• Results


Motivation

• EPS forecasts are subject to forecast bias and dispersion errors, i.e.

uncalibrated

• The goal of calibration is to correct for such known model deficiencies,

i.e. to construct predictions with statistical properties similar to the

observations

• A number of statistical methods exist for post-processing ensembles

• Calibration needs a record of prediction-observation pairs

• Calibration is particularly successful at station locations with long

historical data record (-> downscaling)


Calibration methods for EPS’s

• Bias correction

• Multiple implementation of deterministic MOS

• Ensemble dressing

• Bayesian model averaging

• Non-homogenous Gaussian regression

• Logistic regression

• Analog method


Bias correction

• As a simple first order calibration a bias correction can be applied:

• This correction factor is applied to each ensemble member, i.e. spread

is not affected

• Particularly useful/successful at locations with features not resolved by

model and causing significant bias

N

i

N

iii o

Ne

Nc

1 1

11

with: ei = ensemble mean of the ith forecast oi = value of ith observation N = number of observation-forecast pairs


Bias correctionStation: ULAN-UDE (# 30823, Height: 515m) Lead: 120h

-20

-10

0

10

20

T2

m-a

no

ma

ly /

K

06/012005

05/022005

245

249

CL

IM

06/01 13/01 21/01 28/01 05/02245

249

OBSDETEPS


Multiple implementation of det. MOS

• A possible approach for calibrating ensemble predictions is to simply

correct each individual ensemble member according to its

deterministic model output statistic (MOS)

• BUT: this approach is conceptually inappropriate since for longer

lead-times the MOS tends to correct towards climatology

all ensemble members tend towards climatology with longer lead-times decreased spread with longer lead-times in contradiction to increasing uncertainty with increasing lead-times

• Experimental product at

http://www.nws.noaa.gov/mdl/synop/enstxt.htm, but no objective

verification yet…


• Define a probability distribution around each ensemble member

(“dressing”)

• A number of methods exist to find appropriate dressing kernel (“best-

member” dressing, “error” dressing, “second moment constraint”

dressing, etc.)

• Average the resulting nens distributions to obtain final pdf

Ensemble dressing


Ensemble Dressing

• (Gaussian) ensemble dressing calculates the forecast probability for the

quantiles q as:

• key parameter is the standard deviation of the Gaussian dressing kernel

ensn

i D

i

ens

xq

nqvP

1

~1)(

222 11 ens

ens

yxD n

error variance of the ensemble-mean FC

average of the ensemble variancesover the training data

with: Φ = CDF of standard Gaussian distribution xi = bias-corrected ensemble-member~


Bayesian Model Averaging

• BMA closely linked to ensemble dressing

• Differences:

dressing kernels do not need to be the same for all ensemble members

different estimation method for kernels

• Useful for giving different ensemble members (models) different weights:

• Estimation of weights and kernels simultaneously via maximum likelihood, i.e. maximizing the log-likelihood function:

ensn

j e

je

xqw

xqwqvP

21

11

~~)(

with: w1 + we (nens - 1) = 1

N

i

n

jeijieeii

ens

xvgwxvgw1 2

2,

21,111 ),~(),~(ln)ln(

g1, ge = Gaussian PDF’s


BMA: example

90% prediction interval of BMA

OBSsingle model

ensemble members

Ref: Raftery et al., 2005, MWR


BMA: recovered ensemble members

OBS

single modelensemble members

100 equally likely valuesdrawn from BMA PDF

Ref: Raftery et al., 2005, MWR


Non-homogenous Gaussian regression

• In order to account for existing spread-skill relationships we model

the variance of the error term as a function of the ensemble spread sens:

• The parameter a,b,c,d are fit iteratively by minimizing the CRPS of the

training data set

• Interpretation of parameters:

bias & general performance of ens-mean are reflected in a and b

large spread-skill relationship: c ≈ 0.0, d ≈ 1.0

small spread-skill relationship: d ≈ 0.0

• Calibration provides mean and spread of Gaussian distribution (called non-homogenous since variances of regression errors not the same for all values of the predictor, i.e. non-homogenous)

2

)()(

ens

ens

dsc

xbaqqvP


Logistic regression

• Logistic regression is a statistical regression model for Bernoulli-

distributed dependent variables

• P is bound by 0,1 and produces an s-shaped prediction curve

steepness of curve (β1) increases with decreasing spread, leading to

sharper forecasts (more frequent use of extreme probabilities)

parameter β0 corrects for bias, i.e. shifts the s-shaped curve

)exp(1

)exp()(

10

10

ens

ens

x

xqvP


How does logistic regression work?

+ training data 100 cases (EM) height of obs y/n

+ test data (51 members) height of raw prob

event threshold

event observedyes/no (0/1)

-5 0 5ens-mean anomaly

0.0

0.2

0.4

0.6

0.8

1.0pr

obab

ilityfile: logreg_nh096_ev0a0: 0.2844 a1: 1.019

calibrated prob

GP: 51N, 9E, Date: 20050915, Lead: 96h


-5 0 5ens-mean anomaly

0.0

0.2

0.4

0.6

0.8

1.0pr

obab

ilityfile: logreg_nh168_ev1a0: 0.8185 a1: 0.4136

LR-Probability worse!



event threshold


calibrated prob

GP: 51N, 9E, Date: 20050915, Lead: 168h


-4 -2 0 2 4ens-mean anomaly

0.0

0.2

0.4

0.6

0.8

1.0pr

obab

ility

file: logreg_sh168_ev1a0: 2.017 a1: 1.815LR-Probability better!



event threshold


calibrated prob

GP: 15.5S, 149.5W, Date: 20050915, Lead: 168h


Analog method

• Full analog theory assumes a nearly infinite training sample

• Justified under simplifying assumptions: Search only for local analogs Match the ensemble-mean fields Consider only one model forecast variable in selecting analogs

• General procedure: Take the ensemble mean of the forecast to be calibrated and find

the nens closest forecasts to this in the training dataset

Take the corresponding observations to these nens re-forecasts and form a new calibrated ensemble

Construct probability forecasts from this analog ensemble


Analog method

Ref: Hamill & Whitaker, 2006, MWR

Forecast to be calibrated

Closest re-forecasts

Corresponding obs

Probabilities of analog-ens

Verifying observation


Training datasets

• All calibration methods need a training dataset, containing a number of forecast-observation pairs from the past

The more training cases the better The model version used to produce the training dataset should be

as close as possible to the operational model version

• For research applications often only one dataset is used to develop and test the calibration method. In this case cross-validation has to be applied.

• For operational applications one can use: Operational available forecasts from e.g. past 30-40 days Data from a re-forecast dataset covering a larger number of past

forecast dates / years


2009

Feb March Apr

27 28 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 01 02

1971

1972

1973

1974

1975

.

.

.

.

1996

1997

1998

1999

2000

“Perfect” Reforecast Data Set


Early motivating results from Hamill et al., 2004

Bias correctedwith refc data

LR-calibratedensemble

Bias correctedwith 45-d data

Achieved with “perfect”

reforecast system!

Raw ensemble


The 32-day unified VAREPS

• Unified VarEPS/Monthly system enables the production of unified reforecast data set, to be used by:

EFI model climate 10-15 day EPS calibration Monthly forecasts anomalies and verification

• Efficient use of resources (computational and operational)

• “Perfect” reforecast system would produce for every forecast a substantial number of years of reforecast

• “Realistic” reforecast system has to be an optimal compromise between affordability and needs of all three applications


Unified VarEPS/Monthly Reforecasts2009

Feb March Apr

27 28 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 01 02

1991

1992

1993

1994

1995

.

.

.

.

2004

2005

2006

2007

2008


Unified VarEPS/Monthly Reforecasts2009

Feb March Apr

27 28 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 01 02

1991

1992

1993

1994

1995

.

.

.

.

2004

2005

2006

2007

2008


Calibration of medium-range forecasts

• A limited set of reforecasts has been produced for a preliminary assessment of the value of reforecasts for calibrating the medium-range EPS

• Test Reforecast data set consists of: 14 refc cases (01/09/2005 – 01/12/2005) 20 refc years (1982 – 2001) 15 refc ensemble members (1 ctrl. + 14 pert.) Model cycle 29r2, T255, ERA-40 initial conditions

• Used to calibrate the period Sep-Nov 2005 (91 cases)

• Calibrating upper air model fields vs. analysis demonstrated less scope for calibration

• Greater impact for surface variables, in particular at station locations


I. ECMWF forecasts, though better than GFS forecasts, can be improved

through calibration

II. Main improvement through bias correction (60-80%), but when using

advanced methods (e.g. NGR) calibration of spread adds to general

improvements, in particular at early lead times

III. Improvements occur mainly at locations with low skill

IV. Operational training data can be used for short-lead forecasts and/or light

precipitation events, however, reforecasts beneficial at long leads and/or

extremer precipitation events

V. Usually, near-surface multi-model forecasts are better than single model

forecasts, however, reforecast calibrated ECMWF forecasts are competitive

for short lead times and even better than the MM for longer lead times

Main messages


I. ECMWF vs. GFS

Results from cross-validated reforecast data: 14 weekly start dates, Sep-Dec 1982-2001 (280 cases)

Hagedorn et al., 2008



through calibration











Main messages


2m Temperature, stations: 0-249, 2005090100-2005113000

0 2 4 6 8 10Lead time / days

0.0

0.1

0.2

0.3

0.4

0.5

0.6

CR

PS

S

DMOBC NGR

tr_set = REFCnens_rf = 15nyy_tr = 20ndd_tr = 5ncases_tr = 100

II. Bias-correction vs. NGR-calibration

NGR-calibrationBias-correctionDMO

REFC-data: 15 members, 20 years, 5 weeks

2m temperature forecasts (1 Sep – 30 Nov 2005), 250 European stations

CRPSS2m Temperature, pos. anomaly, stations: 0-249, 2005090100-2005113000


0.5

1.0

1.5

2.0

2.5

3.0

RM

SE

& S

PR

EA

D /

K

DMOBC NGR

tr_set = REFCnens_rf = 15nyy_tr = 20ndd_tr = 5ncases_tr = 100

BC-spread = DMO-spread

NGR-spread

RMSE & spread



through calibration











Main messages


III. Individual locations

0.0 0.2 0.4 0.6 0.8 1.00.0

0.2

0.4

0.6

0.8

1.0

0 3 0

0 3 0

60

60

CRPSS: 2m Temperature, pos. anomaly, 2005090100-2005113000, DMO, Lead: 048h

0.0 0.1 0.2 0.3 0.5 0.7

0.0 0.2 0.4 0.6 0.8 1.00.0

0.2

0.4

0.6

0.8

1.0

0 3 0

0 3 0

60

60

CRPSS: 2m Temperature, pos. anomaly, 2005090100-2005113000, NGR, Lead: 048h

0.0 0.1 0.2 0.3 0.5 0.7

CRPSS: 2m-Temperature, Sep-Nov 2005, Lead: 48h

DMO NGR

0.0 0.2 0.4 0.6 0.8 1.00.0

0.2

0.4

0.6

0.8

1.0

0 3 0

0 3 0

60

60

CRPSS: 2m Temperature, pos. anomaly, 2005090100-2005113000, NGR - DMO, Lead: 048h

-0.20 -0.10 -0.02 0.02 0.10 0.20

NGR - DMO



through calibration











Main messages


IV. Operational training data vs REFC data

• Operational training data can give similar benefit for short lead times

• REFC data much more beneficial for longer lead times

ECMWF GFS

Hagedorn et al., 2008


IV. Operational training data vs REFC data

PAR: 167, stations: 0-249, 2005090100-2005113000, CAL: NGR


0.0

0.1

0.2

0.3

0.4

0.5

0.6

CR

PS

S

/var/tmp/neq/bigtmp/eps/refc_400/psfiles/rfdata_sc_P167_TE2005090100-2005113000_RF1982090100-2001120100_L0-249_W5_E05_REFC_pa_N20.asc

/var/tmp/neq/bigtmp/eps/refc_400/psfiles/rfdata_sc_P167_TE2005090100-2005113000_RF2005070100-2005113000_L0-249_W45_E51_OPER_pa_N20.asc

2001055REFC

20055145OPER

PAR: 167, stations: 0-249, 2005090100-2005113000, CAL: BC


0.0

0.1

0.2

0.3

0.4

0.5

CR

PS

S

/var/tmp/neq/bigtmp/eps/refc_400/psfiles/rfdata_sc_P167_TE2005090100-2005113000_RF1982090100-2001120100_L0-249_W5_E05_REFC_pa_N20.asc

/var/tmp/neq/bigtmp/eps/refc_400/psfiles/rfdata_sc_P167_TE2005090100-2005113000_RF2005070100-2005113000_L0-249_W45_E51_OPER_pa_N20.asc

2001055REFC

20055145OPER

NGR calibrated

• NGR calibration method particularly sensitive to available training data

Bias correction only

20y REFC training data45-day operational dataDMO

20y REFC training data45-day operational dataDMO


IV. REFC beneficial for extreme precip

Precipitation > 1mm Precipitation > 10mm

• REFC data much more beneficial for extreme precipitation events

• Daily reforecast data not more beneficial than weekly REFC

Hamill et al., 2008



through calibration











Main messages


V. TIGGE multi-model


-0.2

0.0

0.2

0.4C

RP

SS

Multi-ModelECMWF

Met OfficeNCEP

Solid: no BC

T-2m, 250 European stations2008060100 – 2008073000 (60 cases)



-0.2

0.0

0.2

0.4C

RP

SS


Multi-ModelECMWF

Met OfficeNCEP

Dotted: 30d-BC Solid: no BC




-0.2

0.0

0.2

0.4C

RP

SS


Multi-ModelECMWF

Met OfficeNCEP

Dashed: REFC-NGR Dotted: 30d-BC Solid: no BC



Summary

• The goal of calibration is to correct for known model deficiencies

• A number of statistical methods exist to post-process ensembles

• Every methods has its own strengths and weaknesses

Analog methods seems to be useful when large training dataset available

Logistic regression can be helpful for extreme events not seen so far in training dataset

NGR method useful when strong spread-skill relationship exists, but relative expensive in computational time

• Greatest improvements can be achieved on local station level

• Bias correction constitutes a large contribution for all calibration methods

• ECMWF reforecasts very valuable training dataset for calibration


References and further reading

• Gneiting, T. et al, 2005: Calibrated Probabilistic Forecasting Using Ensemble Model Output Statistics and Minimum CRPS Estimation. Monthly Weather Review, 133, 1098-1118.

• Hagedorn, R, T. M. Hamill, and J. S. Whitaker, 2008: Probabilistic forecast calibration using ECMWF and GFS ensemble forecasts. Part I: 2-meter temperature. Monthly Weather Review, 136, 2608-2619.

• Hamill T.M. et al., 2004: Ensemble Reforcasting: Improving Medium-Range Forecast Skill Using Retrospective Forecasts. Monthly Weather Review, 132, 1434-1447.

• Hamill, T.M. and J.S. Whitaker, 2006: Probabilistic Quantitative Precipitation Forecasts Based on Reforecast Analogs: Theory and Application. Monthly Weather Review, 134, 3209-3229.

• Hamill, T. M., R. Hagedorn, and J. S. Whitaker, 2008: Probabilistic forecast calibration using ECMWF and GFS ensemble forecasts. Part II: precipitation. Monthly Weather Review, 136, 2620-2632.

• Raftery, A.E. et al., 2005: Using Bayesian Model Averaging to Calibrate Forecast Ensembles. Monthly Weather Review, 133, 1155-1174.

• Wilks, D. S., 2006: Comparison of Ensemble-MOS Methods in the Lorenz ’96 Setting. Meteorological Applications, 13, 243-256.

Documents

Training Course 2009 – NWP-PR: Calibration of EPSs 1/42 Calibration of EPSs Renate Hagedorn European Centre for Medium-Range Weather Forecasts