Huug van den Dool / Dave Unger

Consolidation of Multi-Method Seasonal Forecasts at CPC.

Part I

LAST YEAR / THIS YEAR:• Last Year: Ridge Regression as a Consolidation

Method, to yield, potentially, non-equal weights.• This Year: see new Posters on Consolidation by -) Malaquias Pena (methodological twists and

SST application) and -) Peitao Peng (application to US T&P)

• Last Year: Conversion to pdf as per “Kernel method” (Dave Unger).

This Year: Time series approach is next.

Does the NCEP CFS add to the skill of

the European DEMETER-3 to produce a viable

International Multi Model Ensemble (IMME) ?

“Much depends on which question we ask”

Input by Suranjana Saha and Ake Johansson is acknowledged.

DATA and DEFINITIONS USED

• DEMETER-3 (DEM3) = ECMWF + METFR + UKMO

• CFS

• IMME = DEM3 + CFS

• 1981 – 2001

• 4 Initial condition months : Feb, May, Aug and Nov

• Leads 1-5

• Monthly means

DATA/Definitions USED (cont)

• Deterministic : Anomaly Correlation

• Probabilistic : Brier Score (BS) and Rank Probability Score (RPS)

• Ensemble Mean and PDF

• T2m and Prate

• Europe and United States

Verification Data :• T2m : Fan and Van den Dool

• Prate : CMAP

BRIER SCORE FOR 3-CLASS SYSTEM

1. Calculate tercile boundaries from observations 1981-2001 (1982-2002 for longer leads) at each gridpoint.

2. Assign departures from model’s own climatology (based on 21 years, all members) to one of the three classes: Below (B), Normal (N) and Above (A), and find the fraction of forecasts (F) among all participating ensemble members for these classes denoted by FB, FN and FA respectively, such that FB+ FN+FA=1 .

3. Denoting Observations as O, we calculate a Brier Score (BS) as :BS={(FB-OB)**2 +(FN-ON)**2 + (FA-OA)**2}/3, aggregated over all years and all grid points.

{{For example, when the observation is in the B class, we have (1,0,0) for (OB, ON, OA) etc.}}

4. BS for random deterministic prediction: 0.444 BS for ‘always climatology’ (1/3rd,1/3rd,1/3rd) : 0.222

5. RPS: The same as Brier Score, but for cumulative distribution (no-skill=0.148)

Number of times IMME improves upon DEM-3 :

out of 20 cases (4 IC’s x 5 leads): Region EUROPE EUROPE USA USA

Variable T2m Prate T2m Prate

AnomalyCorrelation

9 14 14 14

Brier Score

16 18.5 19 20

RPS 14 15 19.5 20

“The bottom line” “ NO consolidation, equal weights, NO Cross-validation”

Method CFS alone IMME CON4

No CV 29% 33% 38%

CV-3R

Aspect to be CV-ed

Systematic error in mean

Systematic error in mean

Systematic error in mean and weights

Anom.Corr US T2m February 1982-2002 lead 3 November start

Method CFS alone IMME CON4

No CV 29% 33% 38%

CV-3R 20% 18% 21%

Aspect to be CV-ed

Systematic error in mean

Systematic error in mean

Systematic error in mean and weights

Anom.Corr US T2m February 1982-2002 lead 3 November start

Cross Validation (CV)• Why do we need CV?• Which aspects are CV- ed: a) systematic error

correction ( i) the mean and ii) the stand.dev) and b) weights generated by Consolidation

• How?? CV-1, CV-3, CV-3R• Don’t use CV-1!. CV-1 malfunctions for

systematic error correction in combination with internal* climatology and suffers from degeneracy when weights generated by Consolidation are to be CV-ed.

• *Define internal and external climatology

Raw Nino3.4 SST Correlation SkillAnnual Mean 1981-2001

0

20

40

60

80

100

1 2 3 4 5Forecast Lead [ months ]

Ano

mal

y C

orre

latio

n [ %

] CFS

ECM

MFR

MPI

UKM

ING

LOD

CER

CA

wrt OIv2 1971-2000 climatology

Last year

Bias Corrected Nino3.4 SST Correlation SkillAnnual Mean 1981-2001

0

20

40

60

80

100

1 2 3 4 5Forecast Lead [ months ]

Ano

mal

y C

orre

latio

n [ %

] CFS

ECM

MFR

MPI

UKM

ING

LOD

CER

CA

wrt OIv2 1971-2000 climatology

Last year

nov lead 3 check .35 .04 .27 .35 11 3 1981 left out (and 2 others) nov lead 3 check .36 .00 .29 .35 11 3 1982 left out (and 2 others) nov lead 3 check .29 .02 .33 .36 11 3 nov lead 3 check .40 .04 .31 .36 11 3 nov lead 3 check .38 -.01 .28 .34 11 3 nov lead 3 check .37 .00 .27 .33 11 3 nov lead 3 check .35 -.01 .25 .40 11 3 nov lead 3 check .31 .01 .29 .43 11 3 nov lead 3 check .28 .01 .31 .39 11 3 nov lead 3 check .38 .02 .30 .32 11 3 nov lead 3 check .36 .00 .39 .32 11 3 nov lead 3 check .31 .03 .29 .39 11 3 nov lead 3 check .45 -.01 .23 .35 11 3 nov lead 3 check .37 .00 .31 .41 11 3 nov lead 3 check .35 .03 .28 .40 11 3 nov lead 3 check .33 .02 .36 .35 11 3 nov lead 3 check .42 .01 .33 .31 11 3 nov lead 3 check .33 .04 .31 .42 11 3 nov lead 3 check .33 .00 .29 .35 11 3 nov lead 3 check .40 .02 .31 .35 11 3 nov lead 3 check .33 .00 .24 .38 11 3 2001 left out (and 2 others)

w1 w2 w3 w4

Feb forecast has high co-linearity. Model 2 has high –ve weights for unconstrained regression

Overriding conclusion

With only 20+ years of hindcasts it is hard for any consolidation to be much better than equal weight MME. (Give us 5000 years.)

‘Pooling’ data helps stabilize weights and it increases skill, but is it enough?

20+ years is a problem even for CV-ed systematic error correction.

Further points of study

• The nature of climatology (= control in verification), external, internal, fixed

• Cross Validation method not settled• The Many Details of Consolidation as per

Ridge Regression• Conversion to pdf can be done in very

many different ways (including 3-class BS minimization, logistic regression, ‘count method’, Kernels)

Forecast Consolidation at CPC – Part 2

Ensembles to Probabilities

David Unger / Huug van den Dool

Acknowledgements: Dan Collins, Malaquias Pena, Peitao Peng

Objectives• Produce a single probabilistic forecast from

many tools- Single value estimates

- Ensemble sets• Utilize Individual ensemble members - Assume individual forecasts represent possible

realizations - We want more than just the ensemble mean• Provide Standardized Probabilistic output

- More than just a 3-class forecast

Kernel Characteristics

Kernel Characteristics Unequal Weighting

Ensemble Regression• A regression model designed for the kernel smoothing

methodology - Each member is equally likely to occur - Each has the same conditional error distribution in the event it is closest to the truth. F= Forecast, σF = Forecast Standard Deviation Obs=Observations, σObs= Standard Deviation of observations R=Correlation between individual ensemble members and the

observations Rm = Correlation between ensemble mean and observationsa1 , a0 = Regression Coefficients,

F = a0 + a1 F

a RRm Obs

F1

2

a O bs a F0 1

bRRm 12 2( )

Time series estimation

• Moving Average, Let X11 Be the 10-year running mean known on year 11. N=10

X11 = 1/N(x1+x2+x3+x4+x5+x6+x7+x9+x9+x10)

X12 = X11 + 1/N(x11-x1)

XY+1 = XY + 1/N(xY+1-xY-10)

• Exponential Moving Average (EMA), α = 1/NX12 = X11 + α(x11- X11)

XY+1 = (1- α)XY + αxY+1

Adaptive Ensemble Regression

• F• F2

• (Obs)• (Obs)2

• F (Obs)• Fm2

• (F-Fm)2

R F Obs F O bsF F Obs O bs

( ) ( )( )( ) ( )2 2 2 2

R F Obs F ObsFm F Obs O bs

m ( ) ( )( )

( ) ( )2 2 2 2

EMA estimates

Trends

• Adaptive Regression “learns” recent bias, and is very good in compensating.

• Most statistical tools also “learn” bias, and adapt to compensate.

Steps need to be taken to prevent doubling bias corrections.

Trends (Continued)• Step 1. Detrend all models and Obs. F = F – F10 Obs = Obs – Obs10 F10 , Obs10 = The EMA approximating a 10-year mean

• Step 2. Ensemble Regression Final forecast set, F are anomalies.

• Step 3. Restore the forecast. A) F = F + F10 We believe the OCN trend estimate

B) F = F + C30: C30 = 30-year (1971-2000) Climatology We have no trust in OCN.

C) F = F +C30+ROCN (F10 –C30) : ROCN= Correlation ( F10,Obs) Trust but verify.

Weighting

• The chances of an individual ensemble member being “best” increases with the skill of the model.

• The kernel distribution represents the expected error distribution of a correct forecast.

Final Forecast

• Consolidated Probabilities are the area under the PDF within each of three (Below, Near, Above median ) categories.

Call for ABOVE when the P(above)>36% and P(Below) <33.3%

Call for BELOW when P(Below > 36%) and P(Above) < 33.3%

White area = Equal Chances (We don’t as yet trust our Near Normal percentages)

Performance

Tools 1995 – 2005.• CFS – 15 members hindcast All Members weighted equally with combined area equal to the

tool weighting

• CCA - Single Valued Forecast Hindcasts from Cross Validation

• SMLR - Single valued forecast – Hindcasts from Retroactive Real-time Validation

• OCN incorporated with EMA rather than 10-year box car average.

Performance (Continued)

• First Guess EMA parameters provided by CCA, SMLR Statistics 1956-1980.

• CFS spinup 1981-1994 Validation Period All Initial times, 1-month

lead, Jan 1995-Dec 2005 (11 Years)

Performance (Continued)

• Official Forecast: Hand-drawn, probabilities in 3 classes. PoE obtained from Normal distribution, Standard deviation based on tool skills)

• CCA+SMLR: A Consolidation of the two Statistical Forecasts, equally Weighted.

• CFS: A Consolidation of 15 CFS ensembles, equally weighted.• CFS+CCA+SMLR Wts: A Consolidation of CCA, CFS, and

SMLR, weighted by R/(1-R) for each of the three tools. 15 CFS members each given 1/15th of the CFS weight. Also known as “All”

• All – Equal Wts: CCA, SMLR and the 15 CFS members combined are given equal Weights.

• All – No Trend. Anomalies applied to 30-year mean.

Performance

.046 .076 .191 -.147 63%

.067 .076 .162 -.334 59%

.063 .100 .215 -.268 73%

.005 -.002 .058 -.876 47%

.074 .100 .199 -.203 62%

.023 .040 .098 -.858 38%

CCA+SMLR

CFS

CFS+CCA+SMLR, Wts.

All – No Trend

All – Equal Wts.

Official

HSSCRPSS RPSS - 3 % CoverBias (C)

.074

.046

.067 .023

.102 .067

.081 .044

.182 .119

.177 .097

-.001 -.023-.012 .026

CFS

CCA+SMLR

Official

CRPS Skill Scores: Temperature

1995 – 2005

.015 -.004

.016 -.007

.055 .025

.067 -.001

.092 .076

.086 .042

.058 .035

.062 .009

.218 .195

.202 .057

.052 .021

.020 .032

-.023 -.085.015 .025

.110 .083

.081 .061

.092 .078

.101 .054

High

Moderate

Low

None

Skill

.10

.05

.01

1-Month Lead, All initial times

All

.074

.005

62% 45%

.102 -.02779% 61%

.182 -.04698% 70%

-.001 -.00754% 44%

%cover

No TrendsTrends

CRPS Skill Scores / % cover: Temperature

1995 – 2005

.015 .00047% 32%

.055 -.00138% 26%

.092 .03151% 39%

.058 .00542% 32%

.218 .07488% 63%

.052 .01970% 56%

-.023 -.09779% 63%

.110 -.00581% 64%

.092 .055

.71% 57%

High

Moderate

Low

None

Skill

.10

.05

.01

1-Month Lead, All initial timesCRPSS

The Winter ForecastSkill Weighting Equal Weighting

Conclusions

• Weighting is not critical – within reason.• Consolidation outperforms component

models .• Getting the trends right is essential. • CFS + Trend consolidation provides an

accurate continuous distribution.