Upload
mia-smith
View
220
Download
2
Tags:
Embed Size (px)
Citation preview
Robin HoganRobin Hogan
Ewan O’ConnorEwan O’Connor
University of Reading, UKUniversity of Reading, UK
What is the half-life of a What is the half-life of a cloud forecast?cloud forecast?
Cloud fraction “Bony diagrams”
Winter (Oct-Mar) Summer (Apr-Sep)
EC
MW
F m
odel
C
hilb
olt
on
How good is a forecast?
• Overview of talk– Which skill scores have the most desirable properties?– How does skill depend on spatial scale, lead time etc?– If it has an inverse-exponential decay with forecast lead
time, what is the “half-life” of the forecast?
– Most model comparisons evaluate the cloud climatology
– What about individual forecasts?
– Standard measure shows forecast “half-life” of ~8 days (left)
– But virtually insensitive to clouds!
ECMWF 500-hPa geopotential anomaly correlation
Joint PDFs of cloud fraction
• Raw (1 hr) resolution– 1 year from Murgtal– DWD COSMO model
• 6-hr averaging
ab
cd
…or use a simple contingency table
Desirable properties of skill scores
• Equitable: all random forecasts score zero– This is essential!– Note that forecasting the right climatology versus height but
with no other skill should also score zero
• Proper: not possible to “hedge your bets”– Some scores reward under- or over-prediction (e.g. hit rate)– Jolliffe and Stephenson: not possible to be equitable and proper!
• Independence of how often cloud occurs– Almost all scores asymptote to 0 or 1 for vanishingly rare
events
• Dependence on 10x10 joint PDF, not just 2x2 table– Difference between cloud fraction of 0.9 and 1 is as important
for radiation as a difference between 0 and 0.1
• Linearity: so that can fit an inverse exponential– Some scores (Yule’s Q) “saturate” at the high-skill end
Three quite good scores• 1. Log of odds ratio: LOR=ln(ad/bc)
– Good “properness” properties– Unbounded: a perfect forecast scores infinity!
Generalized skill score = (x-xrandom)/(xperfect-xrandom)– Where “x” is any number derived from the joint PDF– Resulting scores vary linearly from random=0 to perfect=1
• 2. Heidke skill score: x=a+d– Monotonically related to the Equitable Threat Score, but
more linear
• 3. Linear Brier score: x=mean absolute difference– Sensitive to cloud fraction errors in model for all values of
cloud fraction
Score versus lead time, Murgtal 2007
• Both scores well fitted by S=S0exp(-t/t0)– Half life=ln(2)t0
• Met Office NAE has higher scores than DWD COSMO– But apparently a shorter half life (~2.7 days versus ~4.1 days)– Obviously need longer lead-time forecasts to check this!
DWD COSMO versus hours averaged
• Skill and lead time both increase with the number of hours over which cloud fraction is averaged– Larger-scale features are easier to forecast
Met Office versus hours averaged
• Statistics poorer for larger number of hours averaged– Log of odds ratio and Heidke skill score are sensitive to cloud
fraction threshold– Linear Brier score considers all cloud fractions so more robust
Summary• Half-life of a cloud forecast is between 2.5 and 5 days
– Relatively insensitive to skill score (provided a good one is used)– Compare to ~8 days for ECMWF 500-hPa geopotential height
forecast– Skill at forecasting cloud increases somewhat for larger scale
features
• Important to assess the merits of various skill scores– At least 5 criteria to judge against, and none are good on all– Plenty of bad ones to use (hit rate, false-alarm rate etc)!– Worth trying Stephensons’s “Extreme Dependency Score”, which
is good for very rare events
• Wish list– Obtain Met Office cloud forecasts beyond a lead time of 3 days– Compare skill of the Met Office model at different model
resolutions, but averaged to the same scale– Can we see what skill comes from global model at boundaries,
what comes from mesoscale data assimilation etc?
Model cloud
Model clear-sky
A: Cloud hit B: False alarm
C: Miss D: Clear-sky hit
Observed cloud Observed clear-sky
Comparison with Met Officemodel over ChilboltonOctober 2003
Contingency tables
Simple skill score:Hit Rate
• Hit Rate: fraction of forecasts correct = (A+DD)/(A+B+C+DD)– Consider all Cabauw data, 1-9
km– Increase in cloud fraction
threshold causes apparent increase in skill.
• Misleading: fewer cloud events so “skill” is only in predicting clear skies– Models which underestimate cloud will
do better than they should
Met Office short range forecast
Météo France old cloud scheme
More sophisticated scores• Equitable threat score
=(A-E)/(A+B+C-E) where E removes those hits that occurred by chance.
• Yule’s Q =(-1)/(+1) where the odds ratio =ADD/BC.• Advantage: little dependence
on frequency of cloud
– For both scores, 1 = perfect forecast, 0 = random forecast
• From now on use Equitable threat score with threshold of 0.1.
Monthly skill versus time• Measure of the skill of forecasting cloud fraction>0.05
– Comparing models using similar forecast lead time– Compared with the persistence forecast (yesterday’s
measurements)
• Lower skill in summer convective events
Skill versus lead time• Unsurprisingly UK model most accurate in UK,
German model most accurate in Germany!
• Half-life of cloud forecast ~2 days
• More challenging test than 500-hPa geopotential (half-life ~8 days)