Bureau of Meteorology Advisory Board - Weather and Climate · Trends in the Skill of Weather Prediction at Lead Times of 1 to 14 Days: Thu 4-Sep-2014 (Harvey Stern & Noel Davidson)

Trends in the Skill of Weather Prediction at Lead Times of 1 to 14 Days: Thu 4-Sep-2014 (Harvey Stern & Noel Davidson)

1

Trends in the Skill of Weather Prediction

at Lead Times of 1 to 14 Days

Harvey Stern

School of Earth Sciences, University of Melbourne, Parkville, Australia

(formerly Bureau of Meteorology, Melbourne, Australia)

& Noel E. Davidson

Centre for Australian Weather and Climate Research,

Bureau of Meteorology, Melbourne, Australia


2

About The Authors

Harvey Stern



& Noel E. Davidson




3

Acknowledgements

The authors gratefully acknowledge the encouragement from Professor

Neville Nicholls of Monash University regarding this work, and also the

thoughtful comments of Professor Ian Simmonds and Associate Professor

Kevin Walsh of the University of Melbourne.

The constructive feedback from internal BoM reviewers, Beth Ebert, Tony

Bannister, Geoff Feren, and Tim Hume, and other colleagues was most

valuable.


4

Summary: Part 1

Unique, multi-year data sets of weather observations and forecasts are used

to document trends in weather forecast accuracy, and the current level of

forecast skill, for Melbourne, Australia.

The data are applied to define prediction skill out to Day-14 for maximum

and minimum temperature, precipitation amount, and probability of

precipitation.

The accuracy of the current official Day 5-7 forecasts is similar to that of

Day-1 forecasts from 50 years ago, whilst the quality of experimental Day 8-

10 forecasts is comparable to that of the Day 5-7 forecasts, when they were

first officially provided 15 years ago.


5

Summary: Part 2

Some overall skill, albeit limited, is evident out to Day-14 and significance

testing indicates that it is most unlikely that this skill arose by chance.

The results provide evidence of deterministic weather forecast skill out to

the hypothesised 15-day limit on such predictions.

However, in so doing, the results raise the possibility that the limit may be

breached at some stage in the future.


6

Location Map


7

ECMWF Model Accuracy


8

GFS Model Accuracy


9

Melbourne Maximum Temperature

Forecast Accuracy


10

Melbourne Minimum Temperature

Forecast Accuracy


11

Melbourne Precipitation Amount

Forecast Accuracy


12

Melbourne Experimental Week-2

Maximum Temperature Forecast Accuracy


13


Minimum Temperature Forecast Accuracy


14


Precipitation Amount Forecast Accuracy


15


Precip’n Probability Forecast Accuracy


16


All Elements Combined Accuracy


17


All Lead Times Combined Accuracy


18


All Lead Times Combined Accuracy


19

Establishing Predictability Limits

The five-year Day 1-14 component of the forecast data base (February

2009 to January 2014) is compared with a corresponding set of randomly

generated predictions in order to establish how confident one could be that

the skill displayed by the experimental predictions did not arise by chance.


20

Melbourne Randomly Generated



21




22




23




24

Melbourne Experimental Max Temp

Forecast Accuracy & Confidence Limits


25

Melbourne Experimental Min Temp



26

Melbourne Experimental Precip Amt



27

Melbourne Experimental Precip Prob



28

Melbourne Experimental Overall (all elements)



29

Conclusion: Part 1

Documentation is provided of improvement in weather forecasting for

Melbourne, Australia, via analyses of several unique multi-year data sets of

operational, NWP model and experimental forecasts, and associated

weather observations, which may be regarded as a benchmark against

which future increases in extended day-to-day weather forecasting skill

might be measured.

It is found that the skill at forecasting five to seven days in advance is

currently comparable with that displayed 50 years ago by forecasts one day

in advance.

Furthermore, the skill at forecasting eight to ten days in advance (the

experimental forecasts) is currently comparable with that displayed by

forecasts five to seven days in advance, when they were first officially

provided 15 years ago.


30

Conclusion: Part 2

An analysis of trends in the accuracy of forecasts indicates that, although

skill is generally increasing at nearly all lead times, there exist periods when

it decreases.

It is suggested that the increases in skill during recent times may be

attributed to the corresponding improvement in the NWP model guidance

and that the observed fluctuations in the skill may be a consequence of

corresponding variations in the frequency of synoptic-scale and broad-scale

drivers of Australia’s weather and climate.


31

Conclusion: Part 3

The work of Clemen (1989) suggests that forecast accuracy can be

improved through combining multiple individual forecasts automatically via

software, and such a piece of software was utilised to generate day-to-day

predictions out Day-14.

This operation of this software has yielded, in real time, a multi-year data

base of experimental predictions (2005 to 2014), initially for Days 1-7, then

extended to Day-10 (from late 2006) and to Day-14 (from early 2009).

Comparison of the accuracy of the experimental predictions with the

accuracy of corresponding official predictions reveals that, whilst significant

overall improvement in accuracy was achieved through application of the

software during the first 12 months of the trial, much less overall

improvement was achieved more recently, possibly due to the operational

implementation and development of forecast guidance systems that also

combine predictions from various sources.


32

Conclusion: Part 4

The work of Lorenz suggested that there may be a 15-day limit on day-to-

day predictability and this presentation reports on work to establish what

the current limit is for Melbourne.

The five-year Day 1-14 component of the forecast data base (February

2009 to January 2014) is compared with a corresponding set of randomly

generated predictions in order to establish how confident one could be that

the skill displayed by the experimental predictions did not arise by chance.

It is found that the skill displayed at forecasting maximum temperature is

significant at the 1:1,000,000 level out to Day-14 (that is, there is less than a

one in one million chance that the skill displayed arose by chance)

Out to Day-13 for minimum temperature,

Out to Day-11 for precipitation amount, and,

Out to Day-14 for precipitation probability.


33

Conclusion: Part 5

The actual level of skill displayed by the Week-2 forecasts is not great.

However, in an environment where the management of weather risk is

critical, such predictions have a useful planning application (the BoM has,

for example, for many years provided forecasts to authorities involved in

power generation), and financial market products such as weather

derivatives allow the potential for loss to be ameliorated.

The results, in providing evidence of forecast skill almost out to Lorenz’s

suggested 15-day limit on such predictions, lends support for the existence of

that limit.




34

Summary: Part 1

Unique, multi-year data sets of weather observations and forecasts are used

to document trends in weather forecast accuracy, and the current level of

forecast skill, for Melbourne, Australia.

The data are applied to define prediction skill out to Day-14 for maximum

and minimum temperature, precipitation amount, and probability of

precipitation.

The accuracy of the current official Day 5-7 forecasts is similar to that of

Day-1 forecasts from 50 years ago, whilst the quality of experimental Day 8-

10 forecasts is comparable to that of the Day 5-7 forecasts, when they were

first officially provided 15 years ago.


35

Summary: Part 2

Some overall skill, albeit limited, is evident out to Day-14 and significance

testing indicates that it is most unlikely that this skill arose by chance.

The results provide evidence of deterministic weather forecast skill out to

the hypothesised 15-day limit on such predictions.




36

Future Work

• Adapting the system described here to run over different locations and to use different

input global NWP data - the GFS is providing very good guidance but not the best guidance.

• It would be interesting to establish the level of skill at the outer limit of forecasting

capability for different locations and using different input global NWP data.

• Applying an algorithm to interpret the official worded forecast in terms of precipitation

probability.

• Verification of forecasts of high impact weather events, a study of major forecast errors,

evaluation of the system’s performance at identifying differences in weather at the current

Melbourne Central Business District (CBD) site and a new observation site just outside the

CBD, and an extension of the methodology and significance testing to forecasts for other

Australian capital cities.

• A study how fluctuations in skill relate to climate drivers and synoptic regimes.

• Extension of the experimental forecasting system, and its verification, to lead times beyond

Day-14 to clarifying whether or not Lorenz’s suggested 15-day limit to day-to-day weather

forecasting capability has already been breached.


37

Acknowledgements

The authors gratefully acknowledge the encouragement from Professor

Neville Nicholls of Monash University regarding this work, and also the

thoughtful comments of Professor Ian Simmonds and Associate Professor

Kevin Walsh of the University of Melbourne.

The constructive feedback from internal BoM reviewers, Beth Ebert, Tony

Bannister, Geoff Feren, and Tim Hume, and other colleagues was most

valuable.

38

Thank You





39



Harvey Stern



& Noel E. Davidson



Documents

Bureau of Meteorology Advisory Board - Weather and Climate · Trends in the Skill of Weather Prediction at Lead Times of 1 to 14 Days: Thu 4-Sep-2014 (Harvey Stern & Noel Davidson)