1. Introduction

c) Probit Model

Communication of Uncertainty in Weather ForecastsPricilla Marimo, Todd R. Kaplan, Ken Mylne and Martin Sharpe

Format A Format B

Providing probabilistic weather information to users has the potential to improve decision making. Weather is uncertain due to the chaotic and complex nature of the atmosphere, hence the need to communicate this uncertainty information. According to (NRC 2006), a forecast is therefore incomplete if uncertainty information is not included. The Met Office, through the use of ensemble forecasting and other techniques is capable of providing probabilistic estimates of weather forecasts. Including such information, although possibly valuable to users, can increase the complexity and amount of information being conveyed.  Studies that have been done to assess decision making when provided with probabilistic weather information, have concluded that on average, participants who were given uncertainty information made significantly better decisions than those without (Roulston et al. 2006; Roulston and Kaplan 2009). Still questions arise on whether or not the presentation format makes a difference in interpretation and understanding. This study follows the same approach that was used by (Roulston and Kaplan 2009). Their study tested the ability of subjects to understand the information in a fan chart format for expressing uncertainty in 5-day temperature forecasts.

In our study, we test a bar graph format and a table format transition using experimental economics lab techniques. The Met Office Public Weather Service (PWS) is constantly developing new products for disseminating weather information to users. After public consultation, they have a new format for presenting probabilistic forecast information for use on the Met office website. Different presentation formats/designs can be used to illustrate the same data in various fields. However, the way that information is presented and consequently how we interpret or process it has the potential to influence decision making.

2. Method

Question 4: Would you prefer to receive £0.50 if...A: The maximum temperature on Saturday is above 5 ºC OR if

B: The maximum temperature on Wednesday is above 8 ºC

A total of 289 undergraduate students from various disciplines at the University of Exeter were recruited to participate in the experimental sessions. The sessions were computer based and took place in the Finance and Economics Experimental Laboratory (FEELE) at the University of Exeter. Participants were presented with a set of 20 “lotteries” based on the maximum temperature up to five days ahead and asked to chose the ‘most likely’ outcome. If true statement was chosen, participants were rewarded with £0.50. Participants were divided into three treatment groups: A, B and C. The 5-day temperature forecast information was presented as follows: Group A: Table with a point forecast , Group B: Table with point forecast and uncertainty information , Group C: Bar graph with point forecast and uncertainty information. The same graphs were shown for all the participants in a particular group but in randomised order Four question orders were used :

1st order: 1, 2,..., 20 2nd order: 20, 19,..., 13rd order: 11, 12,..., 20, 1, 2,..., 104th order: 10, 9,..., 1, 20, 19,..., 11

This was done to test speed of learning differences between the different presentation formats. An example of a lottery is shown below. After each lottery students were informed of the actual temperatures and whether either of the criteria had been satisfied. At the end of the experiment students were paid their lottery winnings in addition to a £3.00 payment for participating.

Discipline Gender Format A Format B Format C

HumanitiesMale 9 31 27Female 12 14 14

Business/EconMale 16 12 29Female 19 15 35

Science/ EngMale 20 7 -Female 11 18 -

Marginal effects were also computed and are shown as the percentage change in probability. For instance, male participants were 2% more likely to get a question correct.

Participants were 32% less likely to get “swing” questions correct; 74% less likely if they had no uncertainty information

Getting probability or test question wrong would decrease chances Uncertainty information increased chances of choosing most probableBiased towards picking the later option (early day correct) ; if format C increased

furtherArea and Length variablesArea: 1= questions where the greatest area between the high/low range & the asked

temperature does not get the correct answer for Group C participants for questions 1,12 and 13

Length: 1= questions where the greatest length between the high/low range & the asked temperature does not get the correct answer for Group B participants for questions 1, 12 and 13.

e.g. Round 12 of 20 ( shown below)Statement A - The maximum temperature on Saturday is above 13 deg. CStatement B - The maximum temperature on Wednesday is above 12 deg. C

It is possible that some did not understand or could not use the uncertainty information correctly- chose option with biggest area

NRC. 2006. Completing the Forecast: Characterizing and Communicating Uncertainty for Better Decisions Using Weather and Climate Forecasts Edited by Committee on Estimating and Communicating Uncertainty in Weather and Climate Forecasts. Washington D.C: National Academies Press.Roulston, M.S, G.E. Bolton, A.N. Kleit, and A.L. Sears-Collins. 2006. A Laboratory Study of the Benefits of Including Uncertainty Information in Weather Forecasts. Weather and Forecasting 21: 116-122.Roulston, Mark S., and Todd R. Kaplan. 2009. A laboratory-based study of understanding of uncertainty in 5-day site-specific temperature forecasts. Meteorological Applications 16 (2): 237-244.

On average participants who were given uncertainty information made significantly better decisions than those without. Both table and the graph with uncertainty information were significant determinants of choosing the most probable outcome, though the graph is a stronger predictor compared to the table. There was a learning effect as experiment progressed. Graph with uncertainty information took on average less response time compared to those who were shown a table with uncertainty information. Analysis with the general public (underway at the Met Office)

Pricilla MarimoSchool of Business and EconomicsUniversity of Exeter Streatham Court, Rennes Drive, Exeter, EX4 4PU, UKEmail: pm272@ex.ac.uk

Ken MylneMet Office FitzRoy Road Exeter, EX1 3PB, UKEmail:ken.mylne@metoffice.gov.uk

Of the 20 lotteries, eight were easy questions and the remaining were equally divided between hard and swing. Question 4 above was a “swing” question and these were:

Questions in which a hypothetical participant with just a point forecast assuming mode=median and same uncertainty for all forecasts would make a different decision to someone with uncertainty information (i.e. Format B or C)

Format C

3. Results

£7.30 £7.45

% correct (Format A)

% correct (Format B)

% correct (Format C)

Business 64.9 80.0 75.6Humanities 66.4 76.1 74.3Sciences 67.6 77.5 -Female 64.9 75.0 72.5Male 67.6 80.2 78.2Swing questions 18.0 57.9 59.7Non swing questions 86.9 86.0 81.7Order 1 66.6 79.2 75.0Order 2 67.4 79.6 78.0Order 3 62.0 76.7 75.5Order 4 70.0 75.3 70.7Overall 66.2 77.6 75.1

£6.33

£6.33

Pr(correct|predictors) coefficient predictori ii

Statistical analysis to estimate the determinants of choosing the most probable outcome was done using a probit regression model. The model predicts the probability that a participant will answer a question correctly as a function of a collection of predictor variables.

Where Φ is the cumulative distribution function of the standard normal distribution. The list of predictors with the corresponding coefficients found by fitting the model are listed in the table below.

*** sig at 1% level ** sig at 5% level * sig at 10% level

b)Time Analysis

More likely to choose statement B because of the big area above 12 deg. C . Most likely outcome however is A.

Percent correct:Format A – 80.5Format B – 66.0Format C – 49.5

General decrease in time as experiment progressed

Average earnings

4. Conclusion

A summary of the numbers of participants in the different treatment groups is shown in the table on the left

a) Summary statistics

A “correct” response was one in which the participant chose the most probable outcome.On average, participants who were provided with format B or C outperformed those with Format A.

5. References

Predictor Coeff P-Value % change in probabilityRound number 0.0045 0.195 0.14Swing question -0.9611*** 0.000 -32.28Swing question & Format A -1.1290*** 0.000 -41.40Swing question & Format C 0.1506 0.144 4.40Male 0.0720* 0.084 2.20English is first language 0.0524 0.316 1.61Checks internet for weather forecast 0.0042 0.963 0.13Length -0.4162*** 0.000 -14.25Area -0.5460*** 0.000 -19.16Checks weather at least every 2-3 days -0.0528 0.211 -1.62Sample question mistake -0.3607*** 0.000 -12.21Die question mistake -0.1971*** 0.000 -6.20Format B 0.2498* 0.067 7.39Format C 0.3633*** 0.007 10.67Order 1 0.2229** 0.043 6.57Order 1 & Format C -0.2841** 0.036 -9.33Order 1 & Format B -0.0561 0.689 -1.74Order 2 0.1822 0.116 5.39Order 2 & Format B -0.0575 0.693 -1.79Order 2 & Format C -0.1773 0.207 -5.68Order 4 0.3017** 0.026 8.63Order 4 & Format B -0.3968** 0.013 -13.38Order 4 & Format C -0.5807*** 0.000 -20.40Early day correct -0.3565*** 0.000 -11.14Early date correct & Format C -0.2973*** 0.002 -9.73Early date correct & Format A -0.1262 0.238 -3.99Early date correct & Order 1 0.1048 0.333 3.10Early date correct & Order3 -0.0403 0.718 -1.25Early date correct & Order4 0.2734** 0.015 7.64Above & certain -0.1595** 0.039 -5.06Above & uncertainty -0.3156*** 0.000 -9.87Test question dummy 0.0321 0.643 0.97Business 0.0079 0.909 0.24Humanities -0.0121 0.852 -0.37New format B -0.0450 0.661 -1.39Hard question 0.0862 0.149 2.60Constant 1.1657 0.000 0.00

participants took more time on the swing questions compared to the other question types despite the format shown or order except for Group B participants with order 1, where the easy took more time. For the easy and hard questions, participants took almost the same median time for Order 2, 3 and 4.