NCAR, 15 April 20081

Fuzzy verification of

fake cases

Beth Ebert

Center for Australian Weather and Climate Research

Bureau of Meteorology

NCAR, 15 April 2008 2

• Look in a space / time neighborhood around the point of interest

– Evaluate using categorical, continuous, probabilistic scores / methods

– Will only consider spatial neighborhood for fake cases

Fuzzy (neighborhood) verification

t

t + 1

t - 1

Forecast value

Fre

qu

en

cy

Forecast value

Fre

qu

en

cy

forecast

observation

NCAR, 15 April 2008 3

Fuzzy verification framework

Fuzzy methods use one of two approaches to compare forecasts and observations:

single observation – neighborhood forecast

(user-oriented)

neighborhood observation – neighborhood forecast

(model-oriented)

observation forecast

observation forecast

NCAR, 15 April 2008 4

Fuzzy verification framework

good performance

poor performance

NCAR, 15 April 2008 5

UpscalingNeighborhood observation - neighborhood forecast

Average the forecast and observations to successively larger grid resolutions, then verify as usual

% change in ETS

Weygandt et al. (2004)

NCAR, 15 April 2008 6

Fractions skill scoreNeighborhood observation - neighborhood forecast

N

i

N

iobsfcst

N

iobsfcst

PP

PP

1 1

22

1

2

N1

N1

)(N1

1FSS

observed forecast

Compare forecast fractions with observed fractions (radar) in a probabilistic way over different sized neighbourhoods

Roberts and Lean (2008)

NCAR, 15 April 2008 7

single threshold

ROC

Spatial multi-event contingency tableSingle observation - neighborhood forecast

Vary decision thresholds:

• magnitude (ex: 1 mm h-1 to 20 mm h-1)

• distance from point of interest (ex: within 10 km, .... , within 100 km)

• timing (ex: within 1 h, ... , within 12 h)

• anything else that may be important in interpreting the forecast

Fuzzy methodology – compute Hanssen and Kuipers score HK = POD – POFD

Measure how close the forecast is to the place / time / magnitude of interest.

Atger (2001)

NCAR, 15 April 2008 8

Practically perfect hindcasts Single observation - neighborhood forecast

Q: If the forecaster had all of the observations in advance, what would the "practically perfect" forecast look like?

– Apply a smoothing function to the observations to get probability contours, choose yes/no threshold that maximizes CSI when verified against obs

– Did the actual forecast look like the practically perfect forecast?

– How did the performance of the actual forecast compare to the performance of the practically perfect forecast?

Fuzzy methodology – compute

forecast PracPerf

CSIforecast = 0.34 CSIPracPerf = 0.48

PracPerf

forecast

ETS

ETS

Kay and Brooks (2000)

NCAR, 15 April 2008 9

1st geometric case50 pts to the right

bad

good

12.7 mm

25.4 mm

NCAR, 15 April 2008 10

2nd geometric case200 pts to the right

bad

good

NCAR, 15 April 2008 11

5th geometric case125 pts to the right and huge

bad

good

NCAR, 15 April 2008 12

1st case vs. 5th case

~same

Case 1better

Case 5better

NCAR, 15 April 2008 13

Perturbed cases

1000

km

"Observed"

(6) Shift 12 pts right, 20 pts down, intensity*1.5

(4) Shift 24 pts right, 40 pts down

Which forecast is better?

NCAR, 15 April 2008 14

4th perturbed case24 pts right, 40 pts down

bad

good

NCAR, 15 April 2008 15

6th perturbed case12 pts right, 20 pts down, intensity*1.5

bad

good

NCAR, 15 April 2008 16

Difference between cases 6 and 4Case 4 - Shift 24 pts right, 40 pts down

Case 6 - Shift 12 pts right, 20 pts down, intensity*1.5

Case 6 – Case 46

4

NCAR, 15 April 2008 17

How do fuzzy results for shift + amplification compare to results for the case of shifting only?

Case 6 - Shift 12 pts right, 20 pts down, intensity*1.5

Case 3 - Shift 12 pts right, 20 pts down, no intensity change

Case 6 – Case 3

3

Why does the case with incorrect amplitude sometimes perform better??Baldwin and Kain (2005): When the forecast is offset from the observations most scores can be improved by overestimating rain area, provided rain is less common than "no rain".

6

NCAR, 15 April 2008 18

Some observations about methods

Traditional• Measures direct

correspondence of forecast and observed values at grid scale

• Hard to score well unless forecast is ~perfect

• Requires overlap of forecasts and obs

Entity-based (CRA)• Measures location

error and properties of blobs (size, mean/max intensity, etc.)

• Scores well if forecast looks similar to observations

• Does not require much overlap to score well

Fuzzy• Measures scale- and

intensity-dependent similarity of forecast to observations

• Forecast can score well at some scales and not at others

• Does not require overlap to score well

NCAR, 15 April 2008 19

Some final thoughts…

Object-based and fuzzy verification seem to have different aims

Object-based methods• Focus on describing the error

• What is the error in this forecast?

• What is the cause of this error (wrong location, wrong size, wrong intensity, etc.)?

Fuzzy neighborhood methods• Focus on skill quantification

• What is the forecast skill at small scales? Large scales? Low/high intensities?

• What scales and intensities have reasonable skill?

• Different fuzzy methods emphasize different aspects of skill

NCAR, 15 April 2008 20

Some final thoughts…

When can each type of method be used?

Object-based methods• When rain blobs are well defined (organized systems, longer

rain accumulations)

• When it is important to measure how well the forecast predicts the properties of systems

• When size of domain >> size of rain systems

Fuzzy neighborhood methods• Whenever high density observations are available over a

reasonable domain

• When knowing scale- and intensity-dependent skill is important

• When comparing forecasts at different resolutions