UNDERSTANDING INDIVIDUAL AND ENSEMBLE FORECAST BEHAVIOR ... L. Jensen.pdf · UNDERSTANDING INDIVIDUAL AND ENSEMBLE FORECAST BEHAVIOR USING ... P50/P90 Int: Provides . objective

UNDERSTANDING INDIVIDUAL AND ENSEMBLE FORECAST BEHAVIOR USING THE METHOD FOR OBJECT BASED DIAGNOSTIC EVALUATION (MODE)

Tara L. Jensen, Barbara Brown, John Halley Gotway, Tressa Fowler and Randy Bullock National Center for Atmospheric Research, Boulder Colorado, USA and Developmental Testbed Center, Boulder Colorado, USA

6th WMO International Verification Methods Workshop, 17-19 March 2014 ° New Dehli, India

Contributions from Marion Mittermeyer (UKMet Office), Wallace Clark (NOAA/Physical Science Division, Edward Tollerud (NOAA/Global Science Division and Adam Clark (NOAA/National Severe Storms Laboratory)

OVERVIEW • How MODE Works

• Evaluating individual model performance • Examples and results

• Diagnosing ensemble performance • New research

MODEL EVALUATION TOOLS (MET)

Gridded NetCDF

Gridded GRIB Input:

Observation

Analyses

Model Forecasts

PrepBufr Point Obs

STAT

NetCDF Point Obs

ASCII NetCDF

PS

STAT ASCII

NetCDF

Input Reformat Statistics

= optional

ASCII

ASCII Point Obs

Wavelet Stat

STAT ASCII

NetCDF PS

NetCDF Mask

Analysis

MODE

Grid Stat

Ensemble Stat

Point Stat

MODE Analysis

Stat Analysis

PCP Combine

Gen Poly Mask

STAT ASCII

NetCDF

ASCII

MADIS Point Obs

ASCII2NC

PB2NC

MADIS2NC

Developed by the Developmental Testbed Center, Boulder Colorado, USA

Presenter

Presentation Notes

The major components for the MET package are represented in this flowchart. Green areas represent executables, while gray represent data. MET Version 4.0 consists of 12 tools as well as 4 optional tools (not shown in flowchart) organized into 4 groups: 1. Data reformatting and preparation tools (Gen Poly, PCP, etc. and Ensemble-Stat Tool, which is also in Statistical verification), 2. Statistical verification tools (MODE, Wavelet, etc.), 3. Aggregation and analysis tools (MODE-Analysis and Stat-Analysis), and 4. Optional tools (not shown) which include the World Wide Merged Cloud Analysis Regrid Tool, the World Wide Merged Cloud Analysis Plot Tool, the Plot-Point-Obs Tool, and the Plot-Data-Plane Tool. This flowchart looks very similar to the flowchart for MET version 3.1, however, this one, for version 4.0 includes the tool, MADIS2NC, which reformats MADIS point observations and currently supports RAOB and METAR types.

MODE OBJECT DEFINITION

OBJECT ORIENTED METHOD: MODE HOW IT WORKS

OBS ENS FCST Radius=5

ObjectThresh >6.35 mm

MergingThresh > 5.7 mm

Radius=5

ObjectThresh >6.35 mm

MergingThresh >5.7 mm

Merging

Matching

No false alarms

Misses

Merging

Matched Object 1 Matched Object 2 Unmatched Object

Presenter

Presentation Notes

Precipitation fields are smoothed using a user specified radius. The smoothed fields are then thresh-holded to form objects. MODE then acts on the individual fields to merge objects into clusters, which in this case is done by using a second lower threshold – the rubber band and coloring indicate the merged clusters. MODE then compares the objects and clusters between fields and calculates an interest value and matches based on these. MODE compares gridded model data to gridded observations and can be used as a basis for development of diagnostic verification metrics that can provide quantitative measures of uncertainty.

EXAMPLE – REFC > 30 DBZ – MODE OBJECTS Convolution Radius Increases

FSS = 0.64

Total Interest: 0.96 Area Ratio: 0.57 Centroid Distance: 95km P90 Intensity Ratio: 1.00



3 gs

9 gs

15 gs

USE OF ATTRIBUTES OF OBJECTS DEFINED BY MODE

Centroid Distance: Provides a quantitative sense of spatial Displacement. Small is good

Forecast Field

Observed Field

Axis Angle: Provides an objective measure of how well the objects are aligned. Small is good

Area Ratio: Provides an objective measure of whether there is an over- or under- prediction of areal extent. Close to 1 is good

Obs Area

Fcst Area

Area Ratio = Fcst Area Obs Area

5/14/2010

Symmetric Diff: May be a good summary statistic for how well Forecast and Observed objects match. Small is good

Forecast Field

Observed Field

P50/P90 Int: Provides objective measures of Median (50th percentile) and near-Peak (90th percentile) intensities found in objects. Ratio close To 1 is good

Total Interest: Summary statistic derived from fuzzy logic engine with user-defined Interest Maps for all these attributes plus some others. Close to 1 is good

Symmetric Difference: Non-Intersecting Area

Fcst PWT P50 = 29.0 P90 = 33.4

Obs IWV*10 P50 = 26.6 P90 = 31.5

Total Interest 0.75

USE OF ATTRIBUTES OF OBJECTS DEFINED BY MODE

MODE IN USE

6HR ACCUMULATED PRECIPITATION NEAR PEAK (90TH%) INTENSITY DIFFERENCE

D

iffer

ence

(P90

Fcs

t – P

90 O

bs)

High Resolution Deterministic Does Fairly Well

High Resolution Ensemble Mean Underpredicts

Mesoscale Deterministic Underpredicts

Mesoscale Ensemble Underpredicts the most

Overforecast

Underforecast

Impact of Core & Microphysics

90% Intensity shows overforecast of precipitation for ARW-Fer and ARW-MY members especially at higher thresholds – which means when it rains it pours in these members

> 25.4mm

Diffe

renc

e (F

cst-

Obs

) in

Nea

r Pea

k In

tens

ity in

MO

DE O

bj (m

m)

Object Threshold (mm)

NMM ARW

HMT NAM GFS

Using Attributes from MODE Objects

Color groups Different Microphysics

Ferrier Milbrant-Yau Thompson

Intense Cores

Optimal

Presenter

Presentation Notes

Convolution Radius = 5 gs; Convolution Threshold = 25.4 mm; Default Interest Map and Weighting

RADAR ECHO TOPS FOR AVIATION DESK

WRF Members with Different Microphysics Schemes

RADAR ECHO TOPS – SYMMETRIC & CENTROID DIFFERENCE

Symmetric Difference diverse Centroid Difference not Probably an over-forecast of area

Symmetric Difference large Centroid Difference is also May be a displacement error

Symmetric Difference:

Non-Intersecting Area

Centroid Distance

INTEGRATED WATER VAPOR AND PRECIPITABLE WATER

Ratios (Fcst/Obs) Area : 1.7 Median Intensity: 1.1 90th% Intensity: 1.1

Forecast: PWAT – 25.0 W/m2; Observation: IWV – 2.50 W/m2

Presenter

Presentation Notes

Convolution Radius: 5 gs; Threshold: Forecast – PWAT – 25.0 W/m2; Observation – IWV – 2.50 W/m2

Strip approximately 1000 km from Coast 96h GFS IVT Forecast 24h GFS IVT Forecast GFS IVT Analysis

Jan 9, 2010 18Z

• Helps with diagnosing model performance of landfall events

• Useful for qualitative analysis

• Fits with hi-res, smaller scale domains like LAPS. IVT = Wind850mb * IWV [cm m/s]

Slide provided by W. Clark and E. Tollerud

Presenter

Presentation Notes

Convolution Radius: 3; Threshold: 250 w/m2

Centroid Difference Longitudinal and Latitudinal Bias?

Direct use of currently available attributes must be done carefully in this domain due to object truncation. • Centroid Difference

provides Qualitative lead/lag information

• Development of special attributes would benefit Quantitative spatial error analysis

• Area attributes (not shown) should be interpreted with truncation in mind

West

East

North South

Slide provided by W. Clark and E. Tollerud

Presenter

Presentation Notes

Tara

MODE USED ON NON-CLOUDY AREAS

Clear Sky

Clouds

Cloud forecast objects would have taken up most of domain so non-cloudy areas were identified

Slide provided by M. Mittermaier, 2012

MET package used

APPLICATIONS - NOT SHOWN HERE Derived Wind Fields (divergence and convergence) A-Train downward looking radar data (X-Z plane) Time Domain (2 dimensional field and Time) Pressure fields including those for Tropical Cyclones

Operational evaluation of precipitation fields by: US NOAA Weather Prediction Center (WPC) and Saudi Arabia Presidential Ministry of the Environment (PME) (experimental)

ENSEMBLE MODE

APPLYING SPATIAL METHODS TO ENSEMBLES

As probabilities: Areas do not have “shape” of precipitation areas; may “spread” the area

As mean:

Area is not equivalent to any of the underlying ensemble members

TREATMENT OF SPATIAL ENSEMBLE FORECASTS

Alternative: Consider

ensembles of “attributes”

Evaluate distributions of “attribute” errors

EXAMPLE MAY 11, 2013

Ensemble Mean

Matched Forecast Object

Matched Forecast Object

Matched Observed Object

Unmatched Observed Object

SPREAD INCREASES WITH TIME

INDIVIDUAL MATCHED OBSERVED OBJECTS

1

2

3

4

5

median

Mat

ched

For

ecas

t Are

a (g

rid s

quar

es)

Object Areas

Fcst

Obs

2 5 3 1 4

Mat

ched

For

ecas

t Are

a (g

rid s

quar

es)

May 2013: 25 Days of Matched Observed/Forecast Pairs FORECAST AREA

Matched Forecast Centroid Distance (grid squares) Matched Forecast Symmetric Difference (grid squares)

May 2013: 25 Days of Matched Observed/Forecast Pairs

Symmetric Difference:

Non-Intersecting Area

Centroid Distance

ENSEMBLE MEAN MEMBERSHIP How many members typically make up ensemble mean that is matched?

3 or more

USING MODE ON PROBABILITY FIELDS

QPE_06 >12.7 MM VS. 50% PROB(APCP_06>12.7 MM)

Good Forecast with Displacement Error?

Traditional Metrics

Brier: 0.07 Area Under ROC: 0.62

Spatial Metrics

Centroid Distance: Obj1) 200 km Obj2) 88km

Area Ratio: Obj1) 0.69 Obj2) 0.65

1

2 Median Of Max Interest: 0.77

Obj PODY: 0.72 Obj FAR: 0.32

Presenter

Presentation Notes

Convolution Radius = 2gs Threshold = 0.5 Probability and 12.7 mm for QPE

MODE FOR DIFFERENT PROBABILITIES – MAY 11, 2013

Prob>2.54 mm >25%

Prob>2.54 mm >75%

Prob>2.54 mm >50%

Observation Forecast

SUMMARY • MODE can be used for many applications, including situations

when Forecast Field and Observation field are scaled differently

• Matched pair attributes may be used synergistically to diagnose areas for model development focus and trouble-shooting

• Next steps for Ensemble Distribution Verification • Plot as distribution of attribute errors • Develop score using individual member attributes

• Next steps for Probability Verification • Modify definition of weighting and interest maps based on area-ratio to

better compare probability fields

THANK YOU EXAMPLE MODE CONFIGURATION FILES

http://www.dtcenter.org/hwt/2010

http://www.dtcenter.org/hmt/2010 http://www.dtcenter.org/hmt/2011

MET: http://www.dtcenter.org/met/users

MET HELP: [email protected]

Email: [email protected]

MODE TIME DOMAIN

APPLICATION 1: VISUALIZATION/VERIFICATION OF SIMULATED ROTATING STORM TRACK LENGTHS • Time-domain object code is applied to hourly-max updraft helicity (UH) to identify

number, length, and intensity of time-domain UH objects (i.e. rotating storm tracks).

• A study was done on whether total UH path lengths could be used as a proxy for total tornado path lengths (Clark et al. 2012).

EXAMPLE UH FORECAST PRODUCT: 27 APRIL 2011

- Max UH from any ensemble member – blue: high-based, red shading for surface-based UH.

- Thick red line – 27 April exceedence probabilities for total tornado path lengths up to 5000 km.

- Thin red – exceedence probs for all other days

- Green line – climo - Grey vertical lines mark

path lengths corresponding to 1, 2, and 10 year return periods - Length of

entire row is the total UH path length for an ensemble member; members are ordered longest to shortest.

- Lengths of segments correspond to path lengths of individual UH objects. Shading level shows max intensity of UH within each object. Red shading is for surface based and green for elevated UH tracks.

Documents

UNDERSTANDING INDIVIDUAL AND ENSEMBLE FORECAST BEHAVIOR ... L. Jensen.pdf · UNDERSTANDING INDIVIDUAL AND ENSEMBLE FORECAST BEHAVIOR USING ... P50/P90 Int: Provides . objective