Precipitation Verification of CAPS Real-time Forecasts During IHOP 2002

Precipitation Verification of CAPS Real-time Precipitation Verification of CAPS Real-time Forecasts During IHOP 2002Forecasts During IHOP 2002

Ming XueMing Xue1,21,2 and Jinzhong Min and Jinzhong Min11

Other contributors: Keith BrewsterOther contributors: Keith Brewster11 Dan Weber Dan Weber11, Kevin Thomas, Kevin Thomas11

[email protected]@ou.edu3/26/20033/26/2003

Center for Analysis and Prediction of Storms (CAPS) Center for Analysis and Prediction of Storms (CAPS)11

School of MeteorologySchool of Meteorology22 University of Oklahoma University of Oklahoma

s fn

IHOP Related Research at CAPSIHOP Related Research at CAPS

• CAPS is supported through an NSF grant to

Contribute to the IHOP field experiment and Contribute to the IHOP field experiment and Perform research using data collectedPerform research using data collected

• Emphases of our work include

Optimal Assimilation and Qualitative Optimal Assimilation and Qualitative assessment of the impact of water vapor and assessment of the impact of water vapor and other high-resolution observations on storm-other high-resolution observations on storm-scale QPF.scale QPF.

Goals of CAPS Realtime Foreacst During IHOPGoals of CAPS Realtime Foreacst During IHOP

To provide additional high-resolution NWP To provide additional high-resolution NWP support for the real time operations of IHOPsupport for the real time operations of IHOP

To obtain an initial assessment of numerical To obtain an initial assessment of numerical model performance for cases during this model performance for cases during this periodperiod

To identify data sets and cases for extensive To identify data sets and cases for extensive retropective studiesretropective studies

CAPS Real Time Forecast CAPS Real Time Forecast DomainDomain

273×195

183×163

213×131

CAPS Real Time Forecast CAPS Real Time Forecast TimelineTimeline

ARPS Model ConfigurationARPS Model Configuration

NonhydrostaticNonhydrostatic dynamics with dynamics with vertically-stretched vertically-stretched terrain-following gridterrain-following grid

Domain 20 km deep with Domain 20 km deep with 53 levels.53 levels.

3 ice-phase microphysics (Lin-Tao)3 ice-phase microphysics (Lin-Tao)

New Kain-Fritsch cumulus parameterization on 27 and 9 New Kain-Fritsch cumulus parameterization on 27 and 9 km gridskm grids

NCSA Long and Short Wave NCSA Long and Short Wave Radiative TransferRadiative Transfer scheme scheme

1.5-order TKE-based SGS turbulence and PBL 1.5-order TKE-based SGS turbulence and PBL ParameterizationParameterization

2-layer 2-layer soil and vegetation modelsoil and vegetation model

Data and Initial ConditionsData and Initial Conditions

• IC from ADAS analysis with cloud/diabatic initialization

• Eta BC for CONUS grid and background of IC analysis

• Rawinsonde and wind profiler data used on CONUS and 9km grids

• MDCRS (aircraft), METAR (surface) and Oklahoma Mesonet data on all grids

• Satellite: IR cloud-top temperature used in cloud analysis.

• CRAFT Level-II and NIDS WSR-88D data: Reflectivity used in cloud analysis on 9 and 3km grids, and radial velocity used to adjust the wind fields.

Cloud Analysis in the Initial ConditionsCloud Analysis in the Initial Conditions

• Level-II data from 12 radars (via CRAFT) and Level-III (NIDS) data from 12 others in the CGP were used

• The cloud analysis also used visible and infrared channel data from GOES-8 satellite and surface observations of clouds

• The cloud analysis procedure analyzes qv, T and microphysical variables

Computational IssuesComputational Issues

• The data ingest, preprocessing, analysis and boundary condition preparation as well as post-processing were done on local workstations.

• The three morning forecasts were made on a PSC HP/Compaq Alpha-based clusters using 240 processors.

• The 00 UTC SPstorm forecast was run on NCSA’s Intel Itanium-based Linux cluster, also using 240 processors.

• Perl-based ARPScntl system used to control everything

• Both NCSA and PSC systems were very new at the time. Considerable system-wide tuning was still necessary to achieve good throughput. A factor of 2 overall speedup was achieved during the period.

• Data I/O was the biggest bottleneck. Local data processing was another.

Dissemination of Forecast ProductsDissemination of Forecast Products

• Graphical products, including fields and sounding animations, were generated and posted on the web as the hourly model outputs became available.

• A workstation dedicated to displaying forecast products was placed at the IHOP operation center.

• A CAPS scientist was on duty daily to evaluate and assist in the interpretation of the forecast products.

• A web-based evaluation form was used to provide an archive of forecast evaluations and other related information.

• The forecast products are available at http://ihop.caps.ou.edu, and we will keep the products online to facilitate retrospective studies.

CAPS IHOP Forecast PageCAPS IHOP Forecast Pagehttp://ihop.caps.ou.edu

Standard QPF VerificationsStandard QPF Verifications

• Precipitation forecasts scores (ETS, Bias) calculated against hourly rain gauge station data (grid to point) from NCDC (~3000 station in CONUS)

• Scores for 3, 6, 12 and 24 h forecast length calculated

• Scores calculated for full grids and for common domains

• Scores also calculated against NCEP stage IV data (grid to grid)

• Mean scores over the entire experiment period (~40 days) will be presented

Questions we can askQuestions we can ask

• How skillful is a NWP model at short range precipitation forecast?

• Does hi-resolution really help improve precipitation scores, and if so, how much?

• How much did the diabatic initialization help?

• Do model predicted precipitation systems/patterns have realistic propagations, and what are the modes of the propagations?

• Is parameterized precipitation well behaved?

ETS on CONUS gridETS on CONUS grid

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

0.01 0.1 0.25 0.5 0.75 1 1.5 2 3

3hr

6hr

12hr

24hr

ETS on SPmeso (9km) gridETS on SPmeso (9km) grid

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

0.01 0.1 0.25 0.5 0.75 1 1.5 2 3

3hr

6hr

12hr

24hr

ETS on SPstorm (3km) gridETS on SPstorm (3km) grid

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

0.01 0.1 0.25 0.5 0.75 1 1.5 2 3

3hr

6hr

12hr

ETS on all three gridsETS on all three grids

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

0.01 0.1 0.25 0.5 0.75 1 1.5 2 3

3hr

6hr

12hr

24hr

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

0.01 0.1 0.25 0.5 0.75 1 1.5 2 3

3hr

6hr

12hr

24hr

27km

9km

3km

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

0.01 0.1 0.25 0.5 0.75 1 1.5 2 3

3hr

6hr

12hr

Notes on ETS from the 3 gridsNotes on ETS from the 3 grids

• On CONUS grid, 3hourly ETS much lower than that on the two higher-res grids

• 12 and 24-hour precip scores are higher on the CONUS grid (keep in mind the difference in domain coverage)

• Skill scores decrease as the verification interval decreases, but less so on the 9km and 3km grids

• High thresholds have lower skills

• Second conclusion changes when comparison is on a common grid

CONUS and 9km ETS CONUS and 9km ETS in the COMMON 9km domainin the COMMON 9km domain

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.01 0.1 0.25 0.5 0.75 1 1.5 2 3

3hr Spmeso

6hr Spmeso

12hr SPmeso

24hr Spmeso

3hr CONUS

6hr CONUS

12hr CONUS

24hr CONUS

9km (SPmeso) and 3km (SPstorm) ETS 9km (SPmeso) and 3km (SPstorm) ETS in the common 3km domainin the common 3km domain

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.01 0.1 0.25 0.5 0.75 1 1.5 2 3

3hr Spstorm

6hr Spstorm

12hr Spstorm

3hr Spmeso

6hr Spmeso

12hr Spmeso

Comments on ETS in common domainsComments on ETS in common domains

• ETS scores are consistently better on higher resolution grids when verification in the same domain

• The differences are larger for shorter verification intervals

• Improvements at low thresholds are more significant

• Improvement from 27 to 9 km more significant than that from 9 to 3 km (0.28/ 0.17 v.s. 0.27/0.22)

• The forecasts have less skill in the 3km domain (not grid), presumable due to more active convection

• Keep in mind that the high-resolution forecast is to some extent dependent on coarser grid BC’s

Biases of CONUS and SPmeso Grids in Biases of CONUS and SPmeso Grids in COMMON SPmeso DomainCOMMON SPmeso Domain

0.0

0.5

1.0

1.5

2.0

2.5

3.0

0.01 0.1 0.25 0.5 0.75

3hr Spmeso

6hr Spmeso

12hr Spmeso

24hr Spmeso

3hr CONUS

6hr CONUS

12hr CONUS

24hr CONUS

Biases of SPmeso and SPstormBiases of SPmeso and SPstormGrids in COMMON SPstorm DomainGrids in COMMON SPstorm Domain

0. 0

0. 5

1. 0

1. 5

2. 0

2. 5

3. 0

3. 5

0. 01 0. 1 0. 25 0. 5 0. 75

3hr SPstorm

6hr SPstorm

12hr SPstorm

3hr SPmeso

6hr SPmeso

12hr SPmeso

Comments on Bias ScoresComments on Bias Scores

• High biases are seen for high thresholds at all resolutions

• High biases more severe at higher resolutions

• Low biases are only observed at low thresholds on CONUS grid

• Cumulus parameterization (KF scheme is known to produce high biases at high thresholds – e.g., ETA-KF runs of NSSL)?

• Too much initial moisture introduced by cloud analysis?

• Microphysics problem?

• Too strong dynamic feed back?

• Still insufficient resolutions to properly resolve updrafts?

• Other causes?

3-hr accumulated precipitation ETS for different forecast periods

CONUS ETS verified on NCEP 236 grid CONUS ETS verified on NCEP 236 grid (dx~40km)(dx~40km)(May 15 – June 25, 2002)(May 15 – June 25, 2002)

Different 3 hour periods

0.21

Preliminary comparison with WRF, RUC, MM5, and ETA run during the IHOP

3hr accumulated precipitation ETS and Bias

WRF, RUC, MM5 and ETA scores generated at FSL RTVS page at

http://www-ad.fsl.noaa.gov/fvb/rtvs/ihop/station/ (earlier presentation by Andy Loughe)

The scores were calculated by interpolating forecast to hourly gauge stations, and are for the first forecast period only (not the mean of periods over the entire forecast range)

ARPS scores shown are against Stage IV gridded data

0.01 0.1 0.25 0.5 0.75 1.0 1.5 2.0 3.0

0.01 0.1 0.25 0.5 0.75 1.0 1.5 2.0 3.0

WRF(22km) and RUC(20km) ARPS (27km)

Comparison with WRF and RUC for the same period 3hr accumulated precipitation ETS and Bias versus thresholds

http

://w

ww

-ad.

fsl.n

oaa.

gov/

fvb/

rtvs

/ihop

/sta

tion/

0.16

2.7

0.2

1.5

Verified on SPmeso domain6hr accumulated precipitation ETS and Bias versus thresholds

0.01 0.1 0.25 0.5 0.75 1.0 1.5 2.0 3.0

0.01 0.1 0.25 0.5 0.75 1.0 1.5 2.0 3.0


0.3


0.01 0.1 0.25 0.5 0.75 1.0 1.5 2.0 3.0

0.01 0.1 0.25 0.5 0.75 1.0 1.5 2.0 3.0

12hr accumulated precipitation ETS and Bias versus thresholds

0.350.38

SPmeso grid verification

Comparison with WRF, ETA, MM5 and RUC for the same period 3hr accumulated precipitation ETS and Bias versus thresholds

0.01 0.1 0.25 0.5 0.75 1.0 1.5 2.0 3.0

0.01 0.1 0.25 0.5 0.75 1.0 1.5 2.0 3.0

WRF(10km), ETA(12), MM5(12) and RUC(10) ARPS (9km)

0.23


0.01 0.1 0.25 0.5 0.75 1.0 1.5 2.0 3.0

0.01 0.1 0.25 0.5 0.75 1.0 1.5 2.0 3.0



0.01 0.1 0.25 0.5 0.75 1.0 1.5 2.0 3.0

0.01 0.1 0.25 0.5 0.75 1.0 1.5 2.0 3.0


0.35

Hovmoller Diagrams of Hourly y Hovmoller Diagrams of Hourly y (latitudinal) mean Precipitation(latitudinal) mean Precipitation

• Questions: Inspired by Carbone et al (2002)Inspired by Carbone et al (2002)

How does the propagation of precipitating systems How does the propagation of precipitating systems compare at different resolutions?compare at different resolutions?

Does parameterized precipitation propagate at the Does parameterized precipitation propagate at the right speed?right speed?

Is explicit precipitation on high-resolution grid better Is explicit precipitation on high-resolution grid better forecasted? forecasted?

Predictability Implications Predictability Implications

CAPS Real Time Forecast CAPS Real Time Forecast DomainDomain

273×195

183×163

213×131

Hovmoller diagrams of hourly forecast Hovmoller diagrams of hourly forecast rainfall for 15 May to 5 June 2002rainfall for 15 May to 5 June 2002

Hovmoller diagrams of hourly Hovmoller diagrams of hourly forecast rainfall for 6-25 June 2002forecast rainfall for 6-25 June 2002

Hovmoller diagram hourly forecast Hovmoller diagram hourly forecast rainfall for 16-18 May 2002rainfall for 16-18 May 2002

Hovmoller diagram of hourly forecast Hovmoller diagram of hourly forecast rainfall for 23-26 May 2002rainfall for 23-26 May 2002

June 15, 2002, CONUS GridJune 15, 2002, CONUS Grid

NCEP Hourly Precip 27 km Forecast Precip Hourly Rate.

14 hour forecast valid at 02 UTC

L



24 h forecast




L

June 15, 2002, 9km GridJune 15, 2002, 9km Grid



L



24 hour forecast




L

June 15, 2002 – 3km gridJune 15, 2002 – 3km grid

NCEP Hourly Precip 3 km Forecast Hourly Precip Rate


L


NCEP Hourly Precip Analysis 3 km Forecast Hourly Precip Rate

11 hour forecast


NCEP Hourly Precip 3 km Forecast Hourly Precip Rate


L

June 15, 2002June 15, 2002

NCEP Hourly Precip ARPS 3 km Forecast – Comp. Ref.


Hovmoller diagram of hourly forecast Hovmoller diagram of hourly forecast rainfall for 15-18 June 2002rainfall for 15-18 June 2002

Oklahoma

Comments on Hovmoller DiagramsComments on Hovmoller Diagrams

• Propagation of precipitation systems is found on all grids, including CONUS and SPmeso that used cumulus parameterization

• Propagation not necessarily faster on higher-resolution grids

• The short forecast lengths (15 and 12h) of 3 km grid complicate the interpretation

• More detailed process analyses are needed to understand the mode of comparison

• Diagrams of observed precip will be created for comparison

June 12-13, 2002 CaseJune 12-13, 2002 Case

00-12UTC, June 13, 2002, Hourly Precip00-12UTC, June 13, 2002, Hourly Precip

00-12UTC, June 13, 2002, Hourly Precip00-12UTC, June 13, 2002, Hourly Precip

Future PlanFuture Plan

• Refine the QPF verifications• Perform detailed studies on selected CI and QPF cases

with the emphasis on model simulations• Rerun selected cases and the entire periods by

assimilating more data, initially relatively easy ones (e.g., surface network, dropsondes, radiometric profiles)

• Study the sensitivities of forecast to these data• Study the QPF sensitivity to initial conditions (via

forward as well as adjoint models)• Develop new capabilities to assimilate indirect

observations, e.g., GPS slant water delay (want to work with instrument people here)

• Verify model prediction against special IHOP data sets (e.g., AB profiles)

• Make available assimilated data sets to the community

Documents

Precipitation Verification of CAPS Real-time Forecasts During IHOP 2002