Assessing Forecast Assessing Forecast Uncertainty in the Uncertainty in the NDFDNDFD

Report to MDLReport to MDL

Matt Peroutka, Greg Zylstra, John Matt Peroutka, Greg Zylstra, John WagnerWagner

July 1, 2005July 1, 2005

OverviewOverview

Overall ProcessOverall Process MethodsMethods

– Data SourcesData Sources– Transformation to PercentilesTransformation to Percentiles– Diagnostic DataDiagnostic Data

ResultsResults– Transformation to PercentilesTransformation to Percentiles– Modeling the Joint DistributionModeling the Joint Distribution– Diagnostic DataDiagnostic Data

PlansPlans

Overall ProcessOverall Process

NUNCA

NDFD Forecast

NDFD Performance

Related Guidance

Expected Distribution of Observations

What’s in a name?What’s in a name?

NDFD UNCertainty Assessment NDFD UNCertainty Assessment (NUNCA)(NUNCA)

Numerical Uncertainty Numerical Uncertainty Assessment of NDFD via Assessment of NDFD via Climatology and Ensembles Climatology and Ensembles (NUANCE) (NUANCE)

Development PhaseDevelopment Phase

Observations (x)

Forecasts (f)

Diagnostic Data (d)

MergeJoint

Distribution Model p(f,x,d)

Transform to

Percentiles

Implementation PhaseImplementation Phase

Forecasts (f)

Diagnostic Data (d)

Joint Distribution

Model p(f,x,d)

Extract

Inferred Conditional Distribution

p(x | f,d)

Transform from

Percentiles

Transform to

Percentiles

Data SourcesData Sources

Ideally, NDFD grids and Analysis of RecordIdeally, NDFD grids and Analysis of Record Prototype with NDFD points and METAR Prototype with NDFD points and METAR

observationsobservations– October 2004 to April 2005, inclusiveOctober 2004 to April 2005, inclusive

US Historical Climatological Network (USHCN)US Historical Climatological Network (USHCN) Ensemble MOS (ENSMOS) archivesEnsemble MOS (ENSMOS) archives

– One bulletin from control runOne bulletin from control run– Five bulletins created from runs with positive Five bulletins created from runs with positive

perturbationsperturbations– Five bulletins created from runs with negative Five bulletins created from runs with negative

perturbationsperturbations

Transformation to Transformation to PercentilesPercentiles Addresses lack of cases in Addresses lack of cases in

development data with extreme development data with extreme values of values of ff or or xx..

Encourages combining of data.Encourages combining of data. NDFD has a short history.NDFD has a short history. NDFD includes a variety of NDFD includes a variety of

forecasting techniques.forecasting techniques.

Diagnostic DataDiagnostic Data

Standard Deviation (SD) of 11 Standard Deviation (SD) of 11 ENSMOS forecasts.ENSMOS forecasts.

““Ensemble Deviation” (ED)Ensemble Deviation” (ED)– Difference each perturbed forecast Difference each perturbed forecast

with control forecast;with control forecast;– Compute r. m. s.Compute r. m. s.

Results: Results: Transformation to Transformation to PercentilesPercentiles Obtained daily MaxT observations for 168 Obtained daily MaxT observations for 168

stations from USHCN.stations from USHCN. Percentile function computed at 5-day Percentile function computed at 5-day

intervals throughout the year.intervals throughout the year. Standard probability distributions used to Standard probability distributions used to

model distribution.model distribution.– Percentile function fitted to observations.Percentile function fitted to observations.– Fit parameters expressed as cosine series over day Fit parameters expressed as cosine series over day

of the year.of the year.– Quality of fit judged subjectively.Quality of fit judged subjectively.– Additional terms added to cosine series, if needed.Additional terms added to cosine series, if needed.

Statistical Statistical DistributionsDistributionsDistributioDistribution n

Variable Variable CommentComment

Normal Normal MaxT MaxT Poor fit “in the tails” Poor fit “in the tails”

Normal Normal ln (MaxT) ln (MaxT) Improved fit “in the tails” Improved fit “in the tails”

Binormal Binormal MaxT MaxT Skewness improved fit for some Skewness improved fit for some stations. Poor fit “in the tails.”stations. Poor fit “in the tails.”

Logistic Logistic MaxT MaxT Better fit than either version of Better fit than either version of Normal Normal

Laplace Laplace MaxT MaxT Worst fitWorst fit

Gumbel Gumbel MaxT MaxT Skewness improved fit for some Skewness improved fit for some stations. stations.

GumbelGumbel -(MaxT) -(MaxT) Skewness improved fit for some Skewness improved fit for some stations. stations.

Generalized Generalized Lambda Lambda

MaxTMaxT Best overall fitBest overall fit

Results: Modeling Results: Modeling Joint DistributionJoint Distribution Modeling Modeling p(f,x,d)p(f,x,d) straightforward straightforward

for prototype.for prototype.– Small number of stations.Small number of stations.– All values can be retained in All values can be retained in

memory.memory.– Will be re-designed for grids.Will be re-designed for grids.

Results: Diagnostic Results: Diagnostic DataData Stratified scatter diagrams for Stratified scatter diagrams for

day 7 day 7 – ED < 6 ED < 6 °F°F– ED ED ≥ ≥ 6 6 °F°F

Results: SD vs. EDResults: SD vs. ED

NDFD percentile errors binned NDFD percentile errors binned (interval 0.1 (interval 0.1 °F) °F) by associated by associated ED/SD value.ED/SD value.

Plot NDFD percentile error vs. Plot NDFD percentile error vs. ED/SD.ED/SD.

Future PlansFuture Plans

Quantitatively assess uncertainty.Quantitatively assess uncertainty.– Variations by forecast projection.Variations by forecast projection.

Expand to include MinT.Expand to include MinT. Prototype products.Prototype products.

– 50% confidence interval around NDFD50% confidence interval around NDFD– Probability Density FunctionProbability Density Function– Exceedence probabilities for key valuesExceedence probabilities for key values