Probability Forecasts and Verification

Uncertainty

• Forecasts have uncertainty that should be taken into account:– Observational Uncertainty (initial conditions)

• Instrument errors, bias in frequency of measurements, representativeness, reporting errors, random errors, precision errors, conversion errors …

– Model Uncertainty• Insufficient spatial resolution, truncation errors,

parameterization errors, coding errors, initialization errors …

Deterministic and Probabilistic Forecasts

• Deterministic Forecast: Specific Outcome is Predicted

• Probabilistic Forecast: Range of Possible Outcomes Accompanied by Confidence Estimate

Ex: Tercile Probabilities

Probability Forecast

Probabilistic Verification

• Requires Several Forecast Cases – Cannot Verify a Single Probabilistic Forecast

• Compare Forecast Distribution to a Distribution of Observations

• Rank Probability Score (RPS)– Rank Probability Skill Score (RPSS)

• Relative Operating Characteristics (ROC)• Reliability (Not Really Verification)

Rank Probability Score

Rank Probability Skill Score

July 1 startDJF prec forecastRPSS

NMME

CFSv2

Relative Operating Characteristics

Relative Operating Characteristics

Hit Rate = hits/(hits+misses)

False Alarm Rate = false alarms/(hits+false alarms)

Relative Operating Characteristics

For Each Threshold Build aContingency Table SummingOver All Forecasts

Relative Operating Characteristics

Reliability

• Prediction System is Reliable if Predicted Probabilities Agree with Observed Frequencies– Ex: Climatological Probability has Perfect

Reliability (33%, 33%, 33%)

• Reliability Diagram Measure whether the Prediction System is Reliable, Overconfident or Underconfident

• Reliability Diagram also Shows Resolution and Sharpness

Reliability Diagram

Reliability Diagram

T2m October Starts JFM Verification

T2m October Starts JFM Verification

Probabilistic Verification

• Requires Several Forecast Cases – Cannot Verify a Single Probabilistic Forecast

• Compare Forecast Distribution to a Distribution of Observations

• Rank Probability Score (RPS)– Rank Probability Skill Score (RPSS)

• Relative Operating Characteristics (ROC)• Reliability (Not Really Verification)