Upload
miyo
View
61
Download
0
Embed Size (px)
DESCRIPTION
Verification of WAFS Global Icing Products. Jennifer Mahoney 1 and Sean Madine 1,2 1 NOAA/Earth System Research Laboratory 2 Cooperative Institute for Research in the Atmosphere. This work was completed on behalf of the U.S. Federal Aviation Administration - PowerPoint PPT Presentation
Citation preview
Verification of WAFS Global Icing Products
Jennifer Mahoney1 and Sean Madine1,2
1NOAA/Earth System Research Laboratory2Cooperative Institute for Research in the Atmosphere
This work was completed on behalf of the U.S. Federal Aviation AdministrationPresented at: WAFS Workshop on the Use and Visualisation of Gridded SIGWX Forecasts Paris, September 14-15 2009
Acknowledgements
This work was completed on behalf of the U.S Federal Aviation Administration
Project Team• Project Lead: Sean Madine• Project Analyst: Chungu Lu • Programming Support: Mike Kay• Program Chief: Jennifer Mahoney
Outline of Discussion
• Motivation, Background, and Challenges• Approach and Assessment Strategy
– Creation and validation of CloudSat Icing Product (CLIP)
• Forecast Analyses– Comparison of WAFC U.K. and WAFC U.S. Global Icing– Comparison of WAFC Global Icing to Significant
Weather (SigWx) Product• Summary
Motivation
• Icing forecasts are used in strategic planning
– One example: Planning for amount of fuel carried on board aircraft to manage possible icing encounters.
• The WAFC Washington (U.S.) and the WAFC London (U.K.) have developed Global Icing Products, updated frequently, that may improve icing information that is provided to the aviation community.
Background
• Project Goal: Evaluate the quality of the WAFC Global Icing Forecasts; with resource and time constraints:
– Altitudes that typically contain icing (i.e., mid-levels)
– Scoped evaluation to Nov 2008 – Jan 2009; winter in northern hemisphere
– Domains: Global, North Pacific, North Atlantic; with a North Atlantic focus
• Evaluation criteria : 1) accuracy of the products should be at least as good as the human generated Significant Weather (SigWx) forecasts produced today, and 2) WAFC U.S. and U.K icing products need to be compatible
Challenges
• Obtaining a representative observation dataset– Voice pilot reports (PIREPS) for icing are sparse in time and
space– PIREPs poorly sample at flight levels relevant to
operational decisions (enroute flight levels are typically above the occurrence of icing)
– PIREPs disproportionately sample icing conditions• Exploring new techniques for creating a proxy
observation of icing in the atmosphere
Assessment Approach
• Compare the quality of the WAFC U.K icing product to the WAFC U.S. icing products– Measure the quality– Qualitative examination of compatibility (i.e., incompatible
means ambiguity of the forecast in the planning process)• Compare the quality of the WAFC Global Icing
Products to the Significant Weather Product– Measure the quality
Assessment Strategy
• Build an icing verification proxy based on satellite products and use for evaluating the WAFC icing forecasts over global data-sparse domains
• Development of CloudSat Icing Product (CLIP)– Tuned and validated CLIP with respect to the Current Icing Potential (CIP),
current U.S. operational icing analysis
– Study period: November 2007 through January 2008 (Note: Same season, 1 year earlier)
• Use CLIP to assess the quality of the WAFC global icing products and SigWx products over global domains– Study period: November 2008 through January 2009
CloudSat Icing Product (CLIP) Strengths
• CloudSat cloud profiling radar very effective at diagnosing existence of clouds, particularly at levels relevant to our study
– CLIP highly accurate at detecting no cloud and therefore ‘no icing’
• Information for CLIP allows assessment of forecast risk (miss) and forecast efficiency (false alarms)
– PIREPs don’t allow the false alarm measure because of the imbalance of yes/no reporting
• CLIP’S radar input provides– An objective measure of cloud, at high resolution– Continuous reporting along orbiting track– An improvement upon the subjective nature and sparseness of using PIREPs
Comparison of the WAFC Global Icing Products
Analysis of WAFC Global Icing Products
• Many stratifications examined– Geographic domain
Global, N. Atlantic, N. Pacific– Flight level
• All, FL100, FL140– Cloud type (as diagnosed by CloudSat)
• All, Convective, Non-convective– Algorithm threshold
• 0.1, 0.3, and 0.5 – Lead time
• All leads 6, 12, …, 36-h individually
Analysis of WAFC Global Icing Products
• Main thread of investigation– North Atlantic domain
• Domain of interest during assessment winter time period and included significant amount of air traffic density
– Altitudes 10,000 - 20,000 ft • Operationally significant altitudes and typically contain ice
– Non-convective clouds • Products definition specifically includes non-convective icing
– Max icing attribute• Most compatible WAFC forecasts
Domains
Global, North Atlantic, North Pacific; 3 CloudSat orbits, each orbit ~1.5h,
WAFS Maximum Icing12 January 2009
UK/FL100
UK/FL140 US/FL140
US/FL100
Vertical Cross Sections Along CloudSat Path Atlantic Domain
WAFS-UK
WAFS-US
Cloud Classification
CLIP
12 January 2009
WAFS Maximum Icing23 December 2008
UK/FL100 US/FL100
UK/FL140 US/FL140
Vertical Cross Sections Along CloudSat Path Atlantic Domain
23 December 2008
Comparison of Performance: Flight Level(Statistics for 12- and 18-hr leads, Atlantic domain,
Non-convective clouds, Max icing attribute)
ROC Thresholds: 0.1, 0.3, and 0.5
Product
False Alarm Rate (1-PODn)
POD
Atlantic domain, 0.1 Threshold, FL100/140, 12- and 18-hr leads, Max icing intensity, Non-convective clouds
CLIP
12367
6684 54837 6430 3375
15356 2940 3640
Yes Icing No Icing No Cloud
No Icing Warm Cloud
No Icing Cold Cloud
UK
(fcst
)
8987
10063 58617 5709 4764
11576 3661 2251Yes
No
US
(fcst
)
Comparison WAFC Icing Forecasts in Operational Context
Yes Icing No Icing No Cloud
No Icing Warm Cloud
No Icing Cold Cloud
Yes
No
Blue – AgreementYellow – False alarmsRed - Misses
Comparison Summary
• The WAFC U.K and WAFC U.S. forecasts provide roughly the same forecast efficiency, but the WAFC U.K. provides lower operational risk
FAR POD BiasWAFC U.K 64% 65% 1.8WAFC U.s 66% 47% 1.4
Comparison of the WAFC Global Icing Products to the SigWx Product
WAFS Mid Level SIGWX Product
• Thunderstorms/CB• Tropical Cyclones• Mod/Sev Turb (CAT or
IC)• Mod/Sev Icing• Tropopause Heights• Jetstreams (80kt and
above depicted)• Jet Depth• Volcanic Eruptions• Widespread Sand/Dust
storms• Release of Radioactive
Materials
NOTE: Because of choice to use N. Atlantic domain, only U.S SigWx Product was evaluated
Analysis - Main Thread
• Atlantic domain• Altitudes 10,000 - 20,000 ft• Non-convective clouds• Max icing attribute
Comparison of Performance: SigWx Chart(FL100/140, SigWx issuances/lead, Atlantic domain)
ROC Thresholds: 0.1, 0.3, and 0.5
False Alarm Rate (1-PODn)
POD
Product
NOTE: Because of choice to use N. Atlantic domain, only U.S SigWx product was evaluated
Comparison Summary
• The WAFC U.K and WAFC U.S. forecasts provide roughly the same forecast efficiency, but the WAFC U.K. provides lower operational risk
FAR POD BiasWAFC U.K 64% 65% 1.8WAFC U.S. 66% 47% 1.4SigWx Product
69% 31% 1.2
Summary• The U.K. icing product slightly outperforms the U.S. icing product in an
operational; For similar FAR values, the U.K. has higher POD
• Qualitatively, the WAFC U.K. and U.S products are incompatible in volume extent and intensity of potential
• Both U.K. and U.S. products perform better than the SigWx Product in the operational context; For similar FAR, both WAFC products have higher POD values
• Based on this study and understanding of the algorithms, the automated WAFC product could be significantly improved with respect to volume coverage and calibration of potential
Appendix
• False alarm ratio = yn/yy+yn• False alarm rate = 1-PODn = yn/nn+yn• PODn = nn/nn+yn• POD = yy/yy+ny
2x2 Contingency TableObservation
Forecast Y N
Y YY YN
N NY NN