Embed Size (px)
Citation preview
NextGen Network Enabled Weather
Product Formats Workshop, Boulder CO
Decoder Discussion
Brian GockelNOAA/NWS Office of Science and Technology
November 18, 2009
Decoder DiscussionWednesday, November 18
1. Scope of Decoder Discussion
2. What Decoder-Related Problems Should Be Anticipated?a. Based on experienceb. Based on foresight [More of the same?]
3. What Standards Apply (OFCM, WMO, …)?a. Differing standards versionsb. Dealing with overlapping standards
4. Legacy Systems Transition
5. Decoder Metadata & Need for Consistency• Example: Site Identification Inconsistency
6. Rules (what is allowed, not allowed?)• Example: METAR remarks field• Example: Concatenated messages
Slide 2
Decoder Discussion - continuedWednesday, November 18
7. Local Tables
8. Other Decoder Challenges (Input from Group)
9. Governancea. Definition and establishment of governanceb. Leveraging existing governance bodiesc. Authoritative (oversight) and technical (execution) layers neededd. Need for ongoing interagency collaboration– a long-term activity
10. Next Steps• Very near term (e.g., FY10) and beyond FY10
Slide 3
Slide 4
Scope of Decoder Discussion
Software for encoding and decoding environmental products, such as netCDF, GRIB, Radar, BUFR, METAR, etc. This includes the executable software as well as the configuration files (i.e., “decoder metadata”*) upon which the encoders and decoders rely.
Motivation for Discussion: maximize compatibility between data-provider encoders and data-consumer decoders. Incompati-bilities or inconsistencies prevent end users from fully decoding and/or using the desired information (even in cases where product delivery works).
* Note that the term “decoder metadata,” as used here, is different from reg-rep metadata.
Slide 5
Weather Data Clients
FTI WAN / SWIM
Weather DataOrigin Server
Router
Weather DataOrigin Server
Router
Router
Weather Data Clients
DistributionServer
Registry/Repository
Registry/Repository
ED-8 Gateway
Federated
AWIPS
MADIS
NOMADS
NDFD
NEVS
Lightning Data
Sat Data
Cube Input Edge Server
Cube Input Edge Server
Cube Input Edge Server
Cube Input Edge Server
Cube Input Edge Server
Cube Input Edge Server
Cube Input Edge Server
Wx CubeData
Sources
AWIPSMADIS
Radar Data
Wx CubeData Destinations
Cube Output Edge Server
Cube Output Edge Server
Cube Output Edge Server
NWS IP NETWORKS (NOAANET)
NOAA/NWS FAA
CROSS ORGANIZATIONAL NOTIONAL ARCHITECTURE
ITWS ADAS
NWSTG
Weather DataOrigin Server
Router
NWP
DistributionServer
DistributionServer
Consumer Cube Service Adaptors
Consumer Systems
Consumer Cube Service Adaptors
Weather Data Clients
Consumer Cube Service Adaptors
Distribution Servers for
External Access
Consumer Systems
Consumer Systems
DMZ
From Mike Asmussen (Skjei) slide package, NOAA/NextGen System of Systems Workshop.Annotations in green & red.
GridEncode
GridDecode
GridDecode
GridDecode
GridDecode
GridDecode
Decoder DiscussionEstablishing compatibility is a near-term (developmental) and along-term (O&M) challenge. . .
1. New environmental models are developed; legacy models are eventually phased out.
2. With time, new parameters are added (e.g., model-simulated radar reflectivity at certain heights or elevations).*
3. Stations renamed, added, subtracted & moved (e.g., radar relocation or new airport).*
* Examples in backup slides.Slide 6
Decoder Issues – METAR Example
Slide 7
METAR – The domestic METAR code format is described in the Federal Meteorological Handbook (FMH) No. 1 "Surface Observations and Reports", (OFCM). The METAR format contains a REMARKS field… can be of 2 types:
• Automated/Manual/Plain Language• Additive and Maintenance Data
Example:
METAR KOKC 082252Z 14016KT 6SM HZ SCT035 OVC240 26/23 A2957 RMK AO2 PK WND 29031/2201 TSE37RAE25GSE2158GRB2158E10 SLP999 HAILSTONES>3/4 TSE MOV NE P0075 T02610233
From FMH #1: “Hailstone Size [coded as] (GR_[size]) [Plain Language]. At designated stations, the hailstone size shall be coded in the format, GR_[size]… Where plain language is called for, authorized contractions, abbreviations, and symbols should be used to conserve time and space. However, in no case should an essential remark, of which the observer is aware, be omitted for the lack of readily available contractions. In such cases, the only requirement is that the remark be clear. For a detailed list of authorized contractions, see FAA Order 7340 Series, Contractions.” TSE should probably be “TS E” (thunderstorm East). “TSE” most often means “thunderstorm ended.” From: http://www.srh.noaa.gov/oun/storms/20030508/metarobs.php (May 8, 2003 case)
Decoder IssuesMore METAR Examples
Slide 8
Denver:KDEN 150053Z 07013KT 4SM -SN BR SCT018 OVC036 01/M01A2988 RMK AO2 SLP123 P0000 T00061011
Amsterdam:EHAM 150125Z 24012KT 9999 FEW028 11/07 Q1006 NOSIG
Kiev:UKKK 150100Z 04002MPS 9999 SCT010 OVC023 04/01 Q1022 NOSIG
MPS = Meters per sec (versus KT=knots); NOSIG=No significant changes in weather since last report.
dx3-napo> ls -lt | grep -i bufr | grep -i tableb-rwxrwxr-x 1 fxa fxalpha 43062 Nov 3 2008 bufrTableB-rwxrwxr-x 1 fxa fxalpha 52780 Mar 14 2006 LAMPBufrTableB-rwxrwxr-x 1 fxa fxalpha 14029 Jun 29 2005 GFSBufrTableB-rwxrwxr-x 1 fxa fxalpha 14029 Jun 29 2005 ETABufrTableB-rwxrwxr-x 1 fxa fxalpha 12028 Apr 29 2004 NGMBufrTableB-rwxrwxr-x 1 fxa fxalpha 13413 Apr 24 2003 AVNBufrTableB-rwxrwxr-x 1 fxa fxalpha 13337 Apr 14 2003 MRFBufrTableB-rwxrwxr-x 1 fxa fxalpha 12028 Mar 2 2001 HPCBufrTableB-rwxrwxr-x 1 fxa fxalpha 12028 Oct 11 2000 MosBufrTableB
BUFR Table B – Multiple Versions
dx3-napo> wc -l HPCBufrTableB MosBufrTableB 157 HPCBufrTableB 157 MosBufrTableB 314 totaldx3-napo> diff HPCBufrTableB MosBufrTableB 144a145,146> 060071|PROB. OF PCPN AMT GE .01 INCH PAST 6HRS |% | 0| 0| 7> 060072|PROB. OF PCPN AMT GE .01 INCH PAST 12HRS|% | 0| 0| 7156,157d157< 060071|UNKNOW |NUMERIC | 0| 0| 1< 060072|UNKNOW |NUMERIC | 0| 0| 1
Slide 9
Decoder & Decoder Metadata Governance
Slide 10
1. Definition and establishment of governance – oversight body that establishes and maintains consistency in decoders and metadata; includes NextGen participating-system membership, is interagency, and exploits existing oversight/CM bodies.
2. Existing governance bodies:*a. NOAA Data Management Committee/DMIT (NOAA
oversight) b. CCBs (system-level)
3. Need for NextGen-related interagency collaboration (in the development and maintenance phases). How?
4. Is there a need for a NextGen developmental phase decoder team to work demonstration and IOC decoder issues?
5. Other questions: who should participate? how to collaborate?* adequate???
NOAA DMIT FAAoversight organization
Individual-System CCBs
DOD (and other)oversight organizations(?)
NOAANetCCB
NCEPCCB
AWIPSCCB
NWSTGCCB/DRG
Radar/ROCCCB
Decoder & Decoder MetadataProposal for a Long-Term Governance Structure
SPSRB(satellite)
CCB/DRG
JMBLCCBs
FAACCBs
MDLCCB
Slide 11
?? NextGen Decoder Work Group ??
Questions & Possible Next StepsDecoder & Decoder Metadata
1. Role (charge or scope) of any new decoder working group may need more clear definition and NOAA/FAA buy in.
2. Identify participants.
3. Inform system owners.
4. Other next steps???
Slide 12
Decoder Discussion - Backup SlideAddition of New Parameter*
WHY CAN'T I FIND THE [Model simulated] RADAR PRODUCTS…..Assuming you are using the nam.tHHz.awip12FF.tm00 files atftp://ftpprd.ncep.noaa.gov/pub/data/nccf/com/nam/prod/nam.YYYYMMDD/ where HH is the cycletime = 00, 06, 12 or 18, FF is the forecast hour = 00, 01, 02 ... 24 and YYYYMMDD is year,month & day; you are probably not seeing them because the new radar reflectivity fields are defined in NCEP GRIB parameter table version #129(http://www.nco.ncep.noaa.gov/pmb/docs/on388/table2.html#TABLE129), and you are expecting them to be defined in the normal/default Table #2 (http://www.nco.ncep.noaa.gov/pmb/docs/on388/table2.html).
We had to put them in this alternative table because Table 2 had no room for new parametersto be added. According to the GRIB documentation (see http://www.nco.ncep.noaa.gov/pmb/docs/on388/section1.html) you can findwhich Parameter Table is being used in octet 4 of the GRIB PDS (Product Definition Section).
Note the kpds5 entried above and that in Table 129, derived model reflectivity isvariable #211 and composite reflectivity is variable #212. In Table 2 these would have been upward short wave and long wave flux, and this is probably what your existing processing assumed they were.
* Excerpt from EMC model information page.
Slide 13
NWS Region email to NWS/OS&T:
“…in March 2008 we had asked at the request of WFO BTV to change thePlattsburgh, NY sounding point to account for the ASOS (and TAF) relocation from KPLB to KPBG. I'm not sure if NCEP has made this change yet.”
New LocationCD STATION ICAO IATA LAT LONG ELEV(m) M N V U A CNY PLATTSBURGH INTL KPBG PBG 44 39N 073 28W 71 Z T 8 US
Old LocationNY PLATTSBURGH KPLB PLB 44 41N 073 32W 106 X V A 6 US
Decoder Discussion - Backup SlideStation Change
TAFMETARNAM
SoundingASOS
Slide 14
Decoder Discussion - Backup SlideStation Lists: NWS Location Identifier Database
https://ops13jweb.nws.noaa.gov/nwsli/liu/ApprovedStations.jsp(Requires UID & password)
Slide 15
Decoder Metadata Example – Backup SlideExisting Data Format Standard (html format)
http://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc.shtml
Slide 16
