WMO 4.1 A critique of the existing data models for weather radar data exchange as presented in item...

WMO

4.1 A critique of the existing data models for weather radar data exchange as presented in item

3.1 docsCBS/OPAG-IOS Workshop on Radar Data Exchange

Exeter, UK, 24-26 April 2013

Daniel Michelson, SMHI, Sweden

WMO

Data exchange model has four components:

1. File format – container for storing data in physical files.

2. Data/information model – the way in which information is organized/represented, either in computer memory or files, independently of file format.

3. Envelope – extra header announcing/describing the contents of the information being exchanged.

4. Protocol – mechanisms for communication over a network.

test footer 3

Content-Disposition: form-data; name="<baltrad_frame_xml/>"Content-Type: multipart/form-data; charset=UTF-8Content-Transfer-Encoding: 8bit<?xml version="1.0" encoding="UTF-8"?><baltrad_frame> <header mimetype="multipart/form-data" sender="local_production_system"/> <content channel="test_channel" name="pas1.h5" type="file"/></baltrad_frame>--s7Wj4EEwDrDPwOdPM30KMQzaEpfnylContent-Disposition: form-data; name="<baltrad_frame_file/>"; filename="pas1.h5"Content-Type: multipart/form-dataContent-Transfer-Encoding: binary�HDFÿ (… binary payload ODIM_H5 file …)--s7Wj4EEwDrDPwOdPM30KMQzaEpfnyl--

E.g. an early BALTRAD HTML exchange message

WMO

Different categories of file formats

1. Transmission – to facilitate data transfer from site to central facility. Can be optimized for network load balancing, e.g. ray-by-ray.

2. Production – contains all data and metadata required to derive higher-order outputs (e.g. QC:ed data, products) from input data.

3. Exchange – representation of data or product for sharing but not necessarily further processing, e.g. a radar composite or vertical profile.

4. Archive – e.g. to represent data from many different observation systems for storage.

WMO

Basic data exchange concepts

“Passive” Simple availability, e.g. “pull” Sender/recipient are aware of each other but

transact indirectly, e.g. “push” (GTS) Directory polling or file-system event monitoring

“Active” Sender/recipient are aware of each other and

interact/transact directly E.g. BALTRAD

WMO

GTS for radar data exchange?

Advantages+ Already exists+ Active networking

Disadvantage- Hierarchical organization causes delays

For example:

Exeter Offenbach Norrköping Riga

faster?

WMO

BALTRAD for radar data exchange?

Positives+ Already exists+ Active networking+ WIS integration already demonstrated (2010)

Challenges- Still being developed- Requires dedicated software- Not yet officially recognized as part of WIS

File format: GRIB

+ WMO standard+ Already in use with

national composite products in RA IV.

+ Demonstrated use for exchange. (GRIB1)

- No single official, centrally maintained, software.

- No mature information model representation.

File format: TITAN

+ Already in use in two regions (I and III)

+ Storm tracks also available, e.g. XML

+ Open software, centrally maintained

+ Efficient for transmission, production, exchange

- Documentation- No DQ preparedness- Prepared for dual-pol

moments?- Built-in compression?- Unknown outside

meteorology

File format: BUFR

+ WMO standard.+ Suitable for point and

profile data.+ Very flexible, “can encode

anything”.+ Demonstrated use in

Region VI, ODIM_BUFR

- “Unwieldy” for non-trivial data such as radar.

- Many mutually incompatible software and data model implementations.

- No single official, centrally maintained, software.

- Bottleneck to academia.- Not a marketable skill.- Largely unknown outside

meteorology.

File format: HDF5

+ Central, open, modern software developed/maintained by HDF Group.

+ Bundled for various OSes.+ Established in Earth

Sciences.+ ODIM_H5 prepared for

dual-pol and DQ.+ ODIM_H5 is “policed”.+ Efficient for production

and exchange.+ ODIM_H5 compatibility

between regions V and VI

- Steep learning curve.- Currently unsuitable as

transmission format.- ODIM_H5 not an orthodox

information model.- Ambiguities in ODIM_H5

risk incompatibilities.

File format: netCDF, version 4

+ Central, open, modern software developed/maintained by NCAR.

+ Bundled for various OSes.+ Established in Earth

Sciences through “CF Conventions”.

+ GIS-compliant.+ “CF Radial” prepared for

dual pol?

- Requires HDF5, added abstraction layer.

- No mature CF Convention yet for operational radar, although “CF Radial” for R&D.

- Unprepared for DQ?

WMO

ODIM in OPERA, February 2013

Input data Output products

Green = ODIM_H5Brown = ODIM_BUFRYellow = Both

WMO

Summary

Terminology needs clarifying. “Data model”, “exchange”, “file format”, etc.

Several file formats in use. No orthodox information model as such in use.

Not necessarily a disadvantage. Tight connections between file formats and

information models. Good solutions exist already.

www.wmo.int

Thank you for your attentionDaniel Michelson

Swedish Meteorological and Hydrological Institute

Norrköping, Sweden

daniel.michelson@smhi.se