Upload
rodney-cummings
View
215
Download
1
Embed Size (px)
Citation preview
Satellite Application Facility for Climate Monitoring – CM SAF20. April 2023 1
Data Formats and Tools
Deutscher Wetterdienst
R.W. Mueller, R.Hollmann, C.Träger-Chatterjee
Satellite Application Facility for Climate Monitoring – CM SAF20. April 2023 2
Content
• Overview
• HDF5
• netCDF
• Binary
• ASCII
• Conclusion
Satellite Application Facility for Climate Monitoring – CM SAF20. April 2023 3
Overview• New data formats have been developed
better handling of manifold information provided by satellite data, reanalysis or model data
optimise computing performance (IO-process)
reduce disk space needed
• Requirements to the data format: storage of …
… the data itself with high resolution in space-time
Different data layers possible
… the meta-information, e.g.
Calibration coefficients
Geolocation and projection
Statistical error information
Gain and offset
Whatever the operator would like to add as meta information
Satellite Application Facility for Climate Monitoring – CM SAF20. April 2023 4
Overview – the favourites
• For satellite data two formats are important. Different but related – both with associated data model
HDF: Hierarchical data format
netCDF: network Common Data Form
• Further formats for satellite data:
HRIT raw data format not discussed, focus on products
Specific Binary Format always possible, no common data model
ASCII no data model, quite seldom
Satellite Application Facility for Climate Monitoring – CM SAF20. April 2023 5
HDF – Hierarchical Data Format
• HDF5 - general purpose library and file format for storing scientific data
• Create and store almost any kind of scientific data structure
e.g. images, arrays of vectors, structured and unstructured grids, …
one can also mix and match different data formats in HDF5 files
• Efficient storage and I/O
created to address the data management needs of high performance, data intensive computing environments
As a result, library and format emphasize storage and I/O efficiency (especially on parallel machines), including file compression
Satellite Application Facility for Climate Monitoring – CM SAF20. April 2023 6
HDF – Hierarchical Data Format
• The most recent version is HDF5, but a lot of data are still in HDF4 format.
Both are machine independent (no big / little endian problem)
• Information, tools, examples and the HDF software (library) available at http:/hdf.ncsa.uiuc.edu/HDF5 and http://hdf.ncsa.uiuc.edu/hdf4.html
• Widely used, e.g.:
MODIS (HDF4)
Eumetsat, e.g. all SAFs (HDF5)
Satellite Application Facility for Climate Monitoring – CM SAF20. April 2023 7
HDF command line tools
• No downward compatibility
many hdf5 command line tools and interfaces (e.g. implemented in f90,c programs) can not be used for HDF4 files.
• h5dump - dumps displays the input of the hdf file in ASCII
• h5ls - lists the contents of a file, enables fast checks if the needed data is in there
• h5import - imports ASCII to hdf5
configuration file is needed, hence some basic knowledge about HDF data model and structure required
Satellite Application Facility for Climate Monitoring – CM SAF20. April 2023 8
HDF5 as ASCII using h5dump
Common data model but in detail it can look quite different, comments in red !!
HDF5 "TRS_SR_20040708_1200_V000.hdf" { filename GROUP "/" { definition of a group GROUP "Data" { DATASET "TRS" { definition of the dataset DATATYPE H5T_STD_I16BE def. of the data type DATASPACE SIMPLE { ( 3712, 3712 ) / ( 3712, 3712 ) } the dimension DATA { the data (0,0): -32767, -32767, -32767, -32767, -32767, -32767, -32767, (0,7): -32767, -32767, -32767, -32767, -32767, -32767, -32767,…. (883,707): 495, 455, 436, 436, 378, 323, 378, 416, 342, 277, 296, ……} ATTRIBUTE "Gain" { ….. definition of attributes continued on the next slide
Satellite Application Facility for Climate Monitoring – CM SAF20. April 2023 9
HDF5 as ASCII using h5dump
ATTRIBUTE "Gain" { …. Gain and offset DATATYPE H5T_IEEE_F32BE used to reduce needed DATASPACE SCALAR disk space (possible to DATA { save data as integer) (0): 0.25 } } ATTRIBUTE "Offset" {….. DATATYPE and DATASPACE …. DATA { (0): 0 } }
ATTRIBUTE "nodatavalue" {…. DATATYPE and DATASPACE …… DATA { Attribute is also used (0): -32767 for unit, title,… }
Satellite Application Facility for Climate Monitoring – CM SAF20. April 2023 10
GROUP "Geolocation" { definition of a new group, and the DATASET "projection" { dataset needed to define the projection DATATYPE H5T_COMPOUND { H5T_STRING { STRSIZE 128; STRPAD H5T_STR_NULLTERM; CSET H5T_CSET_ASCII; CTYPE H5T_C_S1; } "reference ellipsoid"; H5T_ARRAY { [10] H5T_IEEE_F32LE } "parameter"; } DATASPACE SIMPLE { ( 1 ) / ( 1 ) } DATA { (0): { "geostationary view", "WGS-84", [ 1856, 1856, 667.204, 667.204, -1, -1, -1, -1, -1, -1 ] } }} DATASET "region" { a group usually consists of different datasets}}}
HDF5 as ASCII using h5dump
Satellite Application Facility for Climate Monitoring – CM SAF20. April 2023 11
HDF GUI Tools -HDFView-
• The complex data model might act as a deterrend for beginners
• Graphical User Interface HDFView overcomes this handicap. It is a tool for browsing and editing HDF4 and HDF5 files using a GUI
Relatively easy to install and available for many platforms, e.g Windows, Solaris, AIX, Linux
Everything can be managed with buttons and mouse clicks
Data can be saved as ASCII table
Images can be generated and saved.
• http://www.hdfgroup.org/hdf-java-html/hdfview/index.html
Satellite Application Facility for Climate Monitoring – CM SAF20. April 2023 12
HDF Tools – CMSAF GUI
• Software available for CM-SAF customers via www.cmsaf.eu
Features:• visualisation of CM-SAF products (in HDF5 format)• simple data analysis• Export (ASCII, lat/lon grid)
• Uses free IDL Virtual Machine
Satellite Application Facility for Climate Monitoring – CM SAF20. April 2023 13
HDF Tools CM-SAF GUI
Satellite Application Facility for Climate Monitoring – CM SAF20. April 2023 14
HDF Tools CM-SAF GUI
More on this topic in the exercise session
Satellite Application Facility for Climate Monitoring – CM SAF20. April 2023 15
netCDF
• Information, tools, examples and the netCDF library are available at: http://www.unidata.ucar.edu/software/netcdf/
• Widely used, e.g.:
Reanalysis data of National Centers for Environmental prediction (NCEP) and European Centre for Medium Weather forecast (ERA40)
HOAPS, Hamburg Ocean Atmosphere Parameters and Fluxes from Satellite Data
CM-SAF selected monthly means
Satellite Application Facility for Climate Monitoring – CM SAF20. April 2023 16
netCDF command line tools
• ncdump - file shows the input of the netCDF file
• ncgen - converts ascii to netcdf and vica versa
sounds easy but a configuration file (CDL file) is needed
some basic knowledge about the net CDF data model and structure
however, easier to handle for beginners as HDF5
example of ASCII CDL configuration file:
Satellite Application Facility for Climate Monitoring – CM SAF20. April 2023 17
netCDF as ASCII
netcdf SRBmm200604 {dimensions: lat = 501 ; lon = 741 ; time = UNLIMITED ; // (0 currently)variables: float lat(lat) ; lat:long_name = "latitude" ; lat:units = "degree" ; float lon(lon) ; lon:long_name = "longitude" ; lon:units = "degrees" ; float Z(lat, lon) ; Z:units = "Watt" ; Z:valid_range = 0., 1400. ;data:
lat = 35, 35.05, 35.1, 35.15, 35.2, 35.25, 35.3, 35.35, 35.4, … ; lon= 44,45,….; Z=300,340,…; }
Satellite Application Facility for Climate Monitoring – CM SAF20. April 2023 18
netCDF Tools – Integrated Data Viewer (IDV)
• Free GIS tool
• Display data / generate maps
• Imports netCDF
Satellite Application Facility for Climate Monitoring – CM SAF20. April 2023 19
netCDF GUI Tool CDAT
• Open source integrated environment for data analysis and visualisation.
Mainly netCDF, but can also deal with GRIB and HDF.
Import of binary and ASCII data possible.
Available for different platforms but not for Windows!
Satellite Application Facility for Climate Monitoring – CM SAF20. April 2023 20
Binary Data
• Usually used instead of ASCII
to reduce disk space and to increase the computing performance.
•Machine readable format not readable by humans
Usually files with / without header and data as defined data type e.g. float (2.44) or integer (4)
Reading and writing with e.g. C, C++, Fortran
Formats are not common indivdual read / write routines needed
some tools can read and visualise binary data. e.g.
CDAT, GRADS, idl
data is not self-explanatory The length of the header and the data type has to be known
Satellite Application Facility for Climate Monitoring – CM SAF20. April 2023 21
Binary Data and ASCIII
• Examples for binary data:
International Satellite Cloud Climatology Project (ISCCP). http://isccp.giss.nasa.gov
AVHRR based USGS land use maps.
Satellite Application Facility for Climate Monitoring – CM SAF20. April 2023 22
ASCII
• readable with a text editor
• a quite unusual format
• sometimes provided by the data centre for subsets of the data on request, e.g. CM-SAF
2006 9 27 6 0 71.932006 9 27 6 15 109.752006 9 27 6 30 73.282006 9 27 6 45 96.042006 9 27 7 0 84.162006 9 27 7 15 91.512006 9 27 7 30 110.542006 9 27 7 45 122.44 2006 9 27 8 0 166.66
Satellite Application Facility for Climate Monitoring – CM SAF20. April 2023 23
Conclusion
• HDF5
Header, describing the data. Data in binary format
HDF-View, CM-SAF GUI
Official format of CM-SAF Daten
• netCDF
Header, describing the data. Less cryptic than HDF5. Data in binary format
Diverse GIS, e.g. ArcView, Integrated Data Viewer, CDAT
On demand some CM-SAF data can be provided in netCDF.
• Binary
Instead of ASCII, to reduce disk space
• ASCII
Readable with a text editor.