Upload
tyrone
View
45
Download
0
Tags:
Embed Size (px)
DESCRIPTION
HDF4 OPeNDAP Project Progress Report. MuQun Yang and Hyo-Kyung Lee. OPeNDAP Review. A software framework that allows simple access to remote data D ata A ccess P rotocol Client-server model Widely used by Earth Science Community. OPeNDAP Reivew. Users. Hyrax. IDV. - PowerPoint PPT Presentation
Citation preview
HDF4 OPeNDAP ProjectProgress Report
MuQun Yang and Hyo-Kyung Lee
1HDF Developers' Meeting04/21/23
OPeNDAP Review
• A software framework that allows simple access to remote data• Data Access Protocol• Client-server model• Widely used by Earth Science Community
2HDF Developers' Meeting04/21/23
Users
OPeNDAP Clients
IDV
OPeNDAP Servers
HDF4 Files
Hyrax AIRS
DAP
libnc-dap
Visualization Tools
OPeNDAP Reivew
Client Library(libnc-dap)
DAP Protocol (via http)
Server(apache)
Handler(hdf4_handler)
Remote Data(HDF4)
View Data (User)
OPeNDAP Concept
4HDF Developers' Meeting04/21/23
Visualization Tools
(gradsdap)
Typical Remote Access
FTP/HTTP
Visualization tools
(e.g. grads)
Better Access for HDF-EOS2 via DAP
• Faster download through selection
• Quicker analysis through visualization
5HDF Developers' Meeting04/21/23
Goals
You bought a HDTV but...
• No HD signal
• Needs more Channels in HD
• Cable Service Provider Problem
6HDF Developers' Meeting04/21/23
HDTV Problems
You downloaded a viewer but...
• No CF convention
• Needs more HDF4 data in CF
• Data Service Provider Problem
7HDF Developers' Meeting04/21/23
HDF4 Problems
Challenges: HDF-EOS2
• CF convention (e.g. NetCDF model)
• Syntax
• Clients expect a certain data structure
• Semantics
• Clients expect some key attributes
8HDF Developers' Meeting04/21/23
HD Signal Challenge
Challenges: HDF-EOS2
• DVD/VHS/VCD/LD >>> Blue-Ray
• Can all titles be delivered in HD?
• HDF4/HDF-EOS2 >>> HDF5/HDF-EOS5
• Can all HDF-EOS2 be delivered in CF?
9HDF Developers' Meeting04/21/23
More Channels in HD Challenge
Solution: New HDF4 Handler
10HDF Developers' Meeting04/21/23
HD Signal Solution
• Syntax
Generate the correct DAP data type. Add shared geo-location information.
• Semantics Insert additional key attributes.
Solution: New HDF4 Handler
• Use Metadata parser
• Pro: simple
• Con: limited channels
• Use HDF-EOS2 library
• Pro: maximum channels
• Con: install HE2 library first
11HDF Developers' Meeting04/21/23
More Channels in HD Solution
Solution Comparison
12HDF Developers' Meeting04/21/23
Quick Implementation
• Parse Metadata – hdf5_handler
• Use HDF-EOS2 library – h4toh5
13HDF Developers' Meeting04/21/23
Current Status
• MODIS / AIRS Grid and Swath
• Other Projections (Polar, Sinusoidal)
• Testing
14HDF Developers' Meeting04/21/23
HDF-EOS2 in CF via DAP
• Finally, Happy Data Consumer!
15HDF Developers' Meeting04/21/23
AIRS Grid AIRS.2008.10.27.L3.RetStd001.v5.2.2.0.G08303124144.hdfhttp://daac.gsfc.nasa.gov/daac-bin/whom/mk_page_cgi.pl?PATH=datapool/AIRS/Level3_V005/01_AIRX3STD/2008/10MODIS GridMOD08_D3.A2008284.005.2008289105354.hdfhttp://modis-atmos.gsfc.nasa.gov/MOD08_D3/acquiring.html
AIRS Swath ftp://airspar1u.ecs.nasa.gov/data/s4pa//Aqua_AIRS_Level2/AIRX2RET.005//2008/330/
AIRS.2008.11.25.143.L2.RetStd.v5.2.2.0.G08331121849.hdfMODIS Swath
MOD06_L2.A2001222.0025.004.2003127111841.hdfhttp://modis-atmos.gsfc.nasa.gov/MOD06_L2/acquiring.html
Limitations – Missing Attributes
• MODIS Grid – Rain fall
16HDF Developers' Meeting04/21/23
No Unit!
• AURA OMI Grid - Ozone
Limitations – Missing Names
• How to Match Dimension Names?
17HDF Developers' Meeting04/21/23
HDF4 Handler
NetCDF4 Handler
pressureStd → Level
Limitations – Missing Values
• How to Supply Missing Dimension Values?
18HDF Developers' Meeting04/21/23
MISR_AM1_TC_CLASSIFIERS_P020_O000995_F06_0011.hdf HTTP/1.0 200 OKXDODS-Server: 3.7.9XOPeNDAP-Server: 3.7.9XDAP: 3.1Content-Type: text/plainContent-Description: dods_errorCache-Control: no-cache
Error { code = 1002; message = "An internal error was encountered:This HDF-EOS2 file has multiple Grids with different dimension sizes.Please report this to [email protected]";};
Solution – Customization
• Provide an option for handler customization to generate fully CF-compliant output
• Create a separate XML input file for handler and feed it into the handler.
• Insert attributes, renaming rules, and missing values by putting them in the XML input.
• This requires a knowledge about the characteristics of data in advance.
19HDF Developers' Meeting04/21/23
Solution – Customization
• Example: Attributes, Renaming, Values
20HDF Developers' Meeting04/21/23
<xml><variable name=‘RrLandRain’> <attr name=‘unit’ type=‘String’> inches</attr> <attr name=‘_FillValue’ type=‘Int16’>-1</attr></variable><rename from=‘GeoXTrack’ to=‘lon’ ></rename><rename from=‘pressStd’ to=‘lev’></rename><insert name=‘NAltitudeDim’ size=10 type=‘Int8’>0,1,2,3,4,5,6,7,8,9</insert></xml>
Summary
• HDF4 Access via OPeNDAP Is Easy, Efficient and Fast Is Cool like HDTV Adding CF is not straightforward
• Use HDF-EOS2 library Is Essential Maximize the data coverage
21HDF Developers' Meeting04/21/23
Future Work (Short Term)
• Expand test-suite with data pool
• Release through OPeNDAP website
• Support for Swath via Parsing
• Support for Point data type via HDF-EOS2 library
22HDF Developers' Meeting04/21/23
Future Work (Long Term)
• General HDF4 Files Support
• Customization for CF option
• HE2 to DAP Mapping Documentation
23HDF Developers' Meeting04/21/23