Embed Size (px)
DESCRIPTION
Easy Remote Access via OPeNDAP. Kent Yang and Joe Lee The HDF Group The 14 th HDF/HDF-EOS Workshop September 28, 2010. Easy. Outline. Introduction to OPeNDAP What’s New? HDF4-OPeNDAP Handler (major) HDF5-OPeNDAP Handler (minor). OPeNDAP Concept. Typical Remote Access. View Data - PowerPoint PPT Presentation
Citation preview
www.hdfgroup.org
The HDF Group
HDF/HDF-EOS Workshop XIV 1
Easy Remote Access via OPeNDAP
Kent Yang and Joe LeeThe HDF Group
The 14th HDF/HDF-EOS WorkshopSeptember 28, 2010
September 28,2010
www.hdfgroup.org
Easy
HDF/HDF-EOS Workshop XIV 2September 28,2010
www.hdfgroup.org
Outline
• Introduction to OPeNDAP
• What’s New?
oHDF4-OPeNDAP Handler (major)
oHDF5-OPeNDAP Handler (minor)
HDF/HDF-EOS Workshop XIV 3September 28,2010
www.hdfgroup.org
Client Library(libnc_dap)
Data Access Protocol (DAP) via Internet
Server(Hyrax)
Handler(hdf4/5_handler)
Remote Data(HDF4/5)
View Data (User)
OPeNDAP Concept
4
Visualization Tools
(gradsdap)
Typical Remote Access
FTP/HTTP
Visualization tools
(e.g., grads)
HDF/HDF-EOS Workshop XIVSeptember 28,2010
www.hdfgroup.org
Access HDF via OPeNDAP
HDF/HDF-EOS Workshop XIV 5
Users
OPeNDAP Clients
IDV
OPeNDAP Servers
HDF Files
Hyrax AIRS/OMI
DAP
libnc-dap
Visualization Tools
HDF HANDLER
Aqua/Aura
September 28,2010
www.hdfgroup.org
Outline
• Introduction to OPeNDAP
• What’s New?
oHDF4-OPeNDAP Handler (major)
oHDF5-OPeNDAP Handler (minor)
HDF/HDF-EOS Workshop XIV 6September 28,2010
www.hdfgroup.org
What’s New?
HDF4-OPeNDAP Handler Work
HDF/HDF-EOS Workshop XIV 7September 28,2010
www.hdfgroup.org
HDF4-OPeNDAP Handler Work
• Problems
• Our Work
• Limitations
HDF/HDF-EOS Workshop XIV 8September 28,2010
www.hdfgroup.org
Existing HDF4 Handler Problem
• Some OPeNDAP Visualization Tools
CAN NOT display most NASA HDF-
EOS2 and HDF4 data
HDF/HDF-EOS Workshop XIV 9September 28,2010
www.hdfgroup.org
Existing HDF4 Handler Problem
HDF/HDF-EOS Workshop XIV 10
Users
OPeNDAP Clients
IDV
OPeNDAP Servers
HDF4 Files
Hyrax AIRS Grid
DAP
libnc-dap
Aqua
Existing HDF4 HANDLER
Visualization Tools
September 28,2010
www.hdfgroup.org
HDF4-OPeNDAP Handler Work
• Problems
• Our Work
• Limitations
HDF/HDF-EOS Workshop XIV 11September 28,2010
www.hdfgroup.org
Our Work
• Let Visualization Clients Access HDF-EOS2 and HDF4 files
HDF/HDF-EOS Workshop XIV 12September 28,2010
www.hdfgroup.org
Why Visualization Fail?
• My house costs 1.
HDF/HDF-EOS Workshop XIV 13
• My house in Hawaii costs $1M. • Information – Earth Science - Content
• Geo-Location, Geo-Location, Geo-Location
• Unit, Scale factor
September 28,2010
www.hdfgroup.org
Why Visualization Fails?
• Clients expect certain data structureso Shared & named dimensions
• Clients expect some key attributeso CF conventions
o coordinates attribute
o unit attribute
HDF/HDF-EOS Workshop XIV 14September 28,2010
www.hdfgroup.org
Extreme Makeover
• My house in HI costs 1M.
HDF/HDF-EOS Workshop XIV 15
• My house in Hawaii costs $1M.• Enhanced HDF4 Hander• Interoperability!
September 28,2010
www.hdfgroup.org
Challenges
• Hundreds of HDF-EOS2 and HDF4 products
oGrid Projections
oSwath Dimension Map
oProduct Specific: TRMM / TOMS / MERRA /
CERES / Etc.
September 28,2010 HDF/HDF-EOS Workshop XIV 16
www.hdfgroup.org
Solution
• Step 1: Retrieve Key Information
• Step 2: Transform Output File Structure following CF conventions and DAP 2.0
September 28,2010 HDF/HDF-EOS Workshop XIV 17
www.hdfgroup.org
Step1: Retrieve Key Information
• Coordinate variable values in an HDF-EOS2 file
September 28,2010 HDF/HDF-EOS Workshop XIV 18
Calculate Lat/Lon
www.hdfgroup.org
Step1: Retrieve Key Information(continued)
• Coordinate variable values in an HDF4 file
September 28,2010 HDF/HDF-EOS Workshop XIV 19
Latitude and Longitude areProvided outside the file.
www.hdfgroup.org
Step2: Transform Output
• Flatten Structure and handle special characters
September 28,2010 HDF/HDF-EOS Workshop XIV 20
DAP DAS Output following CFAn HDF file
Flatten
CF attributes
www.hdfgroup.org
Shift of Burden
• From Users to Handler Developers
• Client processing to Server processing
September 28,2010 HDF/HDF-EOS Workshop XIV 21
www.hdfgroup.org
Enhanced HDF4 Handler
HDF/HDF-EOS Workshop XIV 22
Users
OPeNDAP Clients
IDV
OPeNDAP Servers
HDF4 Files
Hyrax
DAP
libnc-dap
Aura
MODIS Swathw/ Dim. Map
Enhanced HDF4 HANDLER
Visualization Tools
September 28,2010
www.hdfgroup.org
Demo: HDF-EOS2 Library Solution
HDF/HDF-EOS Workshop XIV 23
Swath with Dimension Map
September 28,2010
www.hdfgroup.org
HDF-EOS2 Products Supported
• GES DISC• AIRS / MERRA / TOMS
• LAADS / LP DAAC / NSIDC• Many MODIS products
• NSIDC• AMSR_E / NISE
• LaRC• MISR / MOPITT / some CERES(TRMM)
HDF/HDF-EOS Workshop XIV 24September 28,2010
www.hdfgroup.org
Pure HDF4 Products Supported
• GES DISC• TRMM L1B, L2B Swath • TRMM L3 Grid 42B & 43B products
• OBPG (Ocean Color)• SeaWiFS / MODIST / MODISA• CZCS / OCTS
HDF/HDF-EOS Workshop XIV 25September 28,2010
www.hdfgroup.org
Pure HDF4 Products Supported
• CERES• CER_AVG• CER_ES4• CER_ISCCP-D2like-Day• CER_ISCCP-D2like-GEO• CER_SRBAVG• CER_SYN• CER_ZAVG
HDF/HDF-EOS Workshop XIV 26September 28,2010
www.hdfgroup.org
HDF4-OPeNDAP Handler Work
• Problems
• Our Work
• Limitations
HDF/HDF-EOS Workshop XIV 27September 28,2010
www.hdfgroup.org
Their Limitations
• Reasons for Visualization Failures
• IDV / Panoply / Java Heap Memory
• Network / DAP (e.g., max HTTP header size)
• Can’t retrieve key information correctly (e.g.,
LAMAZ projection) from data
• Unable to harmonize with CF-conventions (e.g.,
FillValue inside latitude and longitude)
HDF/HDF-EOS Workshop XIV 28September 28,2010
www.hdfgroup.org
Our Limitations
• HDF-EOS2 + additional HDF4 objects
- Additional HDF4 objects will be ignored.• The rest of pure HDF4 products not listed in
previous slides are not tested and may not be visualized by Java OPeNDAP Clients.
September 28,2010 HDF/HDF-EOS Workshop XIV 29
www.hdfgroup.org
What’s New?
HDF5-OPeNDAP Handler Work
HDF/HDF-EOS Workshop XIV 30September 28,2010
www.hdfgroup.org
HDF5-OPeNDAP Handler Update
HDF/HDF-EOS Workshop XIV 31
Users
OPeNDAP Clients
IDV
OPeNDAP Servers
HDF5 Files
Hyrax OMI Grid
DAP
libnc-dap
Aura
OMI Swath
HDF5 HANDLER Update
Visualization Tools
September 28,2010
www.hdfgroup.org
HDF5-OPeNDAP Handler Update
• Support Aura HDF-EOS5 OMI Swath
• Support Aura HDF-EOS5 MLS
HDF/HDF-EOS Workshop XIV 32September 28,2010
www.hdfgroup.orgHDF/HDF-EOS Workshop XIV 33September 28,2010
OPeNDAP HDF4/HDF5 handlers
• Source codes and RPMs can be found under
http://opendap.org• Demo HDF5/HDF4 servers
http://hdfeos.org/zoo/hdf5_handler
http://hdfeos.org/zoo/hdf4_handler
www.hdfgroup.org
The HDF Group
HDF/HDF-EOS Workshop XIV 34
Thank you !
September 28,2010
www.hdfgroup.org
Acknowledgements
This work was supported by cooperative agreement number NNX08AO77A from the National
Aeronautics and Space Administration (NASA).
Any opinions, findings, conclusions, or recommendations expressed in this material are
those of the author[s] and do not necessarily reflect the views of the National Aeronautics and Space
Administration.
HDF/HDF-EOS Workshop XIV 35September 28,2010
www.hdfgroup.org
The HDF Group
Questions/comments?
HDF/HDF-EOS Workshop XIV 36September 28,2010
