HDF5 OPeNDAP Project

MuQun Yang and Hyo-Kyung Lee (THG)

James Gallagher (OPeNDAP, Inc.)

Question of the Day

• How can I get One Sub from SUBWAY?• Chicken teriyaki with no onion

• How can I get One Subset of EOS data image from NASA?• Today’s satellite image right above the roof

Subway

• Easy: anywhere / accepts Visa|MC

• Efficient: your own recipe• Cool: eat fresh and look good

How can I get One Sub from SUBWAY?

Dumb-way

• Difficult: bring a USB drive and visit NASA

• Inefficient: download 10 Gbyte and search for 10 byte dataset

• Not cool: wrong or useless dataset

How can I get One Subset of EOS data image from NASA?

DAP-way!

• Easy: anywhere / accepts IE|FF

• Efficient: WYSIWYG – less fat!

• Cool: Visualization clients – see fresh!

How can I get One Subset of EOS data image from NASA?

DAP-way Implementation

• OPeNDAP: Free DAP-way

Software

• Server-Client Model APIs

• Widely used by Earth Science

Community

OPeNDAP

• Market Dominance:

•

• http://www.opendap.org

OPeNDAP server sites

Courtesy of OPeNDAP, Inc.

OPeNDAP Servers

Data Data Data Data Data Data Data

Matlab

HDF4 JDBC

FreeFromFITS

CDF CEDAR

Data

netCDF

netCDF HDF4 Matlab

Data

DSP

DSP

Data

JGOFS

Tables SQL FITS CDFFlat

Binary CEDAR

Data

CODAR

Data

ESML

GeneralCODAR

Courtesy of OPeNDAP, Inc.

OPeNDAP Clients

netCDF C netCDF Java

IDVFerret GrADS VisAD ncBrowse Matlab ExcelIDL Access

MatlabClient

IDLClient

Courtesy of OPeNDAP, Inc.

How OPeNDAP works

OPeNDAP Client

OPeNDAPHDF4 Server

HDF4

OPeNDAPNetCDF Server

NetCDF

Architecture of OPeNDAP Server

• Server 3 (CGI)

• C/C++ based HTTP Server

• Insecure / Inflexible

• Server 4 (Hyrax)

• Java based TomCat Servlet Engine

• Secure / Flexible

Server 3 (CGI) Architecture

OLFS

Java Servlet Engine

BES

Unix Daemon

BES Commands

XML- encapsulated object

DataStore

File system with data files like HDF, NetCDF and SQL Database, …

DAP

XML, GIS, KML

HTML

Optional catalogs: XML, GIS, KML

Server 4 (Hyrax) ArchitectureHTTP Server

Courtesy of OPeNDAP, Inc.

http://docs.opendap.org/index.php/Hyrax

Example Usage

OPeNDAP Client

OPeNDAPHDF4 Server

HDF4

Syntactic Structure of DataSemantic Meaning of Data

Actual Content of Data

OPeNDAP Visualization Client

What about HDF5?

OPeNDAP Client

OPeNDAPHDF4 Server

HDF4

OPeNDAPNetCDF Server

NetCDF

OPeNDAPHDF5 Server

HDF5

Why Important?

• HDF5: NASA EOS / NPOESS

• NASA’s MISSION: ACCESS!

• Our MISSION: Build DAP-way Bridge

Goals

•1st: Transform HDF5 access in DAP-way

• 2nd: Yet preserve the beauty of HDF5

OPeNDAP HDF5 Server

1. Map HDF5 Datatypes in DAP-way

2. Plus, tweaks for OPeNDAP Clients

3. Plus, tweaks for HDF-EOS Files

Goal: Transform HDF5 access in DAP-way

Mapping HDF5 in DAP-way• Prototype server in 2001

• NASA Grant from Nov. 2006

• Support for Compound Datatype

• Support for Group

• Support for References / Links

• First Product Release in Mar. 2008

OPeNDAP HDF5 Server

1. Map HDF5 Datatypes in DAP-way

2. Plus, tweaks for OPeNDAP Clients

3. Plus, tweaks for HDF-EOS Files

Goal: Transform HDF5 access in DAP-way

Subway Customers

• I want 3 inch-long sub

• I need Wasabi sauce

• I eat only wheat bread

• I eat only meat balls

OPeNDAP Clients

• I hate foot-long variable names.

• I need special attributes on dataset.

• I care pre-defined Grid data type.

• I care only well-formed attributes.

Not all OPeNDAP clients are created equal!

Tweaks for OPeNDAP Clients Two configuration options

• --enable-short-name = cut bread

• --enable-CF = put Wasabi sauce

OPeNDAP HDF5 Server

1. Map HDF5 Datatypes in DAP-way

2. Plus, tweaks for OPeNDAP Clients

3. Plus, tweaks for HDF-EOS5 Files

Goal: Transform HDF5 access in DAP-way

Subway Customers

• I want 3 inch-long

• I need Wasabi sauce

• I eat only wheat bread

• I eat only meat balls

Subway Suppliers Problem

• No breads, only wheat!

• No meat balls, only meat!

• I eat only wheat bread

• I eat only meat balls

Subway Customers - OK

HDF EOS

• Grid with No Dimension data

• Clients expect Grid with Dimension data

• Metadata Attribute in Two Strings

• Clients expect One Structured Format

Not all HDF5 files are created for DAP-way!

Tweaks for HDF-EOS Two more configuration options

• --enable-eos-grid = bake bread

• --enable-eos-meta = make balls

OPeNDAP HDF5 Server

1. Map HDF5 Datatypes in DAP-way

2. Plus, tweaks for OPeNDAP Clients

3. Plus, tweaks for HDF-EOS5 Files

Goal: Transform HDF5 access in DAP-way

Day After Server Tweaks

• Finally, Happy Clients!

GrADSNCLFerretMATLAB

ncBrowse

IDVODC

Live Demo

Ozone concentration level over the south pole

Goals

•1st: Transform HDF5 access in DAP-way

• 2nd: Yet Preserve the beauty of HDF5

HDF5 Served in DAP-way (Yet Preserve the beauty of HDF5)

• There are some things money can’t buy. For everything else, there is MasterCardTM.

• There are some things DAP can’t serve. For everything else, there is _________?

Minimize

Maximize

Some Things

• Hard: Opaque, Bitmap, Enum, 64 bit Integer, Variable Length types

• Illegal: Reserved characters in DAP are used in Dataset/Group name in HDF5

(that OPeNDAP HDF5 server can’t serve)

For Everything Else

• Let there be HDF5-friendly OPeNDAP client library!

• Package and Deliver in DAP-way first.

• Then, let the client library handle it.

• Concept of Trojan Horse

Example: Group in HDF5Traditional OPeNDAP client library: It’s an attribute that I don’t understand. I’ll ignore it.

HDF5-friendly OPeNDAP client library: I was waiting for this key attribute to re-construct HDF5

Example: Reference in HDF5

• Object / Regional Reference

• Map to DAP URL at server

• No de-referencing of URL at client library

• Important for NPOESS

HDF5-Friendly OPeNDAP Client Library

• One more reason: Tame Clients

• Only ODC can display Swath properly.

• Easy but Evil OPeNDAP NC Client Library! easy: nc style API evil: fixed dimensional attributes

• Our client library must be Easy but Good

No Latitude and Longitude

Courtesy of NASA

libnc-dap

???

liboc-dap

NC-Friendly OPeNDAP Client Library

libdap

dapserver

OPeNDAP NC Server

GrADS

NetCDFView NetCDFHDF5

Group/Ref/Swath

OPeNDAP HDF5 Server

View HDF5

GrADS

View G/R/Swath

HDF5-Friendly OPeNDAP Client Library

Summary

• DAP-way access of HDF5 Is Easy / Efficient / Cool Loses some things Requires HDF5-friendly OPeNDAP Client Library

• HDF5-friendly OPeNDAP Client Library Serves HDF5 better Tames evil visualization clients

Future Work• HDF5 DAP Mapping Document

• Finish HDF5-friendly OPeNDAP Client Library prototype

• Test it on GrADS and display Swath

Project Website

http://hdfdap.hdfgroup.uiuc.edu/joomla

• Beta preview

• Feedbacks are more than welcome!

Credits

Mike Folk (THG)Robert McGrath (NCSA)

Peter Leonard, Daniel Kahn, Marghi Hopkins (ADNET)Christopher Lynnes, James Johnson, Denis Nadeau (NASA)

Jennifer Adams (GrADS)Dave Brown (UCAR)