Copyright 2017, The HDF Group.

The HDF GroupHow can we help you?

1

Collaboration 2

Self-Oriented Products:

For your use and publication

No user’s guide / no metadata

Custom software to access data

Reusable Products:

Outreach to potential users in advance.

Write a good user’s guide.

Add searchable metadata.

Test with existing software.

Interoperable Products:

Think beyond your community

Metadata for use and understanding

Test all software that you can envision.

Product Designer Hub

3

Data Store(users, projects, file

structures, metadata)

Desktop Client(Easy Install)

Python IDL MATLAB

Restful Server

ExportOptions

HDF5JSON

HDF4MAP

NcML(Remote)

HDF online(files, data)

HDF5Template

OPeNDAPDMR

CDLHDF5Excel

Text (CSV)DB

HDFJSON

Import Options

ProductSpecifications

4At HDFEOS.org, we LOVE Earth data!

• Solutions for NASA HDF-EOS data

• 150 visitors per day

• 847 registered forum users

• Goal: Make Earth data access easy.

hdfeos.org 5

100+ Tool and Library examples:

ArcGIS, Google Earth, Panoply, IDV, NCO,

GrADS, ...

GDAL, C, Java, Python, MATLAB, R, Scala,

IDL, NcML, …

300+ NASA HDF Product examples

MATLAB/Python/IDL/NCL visualization

Other Cloud / Bigdata solutions

Spark+Hadoop / Elastic Search / Amazon Web

Service

HDF Tools 6

Distribution Discovery AnalysisArchive

Repurposing

Product

Designer

Data Structures

Metadata &

Convention

Support

ISO, HDF-EOS,

ACDD, CF

HDFView (HDF-

EOS plugin)

HDFCompass

HDF-EOS Dumper

IDL/ENVI, MATLAB,

Panoply, IDV, Ferret,

Python, User support

HDF4 Mapper OPeNDAP

Handlers,

H5Server,

ncISO,

THREDDS

Climate-Forecast

(CF)

Conventions

netCDF API

HDF-EOS

HDF4 to HDF5

Library / Tools

Metadata / Data

AugmentationProduct (Re-

)Designer

HDF4, HDF5 &

HDF-EOS

Libraries

IDL

Supporting the entire data life cycle

ProductGeneration

Mission Planning and

Design

HDF5 Tools 7

GUI Text

Built in Java Built in C

Separate release Part of the library release

HDF4, HDF5, NetCDF, etc. HDF5 only

HDFView HDF5 Command-line tools

HDFView - VIIRS Day/Night band files 8

• floating

windows

• multiple files

and datasets,

• bigger

datasets,

• faster and

consumes less

memory.

HDF5 command-line tools 9

Tool Purpose

h5dump, h5ls Display content of HDF5 file

h5diff Compares structure and data of two HDF5 files

h5copy Copies the whole of parts of the HDF5 file

h5repack Repacks HDF5 file (can change storage layout,

compression and other properties; can use

custom compression via dynamically loaded

filters)

HDF4 to HDF5 conversion tool

Although we continue to maintain HDF4, we do not plan on adding new features to

HDF4, and recommend users move to HDF5 to take advantage of the improved

features and performance of HDF5. This standard tool can be used to:

• Convert any HDF4 files to HDF5 files using HDF4 to HDF5 Mapping Specification

• Package URL: https://support.hdfgroup.org/products/hdf5_tools/h4toh5/

• Example HDF5 files converted from the NASA HDF4 and HDF-EOS2 products:

http://hdfeos.org/zoo/h4toh5/h4toh5-def-demo.php

• Guaranteed to preserve data content and data structures

• New release during the summer of 2017

HDF4 to CF conversion toolkit

Need to connect HDF4 data to the netCDF Library?

• A desktop version of the HDF4 OPeNDAP handler.

• Apply to NASA HDF-EOS2 and some NASA HDF4 products.

• The output files can be either netCDF-3 or netCDF-4 classic files and the

CF conventions are enforced.

• Package URL: http://hdfeos.org/software/h4cflib.php

• Example HDF5 files converted from the NASA HDF4 and HDF-EOS2

products: http://hdfeos.org/zoo/h4cf/index.php

More Information: http://hdfeos.org/software/aug_eos5.php

HDF5

HDF-EOS5

HDF5

netCDF4

Augmentation

HDF-EOS5 file

HDF-EOS5 Augmentation Tool

• Make HDF-EOS5 files accessable by netCDF4 using the netCDF data model.

HDF5 tools for working with NPP files

ADL 4.2 Workshop

13

January 8, 2014

• nagg allows reorganizing already downloaded data

• h5edit provides command line editing of HDF5 file

metadata

• h5augjpss (experimental) provides options to import

metadata from product profiles and remove obstacles

for netCDF-4 tools

Example: aggregate 36 granules from 9 files 14

nagg –n 36 –t SVM07 GMODO-

SVM07_npp_d20121028_*.h5

15HDF-EOS Examples Future Directions:

NASA Developer Portal + CMR

Jupyter Notebook Python Examples

• Search & query collection and granules from Common Metadata

Repository (CMR).

• Access data using OPeNDAP using the query result.

Future: CMR Ingestion and Testing Example

• Integration with HDF Product Designer?

Future Work 16

• Product Designer

• h4h5diff

• OPeNDAP WOS Support

• HDF5 Handler improvements

• DAAC User Needs?

• Join the discussion at the HDF-EOS Forum

Sustainable Archives? 17

• ODL to Groups/Attributes

• Metadata to ISO

• Conventions to HDF-EOS (?) / CF

• Migration from HDF4 to HDF5

• Data to the Cloud?

• HDF4 chunks/bytestreams > objects?