Upload
dorthy-cunningham
View
222
Download
1
Tags:
Embed Size (px)
Citation preview
HDF UpdateHDF Update
Mike Folk, Kent Yang, Elena Pourmal
The HDF Group
March 31st, 2009
March 31, 2009 Annual HDF Briefing to ESDIS 1
• HDF Project Update• Helpdesk• HDF Releases• Library release highlights • Quality Assurance• Java Product Highlights• Other Tools Activities
• Outreach • Related projects and activities
• EOS Support• OPeNDAP• netCDF Support
• Other projects of interest• Platform support issues
HDF Project UpdateHDF Project Update
March 31, 2009 Annual HDF Briefing to ESDIS 3
HelpdeskHelpdeskstatisticsstatistics
March 31, 2009 Annual HDF Briefing to ESDIS 4
Number of HDF4 and HDF5 issues Number of HDF4 and HDF5 issues since 2000since 2000
March 31, 2009 5Annual HDF Breifing to ESDIS
HDF4 issues by categoryHDF4 issues by category
March 31, 2009 6Annual HDF Breifing to ESDIS
HDF5 issues by categoryHDF5 issues by category
March 31, 2009 7Annual HDF Breifing to ESDIS
HDF issues by domainHDF issues by domain
March 31, 2009 8Annual HDF Breifing to ESDIS
HDF issues resolution timeHDF issues resolution time
March 31, 2009 9Annual HDF Breifing to ESDIS
HDF open and closed bugsHDF open and closed bugs
March 31, 2009 10Annual HDF Breifing to ESDIS
HDF5 open and closed bugsHDF5 open and closed bugs
March 31, 2009 11Annual HDF Breifing to ESDIS
HDF4 open and closed bugsHDF4 open and closed bugs
March 31, 2009 12Annual HDF Breifing to ESDIS
QUESTIONS?QUESTIONS?
March 31, 2009 Annual HDF Breifing to ESDIS 13
HDF RELEASESHDF RELEASES
March 31, 2009 Annual HDF Breifing to ESDIS 14
HDF Releases in 2008 - 2009HDF Releases in 2008 - 2009
• HDF4• HDF4r2.4 (February 2009)
• HDF5• HDF5-1.8.1 (May 2008)• HDF5-1.6.8 (November 2008)• HDF5-1.8.2 (November 2008)
• HDF- Java Products• Version 2.5 (February 2009)
• H4toH5• Version 2.0 (June 2008)• Version 2.1 (February 2009)
March 31, 2009 15Annual HDF Breifing to ESDIS
HDF LIBRARIES HDF LIBRARIES RELEASE HIGHLIGHTSRELEASE HIGHLIGHTS
March 31, 2009 Annual HDF Breifing to ESDIS 16
HDF4r2.4HDF4r2.4
• Fixed a flaw in the HDF4 design exposed by applications that use HDF-NetCDF interface to access both HDF4 and NetCDF-3 files with datasets/variables with unlimited dimensions.
• Added several new APIs including API to get compressed size of data.
• Updated hdp and hrepack to display compression information.
• Added support for Fedora 10, Ubuntu 8.10, and OpenSuse 11.1 with GNU C and Fortran.
• Better support for Mac OSX (GNU compilers and 32-bit Intel).
• http://www.hdfgroup.org/release4/obtain.html
March 31, 2009 17Annual HDF Breifing to ESDIS
HDF5-1.6.8 HDF5-1.6.8
• Many improvements in tools to address ESDIS requests• Availability of compression information in h5dump• Files comparison with h5diff• Support for strings in h5import• Support for user block in h5repack
• Added compatibility macros (e.g., H5Gcreate1) for applications to work the same with 1.6.* and 1.8.*
• Support for object and region references in C++• Bug fixes•
http://www.hdfgroup.org/HDF5/release/obtain516.html
March 31, 2009 18Annual HDF Breifing to ESDIS
HDF5-1.8.1&2HDF5-1.8.1&2
• Tools improvements • Support for external links• Support for a simplified way to produce binary output by
h5dump (ESDIS)• Better support for user block with h5repack
• Improved performance for free space tracking in a file
• Fixed bugs with file mounting• Many improvements in configuration, installation
and testing • Better support for shared C, Fortran and C++
libraries • http://www.hdfgroup.org/HDF5/release/obtain5.ht
ml
March 31, 2009 19Annual HDF Breifing to ESDIS
HDF Java ProductsHDF Java Products
• New released is based on JDK 1.6.x, HDF4r2.4 and HDF5-1.8.2 with 1.6 compatibility
• Java products (i.e. HDFView) can read files created by HDF5 1.8.2
• New HDFView:• Supports access to remote HDF5 files stored at the iRODS
server (Windows and Linux only). For information on HDF5-iRODS module, see http://www.hdfgroup.org/projects/irods/index.html.
• Shows large dataset (e.g. 1,000,000 x 100) in a table viewer• Converts JPEG, GIF, PNG, and BMP images to HDF5 or
HDF4• Saves HDF4 or HDF5 images to JPEG, GIF, PNG, and BMP
images• Writes variable length string data • Filters fill values for images
March 31, 2009 20Annual HDF Breifing to ESDIS
H4toH5 tools and libraryH4toH5 tools and library
• Based on HDF4r2.4 and HDF5-1.8.2• h4toh5 utility
• Recognizes HDF-EOS2 files (--with-hdfeos2 configuration option)
• Can generate HDF5 files that can be read by netCDF-4
• h4toh5 library• Bug fixes• Performance improvements
March 31, 2009 21Annual HDF Breifing to ESDIS
QUALITY ASSURANCEQUALITY ASSURANCE
Some work of interest to ESDIS
March 31, 2009 Annual HDF Breifing to ESDIS 22
QA EffortsQA Efforts
• We test at least 250 different configurations for HDF software (OSs x compilers x HDF configurations) every day• Regression testing of HDF5 released branches
(1.6. and 1.8) and development branch (1.9), and HDF4
• Platforms• Linux 32 and 64, Solaris 32 and 64, Free BSD 32 and 64,
AIX 32 and 64 (at LLNL), Windows 32 and 64, Linux Clusters, SGI Altix
• Backward/forward compatibility testing• Third part software testing
March 31, 2009 23Annual HDF Breifing to ESDIS
Backward/forward compatibility Backward/forward compatibility testingtesting
• Goal: assure interoperability between 1.6 and later versions of the HDF5 library used to create AURA files• Regression tests with and without strict format
checks using known problem files (currently MLS Cloud series with corrupted headers)
• Download sample from subdirectories of OMI, MLS, HIRDL, TOMS and TES files. Sample consists of files from the current day of the year for available years in two of the subdirectories listed for the above instruments.
March 31, 2009 24Annual HDF Breifing to ESDIS
Backward/forward compatibility Backward/forward compatibility testingtesting
• FTP sites (about 540 000 files)• OMI Level 2 files
• ftp://aurapar2u.ecs.nasa.gov
• OMI Level2G, MLS and HIRDL files• ftp://acdisc.gsfc.nasa.gov
• TOM files• ftp://acdisc.sci.gsfc.nasa.gov
• TES files• ftp://l4ftl01.larc.nasa.gov
March 31, 2009 25Annual HDF Breifing to ESDIS
Backward/forward compatibility Backward/forward compatibility testingtesting
• Each known subdirectory will be visited ~5 times per year.
• hdf5 tools from versions 1.6.9, 1.8.3, and 1.9.0 are checked for proper operation with downloaded files.
• Downloaded files are copied with h5copy and repacked with h5repack. The resulting files are checked again with all 3 versions of the tools.
March 31, 2009 26Annual HDF Breifing to ESDIS
Third party software testingThird party software testing
• NetCDF-4• Daily tests of the netCDF4 snapshot• Built against current HDF5 v1.8.*• Tested on Linux 32/64 bit and solaris
• HDF-EOS2 and HDF-EOS5 testing• HDFEOS2 with h4cc, 32/64 bit Linux, 32 bit solaris• HDFEOS5 with hdf5 v1.6 and v1.9, 32/64 bit Linux,
32/64 bit solaris
• IDL wrappers• Created IDL/HDF module binaries based on HDF5
1.8.2 and HDF4.2r3
March 31, 2009 27Annual HDF Breifing to ESDIS
QUESTIONS?QUESTIONS?
March 31, 2009 Annual HDF Breifing to ESDIS 28
JAVA PRODUCT JAVA PRODUCT HIGHLIGHTSHIGHLIGHTS
March 31, 2009 Annual HDF Briefing to ESDIS 29
HDF Java Products HighlightsHDF Java Products Highlights
• V2.5 released in February 2009• 50 bug fixes/enhancements• Support for HDF5 1.8-with-1.6 compatibility
(Does not support new HDF5 1.8 features)
March 31, 2009 30Annual HDF Briefing to ESDIS
HDFView HighlightsHDFView Highlights
• Support for iRODS• Show large dataset in table• Write variable length string• Convert image to/from HDF4/5
• JPEG• GIF• PNG• BMP
March 31, 2009 31Annual HDF Briefing to ESDIS
Future work for JavaFuture work for Java
• 26 bugs/enhancements• hdf-java 2.6 release
• Include latest HDF4 and HDF5 1.8 libraries• Fix bugs and add new enhancements
• Full support for HDF5 1.8 (next slide)
March 31, 2009 32Annual HDF Briefing to ESDIS
Future work for Java Future work for Java
• Full support for HDF5 1.8 • Add HDF5 1.8 APIs to the HDF5 Java wrapper
• Initial implementation (done)• Review and revision is in progress
• Implement and test HDF5 JNI C layer• Implementation (200+ APIs), 1 month fte• Testing, 1 month fte
• Support new features in HDF5 1.8• Revise hdf-java object package• Add new features to HDFView
33Annual HDF Briefing to ESDISApril 18-19, 2007
March 31, 2009 Annual HDF Briefing to ESDIS 34
Project Plan for 1.8 support Project Plan for 1.8 support
OTHER TOOLS OTHER TOOLS ACTIVITIESACTIVITIES
March 31, 2009 Annual HDF Briefing to ESDIS 3535
Improvements in HDF4 toolsImprovements in HDF4 tools
• Display compression information in hdf4 tools• Preserve unlimited dimensions in hrepack• Fix bugs in hdp and hdiff reported by DAACs.
March 31, 2009 36Annual HDF Briefing to ESDIS
HDF5 tools highlightsHDF5 tools highlights
• 35 bug fixes/enhancements to HDF5 tools• All bug fixes/enhancements are applied to
both HDF5 1.6 and 1.8 release
March 31, 2009 37Annual HDF Briefing to ESDIS
Highlights of HDF5 Tools Highlights of HDF5 Tools enhancementsenhancements
• h5dump • Display compression information • Follow external links• Add ability to specify precision output• Use MEMORY option for default binary output
• h5diff• Display the precision the differences• Handle NaN values
• h5repack • Keep the original user block information• Multiple filters can be invoked
March 31, 2009 38Annual HDF Briefing to ESDIS
OutreachOutreach
March 31, 2009 Annual HDF Briefing to ESDIS 39
March 31, 2009 Annual HDF Briefing to ESDIS 40
Earth Science Related OutreachEarth Science Related Outreach
• IEEE Transactions on Geosciences and Remote Sensing, vol. 47, issue 1 • Paper: “Ensuring Long Term Access to Remotely Sensed Data with
Layout Maps” by Duerr, Cao, Folk, Lynnes, Yang.• ESIP Federation Summer meeting, July 2008
• Presented a poster about OPeNDAP work• Meeting with ITTVis, Oct. 2008
• Presentation about HDF• Worked with them to test the IDL-HDF5 module with a new HDF5 release
• 7th NASA ESDSWG conference, Oct. 2008• Reported on the OPeNDAP project, participated in several working
groups• AGU Fall meeting, Dec. 2008
• Presentation: “Ensuring Long Term Access to Remotely Sensed HDF4 Data with Layout Maps,” by Folk and Duerr
• American Meteorological Society Annual Meeting, Jan. 2009• Presentations on OPeNDAP and the HDF-EOS2 to netCDF-4 conversion
tool
April 1-2, 2008 Annual HDF Briefing to ESDIS 40
HDF-EOS Workshop XII HDF-EOS Workshop XII
• Tutorials• Introduction to HDF5 Data and Programming Models • Advanced HDF5 Features• Migrating from HDF5 1.6 to 1.8• Introduction to HDF5 Tools• HDFView and HDF Java Products• What will be new in HDF5
• Status• HDF Status and Developments• HDF Group Support for NPP/NPOESS
• Applications and discussions• NOAA Science Data Stewardship project update• HDF5-OPeNDAP Project Update and Demo• Independent HDF4 mapping project update• HDF5 IRODS• Users: What are your needs• No programming language left behind
March 31, 2009 41Annual HDF Briefing to ESDIS
Other outreach - sampleOther outreach - sample
• Presentation “HDF5: Toward a Universal Platform for Science and Engineering Data,” Folk• National Renewable Energy Lab Scientific and Technical Data
Management and Analysis Workshop ’08, May 2008• “HDF Experiences with I/O Bottlenecks,” Folk
• NSF/NARA Collaborative Expedition Workshop: Toward Scalable Data Management – Overcoming I/O Bottlenecks, June 10, 2008
• HDF5 Tutorial, Pourmal and Cheng• 37th SPEEDUP Workshop on HPC, Zurich, Sept. 2008
• “HDF5: Past, Present, and Future”“Tuning Parallel HDF5 for HPCApplications,” Koziol, Aydt• National Energy Research Scientific Computing Center HDF5
Workshop on Scalable Performance
March 31, 2009 42Annual HDF Briefing to ESDIS
Related projects and Related projects and activitiesactivities
March 31, 2009 Annual HDF Briefing to ESDIS 43
EOS SUPPORTEOS SUPPORT
March 31, 2009 Annual HDF Briefing to ESDIS 44
EOS supportEOS support
• HDF-EOS2 and HDF-EOS5• Automatic configuration with szip enabled/disabled
• EOS2 and EOS5 are tested daily with HDF4 and HDF5 development code
• HDF-EOS website• workshop coordination • EOS product release
March 31, 2009 45Annual HDF Briefing to ESDIS04/19/23 The HDF Group for 2008 annual briefing 45
OPENDAPOPENDAP
March 31, 2009 Annual HDF Briefing to ESDIS 46
OPeNDAP – current statusOPeNDAP – current status
• HDF5-OPeNDAP handler• Served OMI and TES data
• HDF5 –awareness OPeNDAP client library• HDF4-OPeNDAP handler
• Tested with AIRS data and some MODIS data
• More information in the afternoon session
March 31, 2009 47Annual HDF Briefing to ESDIS04/19/23 The HDF Group for 2008 annual briefing 47
NETCDF SUPPORTNETCDF SUPPORT
March 31, 2009 Annual HDF Briefing to ESDIS 48
NetCDF4 supportNetCDF4 support
• Help Unidata NetCDF4 developers• Frequent communications with NetCDF4
developers
• Test NetCDF4 daily with the HDF5 development code
• NetCDF4 performance benchmark report• http://www.hdfgroup.org/pubs/papers/
2008-06_netcdf4_perf_report.pdf
March 31, 2009 49Annual HDF Briefing to ESDIS04/19/23 The HDF Group for 2008 annual briefing 49
LONG TERM ACCESS TO LONG TERM ACCESS TO REMOTELY SENSED REMOTELY SENSED
HDF4 DATA WITH HDF4 DATA WITH LAYOUT MAPSLAYOUT MAPS
Discussed later
March 31, 2009 Annual HDF Briefing to ESDIS 50
Other projects of InterestOther projects of Interest
March 31, 2009 Annual HDF Briefing to ESDIS 51
March 31, 2009 Annual HDF Briefing to ESDIS 52
NOAA SDSNOAA SDS
April 1-2, 2008 Annual HDF Briefing to ESDIS 5204/19/23 The HDF Group for 2008 annual briefing 52
• Enhanced HDF4 to HDF5 conversion tool• An HDF-EOS2 file can be converted to a netCDF-4 file• Accumulated knowledge and experience to handle
HDF-EOS2 data
Other projectsOther projects
• “NPOESS HDF5 Development & Support”• “BioHDF – HDF5 for next gen sequencing”
• Human Genome Research Institute SBIR with Geospiza, Inc.
• “HDF support for urban military operations”• Army Engineering Research and Development
Center – Topographic Engineering Center
March 31, 2009 53Annual HDF Briefing to ESDIS
March 31, 2009 Annual HDF Briefing to ESDIS 54
HDFView ERDC Concept Map Plugin HDFView ERDC Concept Map Plugin
Platform support issuesPlatform support issues
March 31, 2009 Annual HDF Briefing to ESDIS 55
HDF4 Platforms SupportedHDF4 Platforms Supported
• Systems• AIX 5.2 (32-bit, 64-bit)• Free BSD 6.2 (32-bit, 64-
bit)• IRIX 64 v6.5 (32-bit, 64-bit)• Linux 2.4, 2.6 (RH3)• Linux ia64• Linux x86_64 (SUSE)• Sun OS 5.8, 5.10 (32-bit,
64-bit)• Windows XP, Vista• Windows XP 64-bit*• Mac OS X Intel
• * - new platform
• Compilers• IBM C and Fortran
compilers• GNU gcc 3.4.* and GNU
Fortran• GNU gcc 3.4 and 4.*• PGI v 7.* compilers• Intel C and Fortran
versions 9.1 and 10.00• SUN WorkShop C and
Fortran• Visual Studio .NET and
2005 and Intel Fortran• Visual Studio 2005 (no
fortran)• GNU gcc 4.0.1 with
gfortran and g95
March 31, 2009 Annual HDF Briefing to ESDIS 56
HDF4 Platforms to drop/addHDF4 Platforms to drop/add
• Drop• Windows XP with
MSVC++ 6.0• Linux 2.4• IRIX64 6.5• SunOS 5.8, 5.9• HP 11.23 (no access)• Sun on Intel (no
access)
• Add ?• XT3 • Blue Gene • Blue Water• Linux flavors?• ?
March 31, 2009 Annual HDF Briefing to ESDIS 57
HDF5 Platforms SupportedHDF5 Platforms Supported
• Systems• AIX 5.2 (32-bit, 64-bit)• Free BSD 6.2 (32-bit, 64-bit)• IRIX 64 v6.5 (32-bit, 64-bit)• Linux 2.4, 2.6 (RH3)• Linux ia64• Linux x86_64 (SUSE)• Sun OS 5.8, 5.10 (32-bit, 64-
bit)• Cray XT3• Windows XP, Vista• Windows XP 64-bit*• Mac OS X Intel• Alpha Open VMS*
• * - new platform
• Compilers• IBM C and Fortran compilers• GNU gcc 3.4.* and GNU
Fortran• GNU gcc 3.4 and 4.*• PGI v 7.* compilers• Intel C and Fortran versions
9.1 and 10.00• SUN WorkShop C and Fortran• Visual Studio .NET and 2005
and Intel Fortran• Visual Studio 2005 (no fortran)• GNU gcc 4.0.1 with gfortran
and g95 • Compaq C and C++
compilers, HP Fortran compiler
March 31, 2009 Annual HDF Briefing to ESDIS 58
HDF5 Platforms to drop/addHDF5 Platforms to drop/add
• Drop• Windows XP with
MSVC++ 6.0• Linux 2.4• IRIX64 6.5• SunOS 5.8, 5.9• HP 11.23 (no access)• Sun on Intel (no
access)
• Add ?• XT3 • Blue Gene • Blue Water• Linux flavors?• ?
March 31, 2009 Annual HDF Briefing to ESDIS 59
Platform support questionsPlatform support questions
• Platform to support for HDF Java products• Windows XP/Vista• Solaris• Linux• Mac Intel• Mac PowerPC• Linux x86_64• Linux ia64• Solaris 64-bit Sparc
• Any to add? Any to drop?
March 31, 2009 60Annual HDF Briefing to ESDIS