View
2
Download
0
Category
Preview:
Citation preview
Proprietary and Confidential. Copyright 2016, The HDF Group.Proprietary and Confidential. Copyright 2016, The HDF Group.
HDF Group Past, Present & Future
1
Outline
2
Who we are and where we come from
Mission and commitments to community
Future directions for HDF Group & HDF R&D
Who we are and where we come from
National Center for Supercomputing Applications - 19874
Big simulations
Desktop visualization
The birth of HDF 5
6
Version 1.0
7
NASA Earth Observing System
• 6,700 Data Products • 12 Data Archive Centers
• 16 terabytes per day • Distributed to 1.7 million
end users worldwide
Non-destructive testing - TRAPPIST
9
Shortcomings of HDF (1996)
Limits on object & file size (<2GB)
Limited number of objects (<20K)
Rigid data models
I/O performance
Code complexity
Accelerated Strategic Computing Initiative
11
How to maintain a nuclear stockpile without testing?
Data can be complicated
12
The birth of HDF5
13
Long-term data
preservation
14
HDF keys to success
15
NeedsHDF
Group
Software for the long term.
API, library, utilities, documentation
User support
Institutional support
Long term storage
Complex formatting
Complex uses
Open source durability
Spinning off
•Business model options•For-Profit or not?•How to make money?• Intellectual property
16
Who is the HDF Group?
Small company ~ 40 employeesfocus on High
Performance Computing and Scientific Data
Offices in Champaign, IL + 7 other locations
Our flagship platform –HDF5 – is at the heart of our
open source ecosystem.
Tens of thousands use HDF5 every day.
1500+ projects on Github
Work with industry, research instituitons,
government to improve HDF5,
build HDF5 solutions, provide support.
18
Mission and commitments to OSS and community
HDF Group Website
19
Software
20
HDF Support portal
21
HDF Forum
22
Blogs
23
Webinars
24
Technology Advisory Board
25
Get engaged
HDFql in Nov/Dec 2019Present a Webinar about your project
About your project or some aspect of HDF5 Write a blog
lori.cooper@hdfgroup.org Contact
27
Future directions for the HDF Group and HDF R&D
1998-2007 - Maturation
• Matured the HDF5 data model and customized I/O including parallel I/O
• Community standards NeXUS, HDF-EOS, etc. emerged
28
2008-2019 - Expanding to non-HPC communities and Exascale computing
VPIC VORPAL
0
5000
10000
15000
Hopper Intrepid Hopper Intrepid
I/O
Ba
nd
wid
th (
MB
/s)
Default
Tuned
0
100
200
300
400
500
600
Num
ber o
f uniqu
e us
ers
Top 15 libraries used on Edison (2014)
• Widespread use• Improved I/O performance• Growth of software ecosystem• New features to support particle accelerators, exascalearchitectures, observational data, etc.
29
Future directions
30
Continue…
31
Support legacy library versions
Adapt to OS changes, etc.
Improve library, tools, docs
Increase community involvement
Leverage new memory, computing and storage architectures
• Deep memory hierarchy
• Alternate storage and I/O
• Non-traditional HDF5 uses
32
Facilitate convergence of Big Data and HPC
• Combine data and compute services
• Large scale query
Ease of use, productivity
• New APIs for C++ community
• Working with tool developers
THANK YOU!Questions & Comments?
Proprietary and Confidential. Copyright 2016, The HDF Group.
35
Recommended