View
130
Download
3
Tags:
Embed Size (px)
DESCRIPTION
The founder of The HDF Group discusses the past, present, and future of the HDF technology and community.
Citation preview
www.hdfgroup.org
The HDF Group
HDF Past, Present, Future
Mike Folk
The HDF Group
The 14th HDF and HDF-EOS Workshop
September 28-30, 2010
September 28-30, 2010 HDF/HDF-EOS Workshop XIV 1
www.hdfgroup.org
The HDF Group
September 28-30, 2010 HDF/HDF-EOS Workshop XIV 2
THE EARLYY
EARS
www.hdfgroup.orgHDF/HDF-EOS Workshop XIV 3September 28-30, 2010
www.hdfgroup.org
NCSA 1986
September 28-30, 2010 HDF/HDF-EOS Workshop XIV 4
www.hdfgroup.orgSeptember 28-30, 2010 HDF/HDF-EOS Workshop XIV 5
GFTF Oct 28, 1987
www.hdfgroup.orgSeptember 28-30, 2010 HDF/HDF-EOS Workshop XIV 6
GFTF Oct 28, 1987
www.hdfgroup.orgSeptember 28-30, 2010 HDF/HDF-EOS Workshop XIV 7
GFTF Nov 9, 1987
www.hdfgroup.org
GFTF Nov 9, 1987
September 28-30, 2010 HDF/HDF-EOS Workshop XIV 8
www.hdfgroup.orgSeptember 28-30, 2010 HDF/HDF-EOS Workshop XIV 9
www.hdfgroup.org
Library
• Version 1 – 1988• Primitive objects
• “raster,” “palette”, “aspect ratio”• “sets” or groupings to form higher level objects
• “raster image set”
• Version 2 – 1989• Scientific data sets• 24-bit raster
September 28-30, 2010 HDF/HDF-EOS Workshop XIV 10
www.hdfgroup.org
Library
• Version 3 – 1990• Subsetting• Vertex sets (Vsets)
• Groups (Vgroups), and tables (Vdatas)
• Version 4 – 1996• Full multiple file, multiple object access
September 28-30, 2010 HDF/HDF-EOS Workshop XIV 11
www.hdfgroup.org
Command line tools
September 28-30, 2010 HDF/HDF-EOS Workshop XIV 12
www.hdfgroup.org
NCSA software
September 28-30, 2010 HDF/HDF-EOS Workshop XIV 13
Big simulations with HDF Desktop visualization with “NCSA Image”
Desktop visualization with PolyView
www.hdfgroup.orgSeptember 28-30, 2010 HDF/HDF-EOS Workshop XIV 14
www.hdfgroup.orgSeptember 28-30, 2010 HDF/HDF-EOS Workshop XIV 15
www.hdfgroup.orgSeptember 28-30, 2010 HDF/HDF-EOS Workshop XIV 16
www.hdfgroup.orgSeptember 28-30, 2010 HDF/HDF-EOS Workshop XIV 17
www.hdfgroup.orgSeptember 28-30, 2010 HDF/HDF-EOS Workshop XIV 18
www.hdfgroup.orgSeptember 28-30, 2010 HDF/HDF-EOS Workshop XIV 19
www.hdfgroup.org
NetCDF Harmonization
September 28-30, 2010 HDF/HDF-EOS Workshop XIV 20
NSF $ = +
www.hdfgroup.org
“NSF SOFTWARE CAPITALIZATION”
September 28-30, 2010 HDF/HDF-EOS Workshop XIV 21
NSF $ = USER SUPPORTDOCUMENTATIONTESTING
www.hdfgroup.org
Other adopters of HDF
September 28-30, 2010 HDF/HDF-EOS Workshop XIV 22
www.hdfgroup.org
Other adopters of HDF
September 28-30, 2010 HDF/HDF-EOS Workshop XIV 23
www.hdfgroup.orgHDF/HDF-EOS Workshop XIV 24
Other adopters of HDF
September 28-30, 2010
Non-destructive testing
www.hdfgroup.orgHDF/HDF-EOS Workshop XIV 25September 28-30, 2010
Earth Observing System
www.hdfgroup.orgSeptember 28-30, 2010 HDF/HDF-EOS Workshop XIV 26
Earth Science (Earth Observing System)Aqua (6/01)
AuraTES HRDLSMLSOMI
Terra
CERES MISR
MODIS MOPITT
AquaCERES MODIS
AMSR
www.hdfgroup.orgHDF/HDF-EOS Workshop XIV 27
Message:HDF needs to support mission critical applications with high quality and performance standards
September 28-30, 2010
www.hdfgroup.org
A need for change?
Shortcomings of HDF (1996)• Limits on object & file size (<2GB)• Limited number of objects (<20K)• Rigid data models• I/O performance• Code complexity
September 28-30, 2010 HDF/HDF-EOS Workshop XIV 28
www.hdfgroup.orgSeptember 28-30, 2010 HDF/HDF-EOS Workshop XIV 29
NCSA Access, Spring 1996
www.hdfgroup.orgSeptember 28-30, 2010 HDF/HDF-EOS Workshop XIV 30
www.hdfgroup.orgSeptember 28-30, 2010 HDF/HDF-EOS Workshop XIV 31
www.hdfgroup.orgSeptember 28-30, 2010 HDF/HDF-EOS Workshop XIV 32
www.hdfgroup.orgSeptember 28-30, 2010 HDF/HDF-EOS Workshop XIV 33
ASCI = Accelerated Strategic Computing InitiativeWe Have A Nuclear Test Ban Treaty
How do we maintain a nuclear stockpile in the absence of testing?
www.hdfgroup.orgSeptember 28-30, 2010 HDF/HDF-EOS Workshop XIV 34
www.hdfgroup.orgHDF/HDF-EOS Workshop XIV 35
Big simulations
A simulation can have billions of elements
Each element can have dozens of associated values
September 28-30, 2010
www.hdfgroup.org
ASCI Data models and Formats Group
September 28-30, 2010 HDF/HDF-EOS Workshop XIV 36
www.hdfgroup.org
Describing data is challenging
September 28-30, 2010 HDF/HDF-EOS Workshop XIV 37
www.hdfgroup.orgSeptember 28-30, 2010 HDF/HDF-EOS Workshop XIV 38
www.hdfgroup.org39HDF/HDF-EOS Workshop XIV
HDF audience (1997)
• Applications facing big data challenges• Academia, government, industry• Hundreds of different applications• Users world-wide
September 28-30, 2010
www.hdfgroup.orgHDF/HDF-EOS Workshop XIV 40
Something is missing
September 28-30, 2010
www.hdfgroup.orgHDF/HDF-EOS Workshop XIV 41September 28-30, 2010
?
What is on these tapes?
www.hdfgroup.orgHDF/HDF-EOS Workshop XIV 42
Message:Communities are committing critically important data to HDF for the long term,
and expect it to last.
September 28-30, 2010
www.hdfgroup.org
The HDF Group
September 28-30, 2010 HDF/HDF-EOS Workshop XIV 43
THE MIDDLE YEARS
www.hdfgroup.orgSeptember 28-30, 2010 HDF/HDF-EOS Workshop XIV 44
www.hdfgroup.org
R&D 100
September 28-30, 2010 HDF/HDF-EOS Workshop XIV 45
www.hdfgroup.orgSeptember 28-30, 2010 HDF/HDF-EOS Workshop XIV 46
www.hdfgroup.org
CHANGING USER PROFILE
September 28-30, 2010 HDF/HDF-EOS Workshop XIV 47
www.hdfgroup.org
September 28-30, 2010 HDF/HDF-EOS Workshop XIV 48
Flight testing
www.hdfgroup.orgHDF/HDF-EOS Workshop XIV 49
Vehicle testing
September 28-30, 2010
www.hdfgroup.orgHDF/HDF-EOS Workshop XIV 50
Bioinformatics
September 28-30, 2010
www.hdfgroup.orgHDF/HDF-EOS Workshop XIV 51
Images
September 28-30, 2010
25-80Å resolution electron tomography8k x 8k x 1k images soon (256 GB)
www.hdfgroup.orgHDF/HDF-EOS Workshop XIV 52
Making movies
September 28-30, 2010
Spiderman 3 The Polar Express
www.hdfgroup.org
New expectations
• “Deliverables”• “What about my data in 2080?”• How soon can you do this for me?• Business opportunities and the academy
September 28-30, 2010 HDF/HDF-EOS Workshop XIV 53
www.hdfgroup.org
Doing business in academia
September 28-30, 2010 HDF/HDF-EOS Workshop XIV 54
We deliver papers, not software.
Data preservation?Talk to the librarian.
I can do that for you maybe by next year.
No profits allowed here!
www.hdfgroup.orgSeptember 28-30, 2010 HDF/HDF-EOS Workshop XIV 55
www.hdfgroup.orgSeptember 28-30, 2010 HDF/HDF-EOS Workshop XIV 56
www.hdfgroup.org
Missions and Goals
• The five missions• Public• Software• Service• Human• Financial/Structural
September 28-30, 2010 HDF/HDF-EOS Workshop XIV 57
www.hdfgroup.org
HDF Group Mission
To ensure long-term accessibility of HDF data through sustainable development and support of HDF technologies.
September 28-30, 2010 HDF/HDF-EOS Workshop XIV 58
www.hdfgroup.orgHDF/HDF-EOS Workshop XIV 59
Spinning off
September 28-30, 2010
• Business options and questions• For-Profit or not?• How to make money?
• Intellectual property
www.hdfgroup.org
Help from an unexpected place
September 28-30, 2010 HDF/HDF-EOS Workshop XIV 60
www.hdfgroup.org
The HDF Group
HDF/HDF-EOS Workshop XIV 61
Running our Business:Challenges and Surprises
September 28-30, 2010
www.hdfgroup.orgHDF/HDF-EOS Workshop XIV 62
Challenges
• Managing contracts• Finance
• Keeping overhead under control• Defining “profit”• Accounting and budgeting expertise
• Business model
September 28-30, 2010
www.hdfgroup.org
Today vs. 2006
September 28-30, 2010 HDF/HDF-EOS Workshop XIV 63
2006 2010
Staff 17 34
Funded projects
6 13
Budget $1.6M $3.4M
www.hdfgroup.org
The HDF Group
September 28-30, 2010 HDF/HDF-EOS Workshop XIV 64
THE FUTURE
www.hdfgroup.org
The near future
September 28-30, 2010 HDF/HDF-EOS Workshop XIV 65
www.hdfgroup.orgHDF/HDF-EOS Workshop XIV 66
Cross the chasm to new users and applications
September 28-30, 2010
Time
www.hdfgroup.orgHDF/HDF-EOS Workshop XIV 67
The near future
• From technology driven to market driven• Raising the open source quality bar• Expanding our ability to help customers• Expanding the kinds of things we're good at
September 28-30, 2010
www.hdfgroup.org
The future
September 28-30, 2010 HDF/HDF-EOS Workshop XIV 68
www.hdfgroup.org
TECHNOLOGY STRATEGIES
September 28-30, 2010 HDF/HDF-EOS Workshop XIV 69
www.hdfgroup.org
Lots of data
September 28-30, 2010 HDF/HDF-EOS Workshop XIV 70
www.hdfgroup.org
Big data
September 28-30, 2010 HDF/HDF-EOS Workshop XIV 71
Image courtesy of the National Center for Computational Sciences, Oak Ridge National Laboratory”.
Cores Petaflops
224,162 1.759
www.hdfgroup.org
Kinds of data
September 28-30, 2010 HDF/HDF-EOS Workshop XIV 72
www.hdfgroup.org
GFTF Nov 9, 1987
September 28-30, 2010 HDF/HDF-EOS Workshop XIV 73
www.hdfgroup.orgHDF/HDF-EOS Workshop XIV 74
A focus on Preservation
September 28-30, 2010
Darwin’s first evolutionary tree - 1837
www.hdfgroup.orgHDF/HDF-EOS Workshop XIV 75
Preservation-based
evolution
September 28-30, 2010
Darwin’s first evolutionary tree - 1837
www.hdfgroup.orgHDF/HDF-EOS Workshop XIV 76September 28-30, 2010
Promoting standardization
and interoperability
www.hdfgroup.orgHDF/HDF-EOS Workshop XIV 77
Institutional strategies
September 28-30, 2010
www.hdfgroup.orgHDF/HDF-EOS Workshop XIV 78
A mission-driven
business
September 28-30, 2010
www.hdfgroup.orgHDF/HDF-EOS Workshop XIV 79
One keeper of the format
and software
September 28-30, 2010
www.hdfgroup.orgHDF/HDF-EOS Workshop XIV 80
Open source
September 28-30, 2010
www.hdfgroup.orgHDF/HDF-EOS Workshop XIV 81
Long-term institutional support
September 28-30, 2010
The National Archives, Washington, D.C.
www.hdfgroup.orgHDF/HDF-EOS Workshop XIV 82
Human, financial, legal foundations for sustainability
September 28-30, 2010
www.hdfgroup.orgHDF/HDF-EOS Workshop XIV 83
Business model
September 28-30, 2010
www.hdfgroup.orgHDF/HDF-EOS Workshop XIV 84September 28-30, 2010
A virtual organization
www.hdfgroup.orgHDF/HDF-EOS Workshop XIV 85
Thank You
September 28-30, 2010
www.hdfgroup.orgHDF/HDF-EOS Workshop XIV 86September 28-30, 2010
www.hdfgroup.orgSeptember 28-30, 2010 HDF/HDF-EOS Workshop XIV 87
www.hdfgroup.orgHDF/HDF-EOS Workshop XIV 88September 28-30, 2010
www.hdfgroup.orgSeptember 28-30, 2010 HDF/HDF-EOS Workshop XIV 89