- 1 - HDFHDF

Mike Folk, Elena Pourmal , Bob McGrath

National Center for Supercomputing Applications

University of Illinois at Urbana-Champaign

NOBUGS 2004HDF-EOS Workshop VIII

HDF Software ProcessHDF Software ProcessLessons Learned & Success FactorsLessons Learned & Success Factors

HDFHDF

- 2 - HDFHDF

OutlineOutline

• What is HDF? and Who is HDF?

• HDF “Architecture”

• Some statistics

• How do we measure success?

• How can we achieve success?

• Group practices

• Summing up – strengths, weaknesses, needs

- 3 - HDFHDF

What is HDF?What is HDF?Who is HDF?Who is HDF?

- 4 - HDFHDF

HDF in a nutshell – what it isHDF in a nutshell – what it is

• File format and I/O Libraries for storing, managing and archiving large complex scientific and other data

• Tools and utilities• Open source, free for any use (U of I license)

• Well maintained and supported

• From HDF group, NCSA Univ of Illinois• http://hdf.ncsa.uiuc.edu

- 5 - HDFHDF

HDF in a nutshell - featuresHDF in a nutshell - features

• General– simple and flexible data model

• Flexible– store data of diverse origins, sizes, types– supports complex data structures and types

• Portable– available for many operating systems and machines

• Scalable– works in high end computing environments– accommodates date of any size or multiplicity

• Efficient – fast access, including parallel i/o– Stores big data efficiently

- 6 - HDFHDF

HDF in a nutshell - usersHDF in a nutshell - users

• Apps in industry, academia, government– More than 200 distinct applications

• Large user base– E.g. NASA estimates 1.6 million users

• Underlying format for community standards– E.g. HDF-EOS, SAF, CGNS, NPOESS, NeXus

- 7 - HDFHDF

Example of HDF file: mixing and grouping Example of HDF file: mixing and grouping objectsobjects

Raster imageRaster image

palettepalette

3-D array 3-D array

2-D array2-D arrayRaster imageRaster image

lat | lon | temp----|-----|----- 12 | 23 | 3.1 15 | 24 | 4.2 17 | 21 | 3.6

TableTable

a

b c

z

x_foo_y

foo

1GB

Text : This file was create as a part of… see http://hdf.ncsa.uiuc.edu

- 8 - HDFHDF

HDF “Architecture”HDF “Architecture”

- 9 - HDFHDF

HDF “Architecture”HDF “Architecture”

File or other data sourceFile or other data source

HDF I/O libraryHDF I/O library– High-level, object-specific APIs.High-level, object-specific APIs.

– Low-level API for I/O to files, etc.Low-level API for I/O to files, etc.

Utilities and applications for Utilities and applications for managing, manipulating, managing, manipulating, viewing, & analyzing data.viewing, & analyzing data.

Low level Interface

HDF5 Applications Programming Interface

Tools & ApplicationsTools & Applications

FileFile

- 10 - HDFHDF

User’s controlled I/O and “storage”User’s controlled I/O and “storage”

• Data pipeline– Data transformation– Compression– Encryption– Storage layout

• Virtual file options– Stdio (normal file)– Split file– MPI-IO & other parallel– Network– Memory– custom

HDF I/O Library

HDF “File”HDF “File”

- 11 - HDFHDF

Supported languages and compilersSupported languages and compilers

• C• Wrappers:

– C++– Fortran90– Java

• Vendors’ compilers (SUN, IBM, HP, etc.)• PGI and Absoft (Fortran)• GNU C (e.g. gcc 3.3.2)

- 12 - HDFHDF

Supported Machines and OSSupported Machines and OS

• Solaris 2.7, 2.8 (32/64-bit)

• IRIX6.5 IRIX64-6.5

• HPUX 11.00

• AIX 5.1 (32/64-bit modes)

• OSF1

• FreeBSD

• Linux (SuSe, RH8, RH9) including 64-bit

• Altix (SGI Linux)

• IA-32 and IA-64

• Windows 2000, XP

• MAC OS X

• Crays (T3E, SV1, T90IEEE)

• DOE National Labs machines

• Linux Clusters

- 13 - HDFHDF

FileFile

Serial Parallel

Linux RH IRIX32 XP SV1

IA32 SGI Wintel Cray

Architecture in contextArchitecture in context

Low level Interface

HDF5 Applications Programming Interface

Tools & ApplicationsTools & ApplicationsC C++ F90 Java

- 15 - HDFHDFFileFile

Serial Parallel

Linux RH IRIX32 XP SV1

IA32 SGI Wintel Cray

Architecture in contextArchitecture in context

Low level Interface

HDF5 Applications Programming Interface

C C++ F90 Java

HDF-EOSHDF-EOS SAFSAF CGNSCGNS

Tools & ApplicationsTools & Applications

- 16 - HDFHDF

The testing challengeThe testing challenge

Machines × operating systems× compilers × languages

× serial and parallel× compression options× configuration options

× virtual file options× backward compatibility

= a large number

- 17 - HDFHDF

““Diversity makes our code better…”Diversity makes our code better…”

Todd Smith, Geospiza

- 18 - HDFHDF

Some statisticsSome statistics

- 19 - HDFHDF

HDF StatisticsHDF Statistics

• HDF Group– 15 FTE + 3-5 students– $2.1million annual budget

• HDF5 source code distribution – 2073 files– 917,186 Lines of code

• HDF Project – HDF5, HDF4, H4toH5, H5Lite, Java– 3,000,000 lines of code (estimate)

- 20 - HDFHDF

Docs33%

Examples1%

Tools4%

Libraries30%

Tools tests 4%

LibraryTests13%

Configure15%

HDF5 source distribution by categories HDF5 source distribution by categories (lines of code)(lines of code)

- 21 - HDFHDF

HDF5 staff investmentHDF5 staff investment

Test writing7%

Comm. with users 2%

Peer-to-peer comm. 12%

Code dev.33%

Porting/release testing 9%

Docs, design, consult 14%

Meetings, etc.9%

User's support14%

- 22 - HDFHDF

How do we measure success?How do we measure success?

- 23 - HDFHDF

How do we measure success?How do we measure success?

• Mission• Goals and objectives• Strong and continuing relationships with users• High quality software• Strong committed development team • Great working environment• Adequate funding

- 24 - HDFHDF

Mission, goals and objectivesMission, goals and objectives

• Mission– To develop, promote, deploy, and support open and

free technologies that facilitate scientific data exchange, access, analysis, archiving and discovery

• Goals (examples)– Innovate and evolve the technologies in concert with a

changing world of technologies – Maintain a high level of quality and reliability– Collaborate and build communities– Build a team

- 25 - HDFHDF

Mission, goals and objectivesMission, goals and objectives

• Objectives - how we reach the goal• Example:

– Goal• Maintain a high level of quality and reliability

– Objectives• Improve testing• Implement a program to insure excellent software

engineering practices• Develop and execute a plan to meet

quality/reliability standards

- 26 - HDFHDF

UsersUsers

• Number of users

• Happy users • Unhappy users • Users achieve their goals by using HDF

technologies

• Users coming back with new needs

• Financial support from users

- 27 - HDFHDF

SoftwareSoftware

• Technology that addresses users’ needs and demands (current and future)– E.g. big files, parallel access, multiple objects

• Usability– Number and types of applications– Appropriate APIs and data models– Available tools– Interoperability with other software

• E.g. IDL, MatLab, Mathematica

- 28 - HDFHDF

SoftwareSoftware

• Stability– Can data be shared?– Can software run on needed platforms

• Sustainability– Can read data written 15 years ago on obsolete platform– Is software available in 15 years?

• Acceptability– De facto standard

• Open standard for exchange of remote-sensed data• Over 3,000,000,000,000,000 bytes stored in HDF and HDF-EOS

- 29 - HDFHDF

How can we How can we achieveachieve success? success?

- 30 - HDFHDF

How can we achieve success? How can we achieve success?

• Maintain strong, responsible, and continuing relationships with users

• An approach to needs identification, software design, and software implementation based on sound principles of software engineering

• Effective technical processes for developing, testing, integrating and maintaining software

• Business and social processes based on sound group management principles

- 31 - HDFHDF

Stages of software development at Stages of software development at HDFHDF

• Getting started

• Creating an implementation approach

• Implementation and maintenance

• Relations with users and sponsors

• Group practices

- 32 - HDFHDF

Getting startedGetting started

• Discover a need• Identify a sponsor• Clarify the need, its role, and its importance• Enter task into the project plan

– Make initial estimate of time and resources for the task– Give it a priority– Identify task’s lead – Identify a person who will work on the task

- 33 - HDFHDF

Creating implementation approachCreating implementation approach

• Write up a needs/approach RFC (Request For Comment)– Actively solicit feedback from developers/sponsors– Revise until satisfied

• Write up a design/approach RFC– Get feedback from developers/sponsors– Revise until satisfied

• Revise project plan according to RFC results• Archive RFC

- 34 - HDFHDF

Implementation and maintenanceImplementation and maintenance

• Identify validation plan (need improvement)• Implement

– Library or tool– Tests– Documentation

• Ask sponsor and friendly users for feedback• Review results and repeat appropriate steps above as

needed• Clean up (documentation, Web, etc.) and announce • Support (debug, fix, add more tests, advertise)

- 35 - HDFHDF

Relations with users and sponsorsRelations with users and sponsors

• Who are our sponsors?– Organizations and communities with

institutional and financial commitment to HDF• NCSA, NASA, DOE ASCI, Boeing, …

– Agencies supporting R&D• NCSA, NASA, DOE, NSF, …

– Collaborators who make in-kind contributions• Cactus, PyTables, NeXUS, CGNS …

– HDF group members

- 36 - HDFHDF

Relations with users and sponsorsRelations with users and sponsors

• Each task is associated with a sponsor• Each task has a priority, which should be

confirmed with sponsor• Each task falls into one of these categories

– Research– R&D (research, possibly integrate into product)– Development

• Technology infusion• Library or tools enhancement

- 37 - HDFHDF

Group practicesGroup practices

- 38 - HDFHDF

Group practices - technicalGroup practices - technical

• Source code management: CVS

• Bug tracking: Bugzilla– Bugs entered by support staff and developers– Prioritized by staff– Easy bugs fixed “on the fly”

- 39 - HDFHDF

Group practices - technicalGroup practices - technical

• The testing challenge

• Code testing– Testing before code check-in– Regression testing– Remote testing– Different configurations testing– Backward compatibility testing

- 41 - HDFHDF

Daily test reportDaily test report

From: HDF group system admin <hdfadmin@ncsa.uiuc.edu>To: hdf5lib@ncsa.uiuc.eduSubject: HDF5_Daily_Tests_FAILED!!!

*** HDF5 Tests on 041022 ***

=============================   Watchers List=============================HDF5 Daily test features/platforms watchers and procedure---------------------------------------------------------

Procedure:The watcher will investigate and report the cause of failure by 11am.The developer who checked in the error code may report so by then too.The watcher or the developer should get thefailure fixed and report it by 3pm.

Platforms watchers:AIX 5.1 (copper)         AlbertFreeBSD                  QuinceyHP-UX                    ElenaIA32    (tungsten)       RaymondIA64    (tg-login)       AlbertIRIX64-6.5 32,64-bit     RaymondIRIX 6.5                 RaymondLinux 2.4                PeterSolaris 2.7&8 32,64-bit  ElenaWindows                  Kent

Features watchers:General Library          QuinceyGeneral parallel         Albertconfiguration            Quincey, Jamesmpich                    RaymondFortran                  ElenaIntel compilers          Elena + Kent (for windows) PGI compilers            ElenaC++                      Binh-Minh             Thread-safety            QuinceyTools                    Padro

---updated: 2004/10/01

=============================   Tests Summary=============================****FAILED eirene: setenv CC icc setenv F9X ifc setenv CXX icc --enable-fortran --enable-cxx****

PASSED arabica: setenv CC /afs/ncsa/projects/hdf/packages/mpich_1.2.4/SunOS64_5.7/bin/mpicc setenv F9X /afs/ncsa/projects/hdf/packages/mpich_1.2.4/SunOS64_5.7/bin/mpif90 setenv ALL_LOCAL 1 --enable-fortran standardPASSED arabica: setenv CC mpicc setenv ALL_LOCAL 1 standardPASSED arabica: setenvN 2 CC cc -xarch=v9 setenvN 2 F9X f90 -xarch=v9 setenvN 2 CXX CC -xarch=v9 standard --with-szlib=/afs/ncsa/projects/hdf/packages/szip_new/SunOS_5.7-64bitPASSED arabica: standard --enable-cxx --enable-fortran --with-szlib=/afs/ncsa/projects/hdf/packages/szip_new/SunOS_5.7PASSED Cu12: --enable-parallelPASSED Cu12: --enable-parallel setenv CFLAGS -q64 setenv FFLAGS -q64 setenvN 3 AR ar -X 64 --enable-fortran --with-zlib=/afs/ncsa/projects/hdf/packages/zlib/AIX5.1-64bit --with-szlib=/afs/ncsa/projects/hdf/packages/szip_new/AIX5.1-64bit

- 42 - HDFHDF

Group practices - technicalGroup practices - technical

• Release levels– Development release– Official release– Past releases

- 43 - HDFHDF

Group practices - technicalGroup practices - technical

• Coding standards

• Maintaining platform-independence

• Maintaining time-independence

• Rules for changing APIs

• Documentation

• Rapid prototyping

- 44 - HDFHDF

Group practices – business and socialGroup practices – business and social

• Staff breakdown– User support– Documentation– QA– Software development– Testing– Team leadership– System administration

Basic library Basic library developmentdevelopmentBasic library Basic library developmentdevelopment

Support, Support, doc, QA, doc, QA,

maintenancemaintenance

Support, Support, doc, QA, doc, QA,

maintenancemaintenance

Tools and Tools and JavaJava

Tools and Tools and JavaJava

Parallel I/O, Parallel I/O, Grid, Grid,

big machinesbig machines

Parallel I/O, Parallel I/O, Grid, Grid,

big machinesbig machines

HDF ProjectHDF ProjectHDF ProjectHDF Project

• Team lead for each team• Most staff in two or more teams• Staff relationships

– Complement each other– Overlap each other– Keep each other honest

- 45 - HDFHDF

Group practices – business and socialGroup practices – business and social

• Accountability of everyone to the whole process• Help desk• Approaches to carrying out tasks

– Paying attention to technical proposals– Weekly HDf5 developer’s meetings– HDF seminars

• Management and administration– Performance reviews with emphasis on goals, development– Critical to success– That’s another talk

- 46 - HDFHDF

Summing upSumming upStrengths, weaknesses, needsStrengths, weaknesses, needs

- 47 - HDFHDF

StrengthsStrengths

• User support• Staff

– High quality, diverse staff with good morale– Staff commitment and enthusiasm

• Ability to address all aspects of product development– Emphasis on quality control – Fast bug fixing and frequent releases– Ability to focus on a single product over a long term

• High level of support from sponsors• Project’s visibility through NCSA, NASA, DOE, users

- 48 - HDFHDF

WeaknessesWeaknesses

• Software development team– Library expertise still concentrated among too few

developers– Team communication is challenging

• Processes– Release/maintenance take too much time and

resources– Configuration and porting are a huge time sink– We don’t do enough prototyping– Hard to keep up with new technologies– Parallel I/O hard to support

- 49 - HDFHDF

More weaknesses & challengesMore weaknesses & challenges

• Usability– Software too hard to use for casual users– Insufficient documentation– Insufficient tools for high level users– Insufficient interoperability with common tools and

formats

• Marketing– Marketing effort is inadequate– Need to connect better with users and potential users

• Viable long-term support

- 50 - HDFHDF

Most immediate needsMost immediate needs

• Configuration and build• Testing and prototyping• Marketing• Reporting

– Performance reports– General reports to users– HDF book

• Sustainable business model

- 51 - HDFHDF

Thank youThank you