05 December, 2002HDF & HDF-EOS Workhop VI1 ESDIS Status Richard Ullman ESDIS Project [email protected]

05 December, 2002 HDF & HDF-EOS Workhop VI 1

ESDIS Status

Richard Ullman

ESDIS Project

[email protected]


Outline

• ESDIS General Status

• HDF-EOS Plans

• Website http://hdfeos.gsfc.nasa.gov


ESDIS Science Data Services Today

• EOSDIS provides support for high data volumes from Aqua, Terra, and Landsat 7, and continues to support QuikSCAT, ACRIMSat, SAGE-III, JASON and pre-EOS-era data including TRMM, UARS, TOPEX/Poseidon, RADARSat, and others. – EOSDIS finalizing preparations to support ICESat and SORCE

– In all, EOSDIS is providing data processing, archival, and/or distribution for over 15 Earth science satellite missions.

• EOSDIS has set a new benchmark for data management. The total volume of the science data in our archives totals over 2 Petabytes. Since 1998, the science data volume managed by the EOSDIS has increased eight-fold, and continues to grow at a rate of over 2 Terabytes per day

• In FY02, EOSDIS provided more than 16 million data and information products to over 1.8 million individuals.


FY94/95 FY96 FY97 FY98 FY99 FY00 FY01 FY02

Science Operations Supported

Heritage Missions

T/P, UARS, TOMS, ERBE

Terra ACRIMSatTRMM

Landsat 7 QuikSCAT

Aqua GRACE

SAGE III JASON

SeaWiFS Radarsat

Products Distributed

15 Million

Archive Growth

Archive Volume has doubled each year for the past three years

2 Petabytes

Science Operations Timeline

http://seawifs.gsfc.nasa.gov/SEAWIFS/SEASTAR/seawifs.mpeg


EOSDIS DAAC Data delivery FY’02October 1, 2002 through September 30, 2002

• Over 1.8 million distinct users

• 16,013,980 products delivered

• Notes to charts on following slides:– Distinct users includes users accessing DAAC web pages,

including web-crawlers

– Distinct user type is based on email addresses of users or URLs

– Product delivered is defined as the smallest deliverable unit of data

– Product delivery breakdown is based on email addresses of users receiving ECS and Non-ECS data

– “FTP Delivery” are to URLs not mapped to specific domains

– “Foreign Other” includes foreign email addresses whose country is known but domain-type could not be determined


16,013,980 Data Products DeliveredOctober 1, 2001 - September 30, 2002


Over 1.8 Million Distinct UsersOctober 1, 2001 - September 30, 2002


ESDIS Status

• After years of development, ECS is operational and generally recognized as successful.

• Primary ECS development contract is essentially completed.

• ECS Maintenance and Development (EMD) will emphasize maintenance more than development.

• Already the majority of ESDIS budget has shifted from development to operations.

• Still two major areas of new capability (see posters at AGU for more information):– Data Pools

– ECHO


HDF-EOS Plans

• HDF-EOS 5 development is nearly complete.

• Continue to maintain, port to newer operating systems, bug-fix.

• Need the advise of this community workshop - we will discuss this afternoon.– What tools or capabilities are now needed?

• HDF-EOS 2 and HDF-EOS 5

– When is it the right time to press EOS science teams to migrate to HDF-EOS 5?

– What steps should NASA take to facilitate?


Data Pools Concept: The Right Data, the Right Way, Right Now!

Value-Added Providers

Data Providers

Data TailoringWorkflow

Management

End Users

User-definedViews, Presentations and Data Access Requests

Dynamic Web and FTP Data Views,User-specified Data Access

Other Data Pools

Other Data Pools

Data Producers

DataServices

DataItems

Data ServiceDevelopers

GeoPIPE

PIPE = Personalized Information Presentation Engine


General Capabilities– The Right Data: Data Location

• Groups, themes, bookmarks and views

• Navigation and machine-based location

• Science views (e.g., science metadata)

• Applications views (e.g., OGIS coverage server)

• Location aids (e.g., geopolitical overlays)

• External location-support services

– The Right Way: Data Tailoring• Data reduction, manipulation and reformatting services

• Virtual data products

• Workflow management and execution monitoring

• External tailoring services

– Right Now: Rapid Access• Low latency data transfers

• Secure remote file access

• On-the-fly data compression

• Automated request routing and load balancing

• Near real-time data


EOS ClearingHOuse (ECHO) http://eos.nasa.gov/echo

• ECHO is a metadata clearinghouse– A single Internet portal for Earth science metadata search

– Index of data provider inventory-level data holdings metadata.

• .ECHO is a data order broker– Forwards orders for data discovered to the data providers to fill.

– Data providers retain customer fulfillment service

• ECHO is a data service broker– Registered service are associated with registered datasets

– Four kinds of service association • Advertised, Context Passing, Brokered, Order Option

• ECHO is an open client API for custom user clients – The EOS Data Gateway (EDG) is ESDIS’ ECHO client


ECHO Data Providers

• Current ECHO Data Providers– EOSDIS Core System DAACs

(17% thus far)• EDC Land Processes DAAC

• Goddard DAAC

• NSIDC DAAC

– ORNL DAAC (100%)

• ECHO holds the metadata for over 3 million granules, and growing

ECHO Metadata Holdings

0

500,000

1,000,000

1,500,000

2,000,000

2,500,000

3,000,000

3,500,000

4,000,000

Jan-06 Feb-06

Mar-06 Apr-06 May-06

Jun-06 Jul-06 Aug-06

Sep-06

Date

ORNL_DAAC

NSIDC_ECS

LP_ECS

GSFC_ECS


HDF-EOS Tools and Information Web Sitehttp://hdfeos.gsfc.nasa.gov

[email protected]


http://hdfeos.gsfc.nasa.gov


Website status

• A resource for discovering about hdf-eos in particular.– This workshop series’ presentation archive

– Links to hdf-eos tools

• Site has been revamped according to comments received at the last workshop.– Tools download page now has opportunity for user feedback.

– Workshop presentations are keyword searchable

• New features planned– Better introductory material.

• Post and organize documentation of HDF-EOS.

• Better navigation to NCSA site for HDF

– Host hdf-eos “web forum” • Incorporate the [email protected] listserv


HDF-EOS Profile

Richard Ullman

ESDIS Project

[email protected]


History

• September 1993, HDF adopted as baseline standard for EOSDIS Core System standard data product generation, archival, ingest, and distribution capabilities

• Dec. 94 - ECS Engineering Support Directive to create HDF-EOS

• June 1996, HDF-EOS v1.0 library released

• Upgrades every 6 mo., – Current version 2.8 on HDF 4

• HDF5 support (called HDF-EOS 5) beginning November 2000– Current version 5.1.3 on HDF5


HDF-EOS Data Objects

• Point

• Swath

• Profile (Swath subtype)

• Grid

• Zonal (HDF-EOS 5 only)


HDF-EOS 5

• Based on HDF5, a complete rewrite of HDF4 with a different interface. – First released in 2000.

• Designed to ‘resemble’ HDF-EOS 2 to the maximum extent possible. – Support same data structures

– Added prefix ‘HE5_’ to HDF-EOS 2 functions.

– Doesn’t preclude HDF5 functionality.

– Data Type changes, e.g. INT64 -> H5T_NATIVE_LONG


HDF-EOS 5 Functionality

• Basic File I/O

• Fill Values

• Compression

• Chunking/Tiling

• Swath Interface

• Grid Interface

• Point Interface

• Profile Interface

• Global (File), Group & Local Attributes

• External Data Files

• Subsetting

• Unix/Linux Support

• Threadsafe Version

• FORTRAN, C, C++

• General Table Interface (proposed)


The new ADDITIONAL Group has global (file) attributesThe new functionality is added to the EH(utility) interface.

STRUC. METADATA

Global (file) Attributes

HDFEOS

SWATH GRID POINT

Root -- “/”

ADDITIONAL

HDFEOSINFORMATION

Top Level of HDF-EOS 5


SwathName

Data Fields Geolocation Fields

Attribute Dataset

DataField.1

DataField.n

Longitude Latitude

Group Attribute<DataFields>:<AttrName>

Global Attribute<SwathName>:<AttrName>

Local Attribute<FieldName>:<AttrName>

CoLatitudeTime

Shaded Objects are implemented in a fixed way. User doesn’t have

direct access via the interface

Each Data Field can have Attributes and/or Dimension Scales

Profile Fields

ProfileField.1

ProfileField.n

Group

Swath Structure


Latitude Longitude Tempera-ture oC

Dew PointoC

61.12 -149.48 15.00 5.00

45.31 -122.41 17.00 5.00

38.50 -77.00 24.00 7.00

38.39 -90.15 27.00 11.00

M M M M

HDF-EOS Point

• Intended use– Discrete points in time and/or

location.

– Table of data linked to table of geographic information.

– 8-level Hierarchical, each level may contain indices to the level below


Latitude Longitude Buoy_ID25.2645 091.2564 012622.3549 -93.4657 356423.2564 -89.2546 1256

Buoy_ID

Time WaveHeight(ft)

Temp(C)

0126 01:26 2.54 18.40126 05:56 3.58 18.23564 06:28 12.64 16.41256 08:12 7.58 17.11256 09:58 7.76 17.20126 09:59 4.23 20.13564 10:16 10.23 17.5

HDF-EOS Point

• Hierarchical links:– Every level in a Point data set

must be linked into the hierarchy.

– Before two levels can be linked, a link field must exist


HDF-EOS Swath

• Intended use– Across track scanning

instruments.

– Sounding instruments

– Level 1: Geolocated Sensor Units

– Level 2: Geophysical Parameters

InstrumentPath

Scan Lines

Along Track

InstrumentPath

Along Track


“Longitude”

“Latitude”DimensionName: GeotrackSize: 21

DimensionName: GeotrackSize: 21

DimensionName: ScanSize: 16

DimensionName: ScanSize: 16

DimensionName: TrackSize: 42

DimensionName: TrackSize: 42

Map1DataDimension: “Track”

Geodimension: “Geotrack”Offset: 1

Increment: 2

Map1DataDimension: “Track”

Geodimension: “Geotrack”Offset: 1

Increment: 2

“Brightness Temperature” “Time”

HDF-EOS Swath

Data Fields Geolocation Fields


HDF-EOS Grid

• Intended use– Variables mapped on uniform

space-time grid scales

– Level 3 - Gridded single measurement parameters

– Level 4 - Modeled or derived from multiple measurements


HDF-EOS Grid

XdimSize: 2000

YdimSize: 800

Projinfo

AltitudeSize 30


HDF-EOS Grid

• Projections Supported

• Geographic

• Transverse Mercator

• Universal Transverse Mercator

• Hotine Oblique Mercator

• Space Oblique Mercator

• Polar Stereographic

• Lambert Azimuthal Equal Area

• Lambert Conformal Conic

• Polyconic

• Interrupted Goode’s Homolosine

• Integerized Sinusoidal

• Compression Methods

• Run-Length Encoding

• Adaptive Huffman

• Gzip


Product Levels

• Level 0 - Reconstructed, unprocessed instrument/pay-load data at full resolution; any and all communications artifacts, e.g., synch. frames, communications headers, duplicate data removed.

• Level 1A- Reconstructed, unprocessed instrument data at full resolution, time-referenced, and annotated with ancillary information, including radiometric and geometric calibration coefficients and georeferencing parameters, e.g., platform ephemeris, computed and appended but not applied to the Level 0 data.

• Level 1B - Level 1A data that have been processed to sensor units (not all instruments will have a Level 1B equivalent).

• Level 2 - Derived geophysical variables at the same resolution and location as the Level 1 source data.

• Level 3 - Variables mapped on uniform space-time grid scales, usually with some completeness and consistency.

• Level 4 - Model output or results from analyses of lower level data, e.g., variables derived from multiple measurements.


Resources

– HDF-EOS on the web:• http://hdfeos.gsfc.nasa.gov/

• http://newsroom.gsfc.nasa.gov/sdptoolkit/toolkit.html

– HDF and HDF5 on the web:• http://hdf.ncsa.uiuc.edu/

– HDF-EOS and HDF via email:• [email protected]

• [email protected]

Documents

05 December, 2002HDF & HDF-EOS Workhop VI1 ESDIS Status Richard Ullman ESDIS Project [email protected]