UML Representation of NPOESS Data Products in HDF5

UML Representation of NPOESS Data Products in

HDF5 Denise Dulaigh

Chad Johnson

William Johnsen, Ph.D.

NPOESS Program

Raytheon Company

Aurora, Colorado

HDF5 Workshop VII, 26 – 28 Oct 2004 2

MTG03-IDP-049

2

The National Polar-orbiting Operational Environmental Satellite System (NPOESS)

– Collects and distributes global satellite sensor data and creates NPOESS Data Products.

– NPOESS Data Products are packaged into HDF5 files and delivered.

– Delivered data can be read an any platform supported by HDF5.

– There are a number of different types of data, discussed in further slides.

– To meet Operational Latency requirements, the data is originally produced in granules (data blocks) that allow the system to operate efficiently.

– A change in the size of the granules critically affects production efficiency.

– Native metadata is associated with granules and aggregated granules.

– Geolocation data in separate HDF5 files are referenced from NPOESS Data Product HDF5 files.


MTG03-IDP-049

3

NPOESS data – user view

How does our granule size affect users?

– The size of the granules may not be the most desirable means to receive, analyze, store, or use the contained data.

– The characteristics (metadata) of individual granules need to be retained for analysis and archival purposes.

To increase usability:

– NPOESS provides a means to package (aggregate) multiple granules into a single deliverable HDF5 file.

– Metadata names are FGDC compliant.

– For consistent access to data in the produced HDF5 files, access to data uses hyperslab references.


MTG03-IDP-049

4

NPOESS data types

– Raw Data Records (RDRs) – Full resolution, unprocessed time-referenced digital sensor data and other data required to compute SDRs.

– Temperature Data Records (TDRs) – Geolocated, antenna temperatures with all relevant calibration data counts and ephemeris data, from passive microwave sensors only.

– Sensor Data Records (SDRs) – Computed from RDRs; full resolution sensor data that are time referenced, earth located, and calibrated.

– Environment Data Records (EDRs) – Data records that contain the environmental parameters or imagery required to be generated as user products, produced by applying an appropriate set of algorithms to SDRs.

– Deliverable Intermediate Products (IPs) – Data records that are produced by applying an appropriate set of algorithms to SDRs, but is not on the official list of deliverable EDRs. Currently, Cloud Mask is the only deliverable IP.

– Selected data, depending on the user: auxiliary data, ancillary data.

– The UML diagram, data structure, and data types of all NPOESS HDF5 data is documented in the Common Data Format Control Book – External (CDFCB-X)

** Formal definitions (not operational) from the NPOESS Glossary


MTG03-IDP-049

5

Dim_1=5

Dim_2=4

A multidimensional array of data elements

Header with metadata• Dataspace (intrinsic)• Datatype (intrinsic)• Storage layout• Attributes

Schematic of an HDF5 Dataset

int16

Datatype

Data

Dataspace

2 Dim_3=2

Rank

Dimensions

Chunked; compressed

Storage layout

start time = 32.4

data type = ‘SDR’

algorithm = ‘1.1’

Attributes

Dataset

ArrayDatatype

Header


MTG03-IDP-049

6

File RootRoot

Attributes

Secondary Data Data Product

Data Product Attributes

AggregatedGranules

Secondary Data

Attributes

Granule 1 Granule NData

ArrayDatatype


ArrayDatatype

ArrayDatatype

Data Ref 1

Data Ref 2

Data Ref 3

Schematic of an HDF5 FileAggregated Granule

* Data arrays grow in 1 direction only


MTG03-IDP-049

7

Data


Schematic of an HDF5 FileGranule 1

File RootRoot

Attributes


Data Product Attributes

AggregatedGranules

Secondary Data

Attributes

Granule 1 Granule N

ArrayDatatype

ArrayDatatype

ArrayDatatype

Data Ref 1

Data Ref 2

Data Ref 3

* Data arrays grow in 1 direction only


MTG03-IDP-049

8

UML CASE tools to describe HDF5 file layout

– To more efficiently communicate and better control our HDF5 structures, we use Computer-Aided Software Engineering (CASE) tools to design the structure.

– Since HDF5 data structures have object-oriented characteristics, we use Unified Modeling Language (UML) to describe the nature of our HDF file structure.

– HDF5 groups, datasets, and array datatypes are UML classes; HDF5 attributes and references are UML attributes. Stereotypes are used to differentiate HDF5 objects.

– UML class derivation (HDF5 group inclusion) and multiplicity (HDF5 file structural relationships) give the user an idea about what to expect before viewing the HDF5 file.

– Data is stored in a separate HDF5 group (Data group).

– Each array datatype contains one or more hyperslab references to a subset of data in the Data group.


MTG03-IDP-049

9

HDF_File_Root

Mission_Name : H_STRN_Quasi-Static_Metadata_ReferenceIdentifier_Information.Dataset_Indentifier : H_STR

<<Root>>

HDF_EDR_Product Group

Band_Measurement_Mode_ID : H_STRInstrument_Short_Name : H_STRN_Dataset_Type_Tag : H_STRN_Instrument_Flight_SW_Version : H_STRN_Spacecraft_SW_Version : H_STRN_Subsystem_SW_Version : H_STRPlatform_Short_Name : H_STRProcessing_Level_Identifier : H_STR

(f rom HDF_File_Root)

<<H_G>>

11

HDF_EDR_Product_Granule_n Dataset

Ascending/Decending_Indicator : H_STRBeginning_Date : H_CUBeginning_Time : H_CUCloud_Cover : H_IEast_Bounding_Coordinate : H_FEnding_Date : H_CUEnding_Time : H_CUG_Ring_Latitude : H_F ArrayG-Ring_Longitude : H_F ArrayInput_Dataset_Identifier.Dataset_Identifier : H_STRN_Algorithm_Version : H_STRN_Anc_Type_Used : N_STRN_Automatic_Quality_Parameter_Value : H_FN_Automatic_Quality_Test_Explanation : H_STRN_Beginning_Orbit_Number : H_UIN_Beginning_Time_IET : H_LLN_Creation_Date : H_STRN_Creation_Time : H_STRN_Day_Night_Flag : H_FN_Degraded_Mode : H_STRN_EDR_Field : H_FN_EDR_Scale_Factor : H_FN_Ending_Time_IET : H_LLN_Graceful_Degradation : H_STRN_Granule_ID : H_IN_Granule_Version : H_STRN_LEO_Flag : H_STRN_Nadir_Latitude_Max : H_FN_Nadir_Latitude_Min : H_FN_Nadir_Longitude_Max : H_FN_Nadir_Longitude_Min : H_FN_Percent_Missing_Data : H_FN_Solar_Zenith_Angle_Max : H_FN_Solar_Zenith_Angle_Min : H_FN_Update_Date : H_STRN_Update_Time : H_STRNorth_Bounding_Coordinate : H_FOperational_Mode : H_STRSouth_Bounding_Cooridinate : H_FWest_Bounding_Coordinate : H_F

(f rom HDF_EDR_Product Group)

<<H_DS>>

1..n1..n

(EDR VIIRS SST)

Skin_Sea_Surface_Tempurature Ref : H_RRBulk_Sea_Surface_Tempurature Ref : H_RRQuality_Flags Ref : H_RR

(f rom HDF_EDR_Product_Granule_n Dataset)

<<H_RR Array>>

11

Schematic to UML mapping

Root

Attributes

Dataset

Data Product

Attributes

File

Group

Dataset

Datatype

Attributes

– HDF structural element

– a Group that also contains a header and data

– internal to HDF, used by datasets to reference read functions, included in Dataset

– where the NPOESS metadata is stored

– hyperslab reference to Data groupData Ref

Datatype

Attributes

Data Ref

* Data types are explained on chart 15-16


MTG03-IDP-049

10

HDF_File_Root


<<Root>>




<<H_G>>

11




<<H_DS>>

1..n1..n

(EDR VIIRS SST)



<<H_RR Array>>

11

Example UML HDF File Structure(EDR)

• There is a single primary Product Group and zero or more secondary Groups of supporting data.

• Each Product Group is accessed from the <<Root>> HDF element.

• Multiple Granule datasets may be provided per Product Group, depending on data requested from NPOESS. An aggregation dataset will always exist regardless of the number of Granules. The aggregation dataset will contain aggregated attributes that characterize the set of all Granules.

• Each Granule dataset as well as the Aggregation dataset contain a single Array Datatype.

• The Array Datatype contains hyperslab references to the Data group.



MTG03-IDP-049

11

Example Complete HDF5 File Structure (EDR)

Aggregate Dataset

Granule Dataset

(EDR VIIRS SST Aggregation)

Skin_Sea_Surface_Tempurature Ref : H_ROQuality_Flags Ref : H_ROBulk_Sea_Surface_Tempurature Ref : H_RO

(f rom HDF_EDR_Product_Aggregation Dataset)

<<H_RO Array>>

HDF_File_Root


<<Root>>

HDF_EDR_Product_Aggregation Dataset

<<NPOESS>> AggregateBeginningDate : H_CU<<NPOESS>> AggregateBeginningGranuleID : H_IL<<NPOESS>> AggregateBeginningOrbitNumber : H_IL<<NPOESS>> AggregateBeginningTime : H_CU<<NPOESS>> AggregateCreationDate : H_STR<<NPOESS>> AggregateCreationTime : H_STR<<NPOESS>> AggregateEndingDate : H_CU<<NPOESS>> AggregateEndingGranuleID : H_IL<<NPOESS>> AggregateEndingOrbitNumber : H_IL<<NPOESS>> AggregateEndingTime : H_CU<<NPOESS>> AggregateG-RingLatitude : H_F Array<<NPOESS>> AggregateG-RingLongitude : H_F Array<<NPOESS>> AggregatePercentMissingDataAve : H_F


<<H_DS>>

11




<<H_G>>

11

11




<<H_DS>>

1..n1..n

(EDR VIIRS SST)



<<H_RR Array>>

11

RootGroup

Data ProductGroup

Data Group

SecondaryGroup

Hidden from view

As applicable



MTG03-IDP-049

12

Example Complete HDF5 File Structure (EDR) Granule Dataset

(EDR VIIRS SST Aggregation)

Skin_Sea_Surface_Tempurature Ref : H_ROQuality_Flags Ref : H_ROBulk_Sea_Surface_Tempurature Ref : H_RO

(f rom HDF_EDR_Product_Aggregation Dataset)

<<H_RO Array>>

HDF_File_Root


<<Root>>

HDF_EDR_Product_Aggregation Dataset

<<NPOESS>> AggregateBeginningDate : H_CU<<NPOESS>> AggregateBeginningGranuleID : H_IL<<NPOESS>> AggregateBeginningOrbitNumber : H_IL<<NPOESS>> AggregateBeginningTime : H_CU<<NPOESS>> AggregateCreationDate : H_STR<<NPOESS>> AggregateCreationTime : H_STR<<NPOESS>> AggregateEndingDate : H_CU<<NPOESS>> AggregateEndingGranuleID : H_IL<<NPOESS>> AggregateEndingOrbitNumber : H_IL<<NPOESS>> AggregateEndingTime : H_CU<<NPOESS>> AggregateG-RingLatitude : H_F Array<<NPOESS>> AggregateG-RingLongitude : H_F Array<<NPOESS>> AggregatePercentMissingDataAve : H_F


<<H_DS>>

11




<<H_G>>

11

11




<<H_DS>>

1..n1..n

(EDR VIIRS SST)



<<H_RR Array>>

11



MTG03-IDP-049

13

HDF5 file structure – Example CDFCB-X Table for RDR Product Granule Group (primary data)



MTG03-IDP-049

14

HDF5 file structure – Example CDFCB-X Table for RDR Auxiliary Data Calibration Coefficients Group (secondary data)



MTG03-IDP-049

15

HDF5 file structure – HDF5 data type to CDFCB-X cross-reference


MTG03-IDP-049

16

HDF5 file structure – HDF5 attribute to CDFCB-X cross-reference


MTG03-IDP-049

17

Outstanding considerations

– Chunking for SDRs and EDRs

– Compression functions and analysis of NPOESS data records

– Storage of bit-meaningful data in bit-field data types


MTG03-IDP-049

18

Questions/Comments

Documents

UML Representation of NPOESS Data Products in HDF5