Page 1

Land PEATE support for CERES processing

Ed MasuokaGang Ye

August 26, 2008

CERES Delta Design Review

Page 2

Element Overview

• Facilitate VIIRS Science Team assessment of Land EDRs for use as Climate Data Records

• Facilitate evaluation of improved Land EDR algorithms provided by Science Team

• Support NICSE Cal/Val activities through processing SDRs as requested

• Provide access to data products and results of product quality assessment to VIIRS Science Team

• Support CERES Instrument Team by providing CERES RDRs and VIIRS SDR subsets and VIIRS Aerosol EDRs in MODIS format (5 minutes granules in HDF-EOS)

Page 3

Overview: SDR/EDR evaluation

• Land PEATE is sized to produce VIIRS SDR and EDRs at 3x post-launch for two versions of code (operational algorithms and improved algorithms) and to produce global gridded products from VIIRS EDRs for comparison with MODIS.

• Products run with specific versions of software are maintained in separate archive sets

• Product retention is governed by online storage• Improvements to algorithms running in IDPS will be

provided by the VIIRS science team.• Land Q/A team will post results of their analyses of product

quality at: http://landweb.nascom.nasa.gov/cgi-bin/NPP/NPPlogin.cgi

Page 4

Science and Calibration Team

Science Team ProductR. Wolfe Geolocation J. Xiong (NICSE), E. Vermote Calibrated RadiancesE. Vermote, A. Lyapustin Land Surface ReflectanceC. Schaff AlbedoZ. Wan, S. Hook Land Surface TemperatureD. Hall, J. Maslanik Snow Cover M. Friedl Surface TypeI. Csiszar, C. Justice Active Fires, (Burned Area)D. Hall, J. Maslanik Ice Surface TemperatureA. Huete, R. Myneni Vegetation Indices, (LAI)

Page 5

Land PEATE Test Result

• Results from evaluation of each test by LDOPE are provided

• Links to following are included:– Global browse– Issues uncovered in

the data products– Time series plots

Page 6

Land PEATE – VIIRS systems

Gigabit Ethernet Switch

Black Diamond 10808

Production Databaseopsdb4

Backup Database

nppdb_bk1

StorageNodenstor1

StorageNodenstor5

...

100

TB

Processing Nodeminion 5055

Processing Nodeminion 5135

...

Product Storage5 storage servers

PROCESSING - 80 servers

19TB

...

19TB

114TB

19TB

...

19TB

114TB

Databases 2 primary, 1 backup

Ingest and Scheduling

hostmtvs4Data

Providers

INGEST, SCHEDULING

Operator

Distribution Databaseladsdb1

Web Nodelandweb1

SEARCH, ORDER AND

DISTRIBUTIONVIIRSTeams

Shared resource with MODIS

Page 7

CERES resources in Land PEATE

• 12 Dual quad-core compute servers for product generation• 1 Dual core server and 19 TB of disk for product storage• Supports 1x forward processing plus reprocessing for SDR

generation and subsetting and Aerosol EDR generation

Page 8

Overview: CERES support

• Land PEATE provides CERES RDRs and VIIRS inputs to CERES processing in MODIS format on PDR server at GSFC using the SIPS Interface

• Land PEATE runs VIIRS SDR subsetting software developed by CERES to produce subsets with desired bands

• Land PEATE runs Aerosol PGE (IDPS or Science Team improved) to produce VIIRS Aerosol in MODIS format

• Land PEATE will reprocess the VIIRS input products as requested by CERES to support of the generation of a long-term climate record using hardware acquired for CERES support.

Page 9

Operations Concept for CERES support

• IDPS creates CERES Raw Data Records (RDRs) and provides RDRs to the SDS SD3E and to ADS/CLASS

• Land PEATE pulls CERES RDRs and VIIRS xDRs from SD3E• Land PEATE places CERES RDRs on GSFC PDR server• Land PEATE runs sub-sampler on SDR and places output on

GSFC PDR server• Land PEATE places Aerosol EDR in MODIS format (5 minute

granules HDF-EDOS) on GSFC PDR server• Land PEATE handles clean-up of files in CERES directory on

PDR server based on PANs provided by ASDC as defined in the SIPS Interface

• Land PEATE will reorder products for CERES from CLASS or remake products upon request

Page 10

Requirements for CERES support• Land PEATE shall ingest CERES RDRs from the SD3E• Land PEATE shall integrate, test and run in production VIIRS

SDR subsetting software developed by CERES IT• Land PEATE shall produce VIIRS Aerosol product in MODIS

format (HDF-EOS, 5 minute granules)• Land PEATE shall transfer CERES RDRs, VIIRS SDR subsets

and VIIRS Aerosol EDR to ASDC via PDR server• Land PEATE shall meet latency requirements in CERES

Science Data Products Working Agreement for product delivery• Land PEATE shall reorder or remake CERES RDRs, VIIRS

SDR subsets or Aerosol EDR upon request by CERES• Land PEATE shall reprocess VIIRS inputs for CERES Climate

Data Record upon request by CERES IT with versions of VIIRS software selected by CERES IT.

Page 11

CERES Operational Data Flow

SD3E L-PEATE

ADS

IDPS

SDS

All RDRs

All xDRs

CERES RDRs, VIIRS SDR sub-sampled, VIIRS Aerosol EDR in MODIS format

C3S

Commands, Loads and requests

RDRsSDRsEDRs

IPs

CERES InstrumentOps Team

Svaalbard

Cmd &Tlm

Mission Data

CERES Data Data Users

Mission Notices and Data requests

RDRs for LEO&A

CERESPEATE Data and Science Operations

Mission Notices and Data requests

Ancillary data

Data Providers

NPOESS Systems

NPP Systems

NOAA Systems

CERES Systems

GSFC PDR server

MODIS System

Page 12

MODAPS/LPEATE-ASDC Interface

• SIPS Interface– ASDC will implement EOSDIS SIPS PDR-PAN interface

in their new system (ANGe).– Products will be placed in a disk partition reserved for

NPP CERES on the MODIS PDR server located at GSFC– ASDC will transfer products from this directory using

SIPS protocol.

Page 13

Operations Concept for CERES: Reprocessing for Climate-quality

• During 2 years of VIIRS post-launch product evaluation, algorithm improvements will be developed by the VIIRS Science Team members and evaluated by NVCSE (VIIRS Calibration Support Element), LDOPE (Land Q/A team) and VIIRS Science Team.

• Results of these evaluations will be available at LDOPE, NVCSE and ST web sites.

• CERES Science Team will determine versions of SDR and Aerosol software and calibration tables* to use in reprocessing on Land PEATE resources dedicated for CERES support.– Reprocessing will proceed from earliest well calibrated scene to present

using best available calibration.

* NVCSE is tasked only with evaluation of SDR, so complete calibration tables to ensure a climate-quality SDR may not be available.

Page 14

Status of PEATE development• Proxy data generator (extended SDRgen) running in Land

PEATE produces VIIRS SDR (MODIS bands with VIIRS geometry).*

• 16-days of proxy SDRs has been produced to test code integrated from IDPS Build 1.5.0.15.1– Cloud Mask, Active Fires, Aerosol Optical Thickness, Surface

Reflectance and Vegetation Indices– Geolocation problem in Build 1.5.0.15.1 fixed in current delivery

1.5.0.25.2 . – Aerosol calculations use 6x6 elements not 8x8

• Additional days of proxy SDRs will be generated for CERES in October

* CERES requested simulated day/night band be added and that modification will complete testing by 10/1/08

Page 15CERES FM-5 Delta Design Review

15

Schedule for CERES support8/20/08: CERES compute servers received - COMPLETED

9/30/08: CERES disk storage received

10/1/08: SDRGEN modified to add simulated day/night band

10/1/08: VIIRS sub-sampling software complete and tested at LaRC

10/3/08: VIIRS sub-sampling software delivered to Land PEATE for integration by CERES

11/5/08: VIIRS sub-sampling software completes PEATE integration tests

11/27/08: FMD connects new power for SDS computing facility

12/1/08: VIIRS sub-sampling software running at SDS Land PEATE, expected sub-sampled output has been verified by CERES DMT

1/2/09: CERES processors and disk storage installed at Land PEATE . Dependency on new power in November.

1/15/09: Large set of VIIRS sub-sampled output produced at SDS Land PEATE and verified at LaRC by CERES DMT

Page 16

Open Items• Sample of CERES RDRs are needed to work ingest of these

files into Land PEATE for delivery to CERES. When?• Aerosol using 8x8 calculation (in Build 1.5.?). • SIPS-interface/PDR server tests with ASDC does not involve

Land PEATE delivery to ASDC system but rather ASDC pulling files from GSFC PDR server per working agreement.

• Define how product re-order requests will come from CERES (email message with or w/o an attached file, or a file pushed to an ftp directory). Define content of message or file.

• NVCSE is not responsible for ensuring the calibration tables are adjusted to maintain a climate quality NPP VIIRS record

Page 17CERES FM-5 Delta Design Review

17

SDS/Land PEATE Assumptions

• Subscribe to SD3E for all CERES RDRs (Science, Diagnostic, and Housekeeping/Telemetry)

• Acquire CERES RDRs and forward to ASDC via existing Land PEATE interface

• Support CERES reordering of missing data• Produce aggregated VIIRS SDRs (MODIS HDF-EOS

format) in 5 minute granules (VIAE, VMAE, VDAE). Sub-sample and ship 5 minute granules (VIMD) to ASDC

• Produce VIIRS aerosol in 5 minute granules (VAOT) and ship to ASDC

• Post-launch: When VIIRS calibration and/or algorithm changes, reprocess VIIRS from covers-open forward and ship to ASDC to support CDRs