NESDIS Data Center Users Workshop (May05)

1 2005 NOAA Data and Information Users' Workshop

1

Comprehensive Large Array-data Stewardship

System (CLASS) http://www.class.noaa.gov

“What is CLASS and How Will It Serve NOAA Clientele?”

12 May 2005Richard G. Reynolds / CLASS Project Manager


2

Agenda

• Data Centers and Data Stewardship• CLASS System Introduction• Project Management• Accomplishments and Near-term Goals• Budgets and “Campaigns”• Functionality & User Services


3

NOAA’s National Data Centers

• NOAA’s National Data Centers are major archive, access, and assessment sites maintaining, processing, and distributing environmental and geospatial data. – National Climatic Data Center – WWW.NCDC.NOAA.GOV

• Asheville, NC

– National Coastal Data Development Center – WWW.NCDDC.NOAA.GOV

• Stennis, MS

– National Geophysical Data Center – WWW.NGDC.NOAA.GOV• Boulder, CO

– National Oceanographic Data Center – WWW.NODC.NOAA.GOV• Silver Spring, MD

http://www.ncdc.noaa.gov/

http://www.nodc.noaa.gov/

http://www.ngdc.noaa.gov/

http://www.nodc.noaa.gov/


4

NOAA’s National Data Centers(Continued)

• These Centers provide long-term stewardship for most of NOAA’s environmental and geospatial data, and a broad range of user services.

• Centers of data are facilities where extensive collections of given environmental parameter(s) are maintained because of individual or institutional research or operational requirements

• They also serve as Agency Record Centers


5NOAA’s National Data Centersare Environmental Data Stewards

Scientific Data Stewardship is ownership, knowledge, utilization, and

application of the data

CLASS is the Information Technology infrastructure

(hardware and software environment, and tools)

underpinning SDS

Data Rescue preserves and makes available

historical data sets from obsolete media


6

Scientific Data Stewardship (SDS)

• Observing system performance monitoring– Bias monitoring– Data character and metadata

• Climate Data Records (CDRs) – transition from research to products and services– Sentinel science teams– Blended products

• Provide useful information for national and regional management decisions– Applied climatology– Climate monitoring– Climate forcings and feedbacks


7

Principles of Environmental Data and Information Stewardship

1. Archive and access to fundamental measurements, products and metadata - CLASS

2. Data archaeology and improved use – CLASS / SDS3. Careful monitoring of observing system performance

for long-term applications - SDS4. Generation of authoritative long-term records through

validation of the calibration process, reprocessing, product generation and the blending of in situ and satellite measurements - SDS

5. Provide state of the environment information for decision makers, and place the current state in its historical context - SDS


8

CLASS Background, Mission, and

Overview


9

CLASS BackgroundThe CLASS project derives in part from an effort

by NOAA to centralize its numerous systems for

(satellite) data access.

The goal of this effort is to eliminate the various "stove-pipe” systems and

produce a unified "enterprise” access system for the NOAA environmental data holdings.


10

CLASS Mission Statement

NOAA's National Data Centers and their world-wide clientele of customers look to CLASS as the sole NOAA IT infrastructure project in which all NOAA’s current and future environmental data sets will reside. CLASS provides permanent, secure storage, and safe, efficient data discovery and access between the Data Centers and the customers.


11

CLASS Goals

• Give any potential customer access to all NOAA and non-NOAA data

through a single portal.

• Eliminate the need to keep creating “stovepipe” systems for each new type of data,

but, in as much as possible, use already polished portions/modules

of existing legacy systems.

• Describe a cost-effective architecture that can primarily handle large-array data sets,

with the capability of handling smaller ones as well.


12

CLASS Overview• CLASS is a web-based data archive and distribution

system for NOAA/NESDIS environmental data– Archive … ingest, storage, metadata management, and data

quality assurance– Distribution … access, visualization, and data delivery

• CLASS is an extension of an 1995 operational system … SAA (Satellite Active Archive)– Transition to the CLASS architecture began in 2001– CLASS subsumed SAA as the Operational Archive and Access

System for NOAA in 2004

• CLASS currently supports POES, DMSP, and GOES data sets

• CLASS will support additional campaigns, broader user base, new functionality as it evolves– CLASS concurrently supports ongoing operations and new

requirements implementation


13

CLASS Overview (Continued)

• Provide one stop shopping and access capability for NOAA and NESDIS environmental data and products

• Provide a common look and feel for accessing NOAA and NESDIS environmental data and products

• Provide an efficient architecture for archiving and distribution of NOAA and NESDIS environmental data and products

• Reduce implementation costs by using reengineering, and evolutionary effort

• Allow NOAA to fulfill its requirements regarding archive, access and distribution of large array data sets


14

CLASS Overview (Continued)

• Accommodate expanding number of data sources – MetOp, NPP, NPOESS, EOS, In-situ, NexRAD,

GOES-R, etc.

• Data volume is growing exponentially– Anticipating up to 100 Petabytes by 2015

• User volume is also growing exponentially


15

“CLASS” Synonyms

• Comprehensive Large Array-data Stewardship System (CLASS)

• Archive and Distribution Segment (ADS)

• NPP/NPOESS

• Archive and Access System (AAS) • METOP/IJPS

• Long-term Archive (LTA)• Jason


16CLASS … as part of the NOAA Observing System Architecture

NOSA … The “End-to-End System”-- Notional Architecture --

Larger System

Observing System

Data Handling System

Human

Environmental Phenomenon

Environmental Parameter

Sensing Element

Sensor

Platform / Station

part of

measures

is a

contains

characterizes

provides data to

Observation Control System

is controlled by

Location

located at

MobileFixed

is

SpaceAirGroundOcean

SpaceAirGroundOcean

Basic Service Requirement

< drives

provides data directly to

Userprovides info to is type of

Stake-holder

has

Operatoroperated by

situated on

Support

supported by

Owner

owned by

Stakeholder Requirement

< drives< drives

Processing Element

RemoteIn Situ

is type

CLASS


17CLASS Technical Description Functional Flow Diagram

Ingest and Store Data

VisualizationData

Data SetInventory

DataCaches

Orders

Maintain,Monitor,Control

ProcessOrders

AccessData

VisualizeData

Interfacewith Users

Data Productsand Metadata

DataProviders

USERS

CLASSOperators

Archive

CLASS Internet/Intranet

DataProvidersData

Providers

DataProvidersData

Providers

USERSUSERS

USERS

USERS


18CLASS Technical Description Functional Block Diagram


19

CLASS Project Management


20

CLASS Project• Consortium of NOAA Projects and Budget

Lines have been consolidated under the CLASS Project:

– CLASS

– Satellite Active Archive (SAA) – GOES Active Archive (GAA) – Earth Observing System (EOS) – Integrated Joint Polar Program (IJPS) /

MetOp AAS– GOES R-series (GOES-R) AAS– [NOAA Virtual Data System (NVDS) and NOAA E-commerce

System (NeS) moved to NCDC]


21

CLASS Project Plan

• 10 year “PAC” Plan (Procurement, Acquisition, & Construction)

– Road Map for CLASS Program acquisition

– Budgetary Funding Requirements for all CLASS elements

– Life Cycle Planning document


22

NOAA DataStewardship Committee

(Tom Karl/NCDC)

CLASS (Richard G. Reynolds/OSD)

NESDIS - Office of Systems Development,

Data Centers, and contractors

Archive Requirements

Working Group (ARWG)

(John Bates/NCDC)

Information Exchange


23

CLASS Project ManagementNOAA

Data Stewardship Committee

CLASS ProjectRichard G. Reynolds

Charles S. Bryant

CLASS Project Management Team (CPMT)

NGDC Development

Teams (Boulder, CO)

OSD/TMCDevelopment

Team(Fairmont, WV)

OSD/CSCDevelopment

Team (Suitland, MD)

System Integration & Test Team

(Suitland, MD)

OSDPD-CSC Operations

(Suitland, MD)

NCDC-TMC Operations

(Asheville, NC)

Archive RequirementsWorking Group (ARWG)

NESDIS ITATUsers

System Engineering Team (SET)

CLASS Operations Team (COT)

System Administration Team (SAT)


24

CLASS - Process DocumentationCLASS Master Project

Management Plan

CSC Contract Activity Plan

CLASS Software

Development Guide

CLASS Configuration

Management Plan

Other Activity Plans

Procedures

CLASS Quality

Management Plan

TMC Contract Activity Plan


25

Project Management (Continued)

• CLASS Project Oversight Groups– Data Stewardship Committee (DSC)– NESDIS Headquarters / AA and DAA– NESDIS Information Technology Architecture Team (ITAT)

- - - - - - - - - - • CLASS Project Management Team (CPMT)

– Management oversight and coordination– Risk management– Project tracking– Overall decision-making body for CLASS

• System Engineering Team (SET)– Technical oversight and coordination– Provides recommendations to CPMT on technical direction– Provides technical input to CCB


26

Project Management (Continued)

• Software Engineering Process Group (SEPG)– Process definition and improvement– Provides recommendations to CPMT on process definition

• Configuration Control Board (CCB)– Change review and control– Includes CPMT members and SET members

• Systems Administration Team (SAT)– Develop, Maintain, and Supervise Policies and Procedures

regarding configuration and operation of CLASS computing resources

• CLASS Operations Team (COT)– Develop and Implement operational policies and Standard

Operating Procedures for the operational CLASS environments – Includes operations support personnel, system administrators,

database administrators and archive managers


27

CLASSAccomplishments


28

CLASS Accomplishments• Completed overall design of CLASS top-level architecture • Prepared Key Project and System Documentation

– CLASS Business Case– System Requirements– Interface Control Documents (ICDs)– Concept of Operations (CONOPS)– Management Plans and Procedures

• Established a project wide risk management program• CSC Development Team Certified at SEI-CMM Level-3• Established the CLASS Operations Team (COT)• Completed … Summary “10-year” CLASS Project Budget

Requirement - $25M/year


29

CLASS Accomplishments (Continued)

• Delivered baseline systems to Suitland and Asheville– Established Operational Dual-site Configuration

• Established Operational, Integration and Test, and Development environments in Suitland– Completed migration from SAA to CLASS

• CLASS Operational with POES, DMSP, and GOES data sets

• Completed Preliminary & Critical Design Reviews for the IJPS/Metop – Archive and Access Segment

• Coordinated with NPP/NPOESS for defining the IDPS-to-CLASS Interface Control Document – Completed … NPP/NPOESS Campaign Implementation Plan

• Worked with NASA personnel to define initial requirements to archive EOS/MODIS Level-0 data.


30

CLASS Accomplishments (Continued)

• Completed … Software Release 2.0 / 2.1 / 2.2 – Operational Dual-site configuration -- 02 April 2004– CLASS Operational with POES, DMSP, and GOES data sets, plus

RadarSat (Synthetic Aperture Radar) and SeaWiFS (Ocean Color Product)

• Completed … Software Release 3.0 – 12 July 2004– Provides: Delivery Manifest and Web Enabled Subscription

Management • Completed … Software Release 3.1 – October 2005

– Provides: Ingest Enhancements to support IJPS NOAA data

• Completed … Integration of the Development Teams – CSC-TMC

• Began … SEI/CMMI Certification Process for the total Development Team


31

FY05 CLASS Goals & Plans• Prepare System security Certification & Accreditation (C&A)

– CLASS currently encompassed by the Satellite Active Archive C&A– Draft stand-alone CLASS C&A prepared (March 2005)– Final to be completed to support relocation to new NSOF Building in

September 2005

• Update the CLASS Long-term Architecture– Held 2-day architecture session confirming/upgrading the long-term

architectural plan for CLASS– FY2005 CLASS Long-term Architecture Plan to be released in September

2005

• Achieve Hardware/Software Commonality among all Nodes

• Project/Development Teams SEI-CMMI Certified– Initial review completed (January 2005)– Final assessment to be conducted in May 2005

• Relocate Suitland Node to Boulder (NGDC)


32

FY05 CLASS Goals & Plans (Continued)

• METOP-1 Pre-Launch Testing and Operational Readiness (Completed March 2005)– Capability contained in Release-3.2– Awaiting data flows from the Spacecraft for final functional verification

• NPP “Campaign” Development and Testing • EOS-MODIS “Campaign” Development and Testing• Establish an interface with NeS• Establish an interface with NMMR• Metadata “Campaign” development continues• Geospatial Capability development begins• Jason/OSTM ”Campaign” development begins• Establish a Development Environment at TMC/Fairmont

----------------• Operations Continue


33

FY05 Hardware/Software Plans• System SAN Capacity Upgrade (Completed January

2005) • Additional disk space at both CLASS operational sites• Data Direct Networks … 56 Tbytes (expandable to 302 Tbytes)

• CLASS Release 3.2 (Completed April 2005)• Support for Metop-1 data / readiness for IJPS End-to-End test • Subscription for GOES data w/ separate GVAR data ‘families;”

GOES-N• Upgrade all systems to AIX 5.1/5.2 (64-bit structures)

• CLASS Release 3.3 (Scheduled for July 2005)• McIdas-less ingest • Upgrades to the Help Pages/Static Pages • Map server upgrades; Point Searches• CLASS-NMMR Interface • Security enhancements, including capability to deliver data

encrypted• UTC Time utilization


34

FY06 CLASS Goals & Plans

• METOP-1 Operational Activation• NPP “Campaign” End-to-End Testing,

Compatibility Testing, and Operational Readiness • Metadata “Campaign” development continues• Geospatial Capability development continues• Jason/OSTM “Campaign” Testing• Data QA/QC “Campaign” begins• Reprocessing “Campaign” begins


35

FY06 Hardware/Software Plans• System Storage Capacity Upgrade

– Scheduled for September 2006– LTO-2 to LTO-3 Migration

• CLASS Release 4.0 (Scheduled for October 2005)• Basic NPP Support • Final IJPS/Metop Pre-launch Release • CLASS – NeS Interface • CLASS – NMMR Interface

• CLASS Release 4.1 (Scheduled for February 2006)• NPP Readiness for NCT-#3 • Initial Data Delivery Upgrades

• CLASS Release 4.2 (Scheduled for August 2006)• NPP Readiness for NCT-#4

• CLASS Release 4.3 (Scheduled for October 2006)• NPP Pre-Launch Release• Complete Data Delivery Upgrades


36

CLASS Statistics(Average Last 12 Months)

• Ingest – 71 GB/Day … 26 TB/Year

• Ingest – 860,000 Data Sets/Year

• Distribution (On-line & Subscriptions) - 44 TB/Year …. 3.63 TB/Month

• Distribution (On-line & Subscriptions) - 3,170,000 Data Sets/Year … 263,888 Data Sets/Month


37

CLASS BUDGETSand “Campaigns”

Budget numbers are shown for the purpose of establishing a reference for relative complexity of a

requirement and level of completeness, and do not represent NOAA, Department of Commence, or

The President’s position regarding specific Congressional Budget Requests.


38

Major Core-CLASS Project“Functional Campaigns”

• “Core CLASS” Baseline System Development, Expansion, & Evolution

–FY04-FY16 $94M•Metadata “Campaign”

–FY04-FY14 $12M•Reprocessing “Campaign”

–FY06-FY16 $35M•QA/QC

–FY06 …. $2M/year--------------------

•System O&M–FY04 ($2M)-FY14($10M) $11M/year thereafter


39

General Core-CLASS Activities

• Support for CLASS Development Activities– Architecture Design– System Design– Software Development and Integration– Quality Assurance– System Engineering Team Activities

• Support for CLASS Operations and Maintenance– Contractor and Government Project Support– Licenses– Hardware Maintenance– Hardware Refresh


40General Core-CLASS Activities (Continued)

• Metadata– Assure FGDC Compliance – Establish Rich Metadata Standards– Develop Integrated data and metadata access

• Geospatial– Capability development and implementation– Integration of all CLASS data

• Web– Integrate SABR into CLASS– Design and implement web mapping system – Enhancements to support Data Mining capability

• Data Mining– Architecture design and implementation– Application design and implementation

• Reprocessing– Architecture design and implementation– Establish operational processes and standards


41Major CLASS Project“Data Campaigns”

•Metop-1–FY01-FY07 $6.5M

•NPP –FY04-FY11 $15.8M

•EOS-MODIS –FY04-FY10 $16.7M

•NDE –FY06-FY12 $1.5M

•NEXRAD–FY06-FY09 $8.1M

• Insitu –FY06-FY14 $8.0M

•EOS Retrospective–FY07-FY13 $8.2M

•GOES-R–FY07-FY14 $41M

•Metop-2–FY08-FY14 $5.0M

•NPOESS-C1–FY08-FY12 $11.1M


42

General “Data Campaign” Requirements

• Requirements Definition and ICDs

• Data and Products Ingest

• Storage, Processing, and Communications Upgrades

• Metadata extensions

• Catalogue extensions

• Visualization extensions

• Reprocessing extensions


43

CLASS Budgets• FY01 $1.995M • FY02 $3.599M• FY03 $2.881M

-----------• FY04 $10.5M• FY05 $14.6M• FY06 $11.9M • FY07 $18.0M *• FY08 $20.8M *• FY09 $25.5M *• FY10+ $29.3M/yr

*– * FY07 PA&E Recommendations

– FY05 Funding Sources

– CLASS $6.6M– GAA $2.5M– EOS $3.0M– SAA $1.5M– OSD $1.0M **

** POES Ground Systems PAC – METOP/IJPS Archive & Access $200K

CLASS Hardware $800K


44

FY06-07 CLASS Budgets• FY07 $20.0M

• CLASS $15.1M

• SAA $1.5M

• EOS $1.0M

• GOES-R $2.0M

• OSD $0.4M

FY06 $11.9M

CLASS $6.6M

SAA $1.5M

EOS $3.0M

OSD $0.8M


45

CLASSFunctionality and

User Services


46

Core Functionality

• Data and metadata ingest, archive, and replication (NARA compliant)– Delivery Manifests– Data Quality– Companion Files– Metadata Management

• User Data Discovery– Access, Search, and Visualization


47

Core Functionality (Continued)

• User Access– Web– Single Sign-on/On-line– Subscription– Machine-to-Machine/Bulk Order

• User Search– Inventory– Temporal /Spatial – Data Type– Metadata Characteristics

• User Visualization– Imagery Chips– Geospatial Navigation– Overlays


48


• Data Delivery System– Improved use of LTO tape robotic system capabilities– Consolidate multiple order delivery systems (Online,

Subscription, Bulk Order)– Improved cache management– Delivery Options

• Physical media creation and shipment• Electronic delivery

– E-Commerce (NESDIS E-commerce System – NeS)• Pricing, Payment receipt, Payment processing

– User Profiles

• Data Denial


49


• Data Delivery to the User– Format options– Push– Pull– Physical Media – Delivery Notifications

• User Helps

• New Requirements … via the ARWG!


50

THANK YOU!