NOAA’s NWS Telecommunication Gateway RTH Washington ICM MTN ET-OI, Sep 2008 Office of the Chief Information Officer NOAA’s National Weather Service Office

NOAA’sNWS Telecommunication Gateway

RTH WashingtonICM MTN ET-OI , Sep 2008

NOAA’sNWS Telecommunication Gateway

RTH WashingtonICM MTN ET-OI , Sep 2008

Office of the Chief Information Officer

NOAA’s National Weather Service

Office of the Chief Information Officer

NOAA’s National Weather Service

“Taking the pulse of the planet”“Taking the pulse of the planet”

Data Input to the NWSTGData Input to the NWSTG

MPLS

IP / Sockets

X.25

Asynchronous

FTP – http://weather.gov/tg/ftpingest.html

Email – http://weather.gov/tg/emailingest.html

Web - http://weather.gov/tg/bullguid.html

MPLS

IP / Sockets

X.25

Asynchronous

FTP – http://weather.gov/tg/ftpingest.html

Email – http://weather.gov/tg/emailingest.html

Web - http://weather.gov/tg/bullguid.html


Replacement & Backup RTHsReplacement & Backup RTHs

Full functional replacement of existing capabilities

Expanded capacity and capability

– Input data growth from 200 to 800 GB/day

– Output data growth from 800 to 2400 GB/day

Transition to new technology

– Middleware for internal transport

– Network-centric systems interconnectivity

– Central switching engine with relational database

– SAN and NAS storage solutions

Highly scaleable architecture

Hardware refresh

Full functional replacement of existing capabilities

Expanded capacity and capability

– Input data growth from 200 to 800 GB/day

– Output data growth from 800 to 2400 GB/day

Transition to new technology

– Middleware for internal transport

– Network-centric systems interconnectivity

– Central switching engine with relational database

– SAN and NAS storage solutions

Highly scaleable architecture

Hardware refresh


RTG/BTG PerformanceRTG/BTG Performance

Performance Metric Threshold System Availability (averaged monthly) 99.90% Warning Message Latency (averaged monthly) 10 seconds

Routine Message Latency (averaged monthly) 60 secondsDaily Traffic Volume (averaged monthly) 1.2TB

Typical Performance Results (averaged monthly)

Performance Category Threshold ActualSystem Availability 99.90% 100%Warning Message Latency 10 seconds < 1 secondRoutine Message Latency 60 seconds < 1 secondDaily Traffic Volume 1.2TB 1.2 - 1.5TB

Performance Metric Threshold System Availability (averaged monthly) 99.90% Warning Message Latency (averaged monthly) 10 seconds

Routine Message Latency (averaged monthly) 60 secondsDaily Traffic Volume (averaged monthly) 1.2TB

Typical Performance Results (averaged monthly)

Performance Category Threshold ActualSystem Availability 99.90% 100%Warning Message Latency 10 seconds < 1 secondRoutine Message Latency 60 seconds < 1 secondDaily Traffic Volume 1.2TB 1.2 - 1.5TB


RTH WashingtonRTH Washington

Can route more than 50 routine messages per second with reliability for all dissemination to all of its users of 99.9 percent

Latency for high priority traffic of 10 seconds or less and routinely disseminates 1.2 terabytes of information per day

On Main Trunk Network (MTN) of the WMO Global Telecommunication System (GTS)

– GTS delivers tsunami data and warnings to connected MTN centers within two minutes

On AFTN network (KWBCYMYX) & ICAO OPMET Databank

Can route more than 50 routine messages per second with reliability for all dissemination to all of its users of 99.9 percent

Latency for high priority traffic of 10 seconds or less and routinely disseminates 1.2 terabytes of information per day

On Main Trunk Network (MTN) of the WMO Global Telecommunication System (GTS)

– GTS delivers tsunami data and warnings to connected MTN centers within two minutes

On AFTN network (KWBCYMYX) & ICAO OPMET Databank


NOAANet Logical DesignNOAANet Logical Design


Primary / BackupNWSTG Access

Primary / BackupNWSTG Access


MS3

FEP #1Sockets

FEP #2X.25

FEP #8 MSD

FEP #3FTP

FEP #4Async

FEP #7Marta

FEP #6eMail

FEP #5Web

MS3 #1

#1

MS3

#2

#3

#4#8

#7

#6

#5

MS3 #2

MS3 #3

MS3 #4

MS3 #5

MS3 #6

MS3 #7

MS3 #8

Legend

Front EndProcessor (FEP)

MS3 Hub

Process FlowsWebSphere MQ Integrator

MS3

#1

MS3 inbound queue

MS3 customer queue

WebSphere MQ Queues

WeatherData

Source/Customers

Weather ProductPrefix

MS3 Message

Logical cluster queue

Weather Product

Weather ProductPrefix


Major Software ComponentsMajor Software Components

Core software components of the NWSTG replaced in 2006

Data switching software

– Core software rewritten and implemented in IBM WebSphere Message Broker

– Message Broker provides an application execution environment, threading, and fourth generation language support (ESQL), integrated with middleware services

Data transport infrastructure (middleware)

– Shared data stores replaced with IBM MQSeries Middleware

– Applications send data to each other without concern for target location

– Data delivery guaranteed even during failures

Data storage infrastructure (relational database)

– All data stores moved into Sybase relational database

Core software components of the NWSTG replaced in 2006

Data switching software

– Core software rewritten and implemented in IBM WebSphere Message Broker

– Message Broker provides an application execution environment, threading, and fourth generation language support (ESQL), integrated with middleware services

Data transport infrastructure (middleware)

– Shared data stores replaced with IBM MQSeries Middleware

– Applications send data to each other without concern for target location

– Data delivery guaranteed even during failures

Data storage infrastructure (relational database)

– All data stores moved into Sybase relational database


Internal RedundancyInternal Redundancy

Internal redundancy implemented for critical processes

Active – Warm Standby

– Standby system running at all times, system disks moved to standby system during failover

– EXAMPLES: Individual front-end processors, application servers

Active – Hot Standby

– Data replicated to Hot Standby system in realtime, automatic failover

– EXAMPLE: Sybase relational database servers

Active - Active

– Multiple systems active in parallel, data shared/routed between systems groups

– EXAMPLES – Switching system, HTTP/FTP server groups

Internal redundancy implemented for critical processes

Active – Warm Standby

– Standby system running at all times, system disks moved to standby system during failover

– EXAMPLES: Individual front-end processors, application servers

Active – Hot Standby

– Data replicated to Hot Standby system in realtime, automatic failover

– EXAMPLE: Sybase relational database servers

Active - Active

– Multiple systems active in parallel, data shared/routed between systems groups

– EXAMPLES – Switching system, HTTP/FTP server groups


Architecture OverviewArchitecture Overview


Architecture OverviewArchitecture Overview


System ExtensibilitySystem Extensibility

Virtualized server hardware platform selected for extensibility

Virtualization

– System resources (CPUs, Memory, Internal Storage, Network Interfaces, etc.) available within hardware frame

– Individual resources selected to create virtual servers

– Major storage implemented in Storage Area Network

– Virtual disks allocated to virtual servers

Extensibility

– Unused resources can be allocated to virtual servers, as needed

– Underused resources can be removed from a virtual server

– Virtual disks can be extended or reduced as storage requirements change


Virtualization

– System resources (CPUs, Memory, Internal Storage, Network Interfaces, etc.) available within hardware frame

– Individual resources selected to create virtual servers

– Major storage implemented in Storage Area Network

– Virtual disks allocated to virtual servers

Extensibility

– Unused resources can be allocated to virtual servers, as needed

– Underused resources can be removed from a virtual server

– Virtual disks can be extended or reduced as storage requirements change




Advanced Virtualization

– Network and SAN I/O interfaces can be shared through virtual I/O

– CPUs can be carved into fractional components down to 1/10th of a CPU

Advanced Extensibility

– Resources (CPU, Memory) included within system that were not purchased

– If additional resources are needed, resources are “turned on” via software key after purchase and are immediately available


Advanced Virtualization

– Network and SAN I/O interfaces can be shared through virtual I/O

– CPUs can be carved into fractional components down to 1/10th of a CPU

Advanced Extensibility

– Resources (CPU, Memory) included within system that were not purchased

– If additional resources are needed, resources are “turned on” via software key after purchase and are immediately available


RTH Washington WIS approach


RTG/BTG upgrades made with knowledge of coming WIS/GISC requirements

– Production and Replicated core systems

• All functions not essential to mission critical operations (i.e. switching) execute against replicant

– SOA Approach to design

– Integral file, message and application switching

– RDBS core design

• ~ 2000 destinations defined

• ~ 400K defined products

• ~ 10M entries (rows) in switching table

– Many tables used for managing bulletin, file and report storage including dissemination and services metadata.

RTG/BTG upgrades made with knowledge of coming WIS/GISC requirements

– Production and Replicated core systems

• All functions not essential to mission critical operations (i.e. switching) execute against replicant

– SOA Approach to design

– Integral file, message and application switching

– RDBS core design

• ~ 2000 destinations defined

• ~ 400K defined products

• ~ 10M entries (rows) in switching table

– Many tables used for managing bulletin, file and report storage including dissemination and services metadata.




We view Data Access & Retrieval (DAR) service as the only new service provision

– Plan to lay this service onto existing architecture

– Will break up DAR functions

• DAR catalog creation & update

• discovery & access servicing

– DAR will not execute on primary production system (at least in early stages of implementation

We view Data Access & Retrieval (DAR) service as the only new service provision

– Plan to lay this service onto existing architecture

– Will break up DAR functions

• DAR catalog creation & update

• discovery & access servicing

– DAR will not execute on primary production system (at least in early stages of implementation




Develop metadata catalog first in conjunction with national & regional partners

– Will leverage knowledge expertise of partners

– Will expand to all RTH (GISC) data holdings and holdings of appropriate Centers (RSMC/DCPC/NC) and willing partners.

– Work will be made available to all Members

– Will collaborate with WIS Project Team, WMO ETs and developers in other Regions

– Regional WIGOS Development Project will meet WIS requirements

• RA-IV Integrated Atmosphere Observing System

• With initial emphasis on upper-air observations

Develop metadata catalog first in conjunction with national & regional partners

– Will leverage knowledge expertise of partners

– Will expand to all RTH (GISC) data holdings and holdings of appropriate Centers (RSMC/DCPC/NC) and willing partners.

– Work will be made available to all Members

– Will collaborate with WIS Project Team, WMO ETs and developers in other Regions

– Regional WIGOS Development Project will meet WIS requirements

• RA-IV Integrated Atmosphere Observing System

• With initial emphasis on upper-air observations


Meteorological Assimilation Data Ingest System (MADIS) Transition to Operations

SFC-LANDSFC-MARINE

U/A-IN SITU

U/A-REMOTESENSING

SATELLITE

GRIDSMETADATA

MADISCollection,

QC, andDistribution

InformationBases

(QCed Datasets)

Quality ControlInformation

(Data QC Flags)And Metadata

PM 12/12/05


MADIS Ingests and Performs QC on Meteorological Observational DataMADIS Ingests and Performs QC on Meteorological Observational Data

Meteorological Surface

– METAR

– Airways

– Maritime

– Modernized NWS Cooperative Observer

– UrbaNet

– Integrated Mesonet

• State DOT

• Mesonets

• AWS

RSAS Surface Grids

Meteorological Surface

– METAR

– Airways

– Maritime

– Modernized NWS Cooperative Observer

– UrbaNet

– Integrated Mesonet

• State DOT

• Mesonets

• AWS

RSAS Surface Grids

NOAA Profiler Network

Hydrological Surface

Automated Aircraft

Multi-Agency Profiler

Cooperative Agency Profiler

Radiosonde

Radiometer

Satellite Wind

– NOAA GOES Products

Satellite Radiances and Soundings

– NOAA POES

Snow

NOAA Profiler Network

Hydrological Surface

Automated Aircraft

Multi-Agency Profiler

Cooperative Agency Profiler

Radiosonde

Radiometer

Satellite Wind

– NOAA GOES Products

Satellite Radiances and Soundings

– NOAA POES

Snow

“Taking the pulse of the planet”“Taking the pulse of the planet”TOC Environment

Gateway

MADIS-T

FTP

Ex

tern

al I

SP

LDM

FTP

Storage(SAN)

Automated QC processing

Gateway Customers (NCDC, NCF, GTS etc)

Data Processing

LDM

NOAA Customers

Non-NOAAObservations

LDM

Integrated Datasets

Web Server

Internet

Web

Ser

ver

?

?

?Individual Data

Products

Tunnel


RTH Washington Other Issues

RTH Washington Other Issues

– Foundation of Programs is Observing Systems & Data Exchange

– Transition of GTS-MTN

• Two existing clouds to ???

– Cloud 1 - frame relay

– RMDCN – satellite MPLS

– Addition of Guam/Hono to AP-RARS

– Future operations locations

• Backup first then Primary

– Foundation of Programs is Observing Systems & Data Exchange

– Transition of GTS-MTN

• Two existing clouds to ???

– Cloud 1 - frame relay

– RMDCN – satellite MPLS

– Addition of Guam/Hono to AP-RARS

– Future operations locations

• Backup first then Primary

© Crown copyright Met Office

GTS Link with NOAA-TOC Washington

Inbound flow may be inhibited due to the saturated Outbound flow being unable to return IP sockets ‘ack’ packets quickly enough.


Additional SlidesAdditional Slides


NWSTG Functional OverviewNWSTG Functional Overview

The NWSTG is:

the central communication facility of the NWS;

the primary acquisition and distribution center for NWS data and products;

the primary acquisition and distribution center for international data and products to meet WMO, ICAO and bi-laterally agreed US requirements;

A major data exchange hub for NOAA and other agency data and products.

The NWSTG is:

the central communication facility of the NWS;

the primary acquisition and distribution center for NWS data and products;

the primary acquisition and distribution center for international data and products to meet WMO, ICAO and bi-laterally agreed US requirements;

A major data exchange hub for NOAA and other agency data and products.


NWSTG Functional OverviewNWSTG Functional Overview

The NWSTG includes

WMO Regional Telecommunication Hub (RTH) Washington

ICAO OPMET Databank

the ASOS Operations and Monitoring Center

• Operational oversight of U.S. federal automated surface observing systems

the AWIPS Network Control Facility

• AWIPS is the main NWS’ system which supports the NWS forecast and warning mission requirement

• The NCF is the central communications hub and technical support center for AWIPS

The NWSTG includes

WMO Regional Telecommunication Hub (RTH) Washington

ICAO OPMET Databank

the ASOS Operations and Monitoring Center

• Operational oversight of U.S. federal automated surface observing systems

the AWIPS Network Control Facility

• AWIPS is the main NWS’ system which supports the NWS forecast and warning mission requirement

• The NCF is the central communications hub and technical support center for AWIPS


Dissemination Systems Dissemination Systems

GTS

ISCS

EMWIN

Internet

GMDSS

NWS Telecom. Gateway

GTS

ISCS

EMWIN

Internet

GMDSS

NWS Telecom. Gateway

NWWS

NWR

LDAD

Family Of Services (FOS)

Interagency Connections

AWIPS SBN / NOAAPORT

NWWS

NWR

LDAD

Family Of Services (FOS)

Interagency Connections

AWIPS SBN / NOAAPORT

International National


Dissemination and Distribution

NWSTG

FOS

SBN/NOAAPORT• GOES •NCEP Product Suite

NWWS(10 sec)

WAN

NESDIS

Internet-BasedDissemination

EMWIN(> 1 min)

Telecommunications Operations Center

Satellite Legend:EMWIN

NOAAPORT

NWWS

Commercial Satellite

GOES

NWRWSR-88D

Field Offices

NCEP

DataServers

NCF

Media &Other

Customers

Public

SpecializedCustomers

Other

Agencies

LDAD

Designated Local

Customers

•Commercial Weather Services•Research Institutions•FAA, etc.

ISCS

GTS EMWIN

Private line Dissemination

• Observations

Imagery

WMO

ICAO



Geographically separatedbackup system

Replacement / BackupTG Description

Replacement / BackupTG Description

Worldwide

users• ICAO / WMO• Govt. agencies• Family of

services• Internet users• Foreign

countries• Emergency

mgrs

NWS users• Nat’l Centers for Environ. Prediction• NWS Regional Offices• Domestic/int’l observation & forecast offices• AWIPS

NWS Telecom Gateway• Located in Silver Spring• New message switchingsystem allows future upgrades• 2x upgradeability• Much improved response time • Redundancy ensuresuninterrupted service• Full configuration management

120GB/46600 GB

80GB/38200 GB

90GB/46375 GB

700GB/382000 GB

Daily Throughput/# of circuitsLegacy over Replacement




Backup NWSTGBackup NWSTG


Government Networking Requirements - NOAANetGovernment Networking Requirements - NOAANet

• IP-based networking solution

− Any-to-any connectivity

− High degree of bandwidth scalability

− Optimum redundancy and survivability

− IP convergence (i.e., voice, video and data over IP)

− High-end performance

• Network security remains paramount, particularly in light of today’s socio-political threats

• Segmentation from the public Internet

• Minimizes risk of security or privacy breaches


Sprint Peerless IP Network MapSprint Peerless IP Network Map

Internet Transport Node

OC48 (significant portions upgrading to OC192)

ChicagoChicago

AnaheimAnaheim

San JoseSan Jose Pennsauken, NJPennsauken, NJ

Kansas CityKansas City

StocktonStockton CheyenneCheyenne

FortWorthFort

Worth

New YorkNew York

TacomaTacoma

AtlantaAtlanta

Relay, MDRelay, MD

DCDC

Documents

NOAA’s NWS Telecommunication Gateway RTH Washington ICM MTN ET-OI, Sep 2008 Office of the Chief Information Officer NOAA’s National Weather Service Office