Integrated Ocean Observing System Data Management and Communications Sept 2003 The US Integrated Ocean Observing System (IOOS) Plan for Data Management

Integrated Ocean Observing SystemIntegrated Ocean Observing SystemData Management and CommunicationsData Management and CommunicationsSept 2003Sept 2003

The US Integrated The US Integrated Ocean Observing Ocean Observing System (IOOS)System (IOOS)

Plan forPlan forData Management and Data Management and

Communications Communications (DMAC)(DMAC)

IOOS DMAC Steering IOOS DMAC Steering CommitteeCommittee

June 2003June 2003

IOOS Data Management IOOS Data Management andand

Communications Communications SubsystemSubsystem

UsersUsers

Archive Archive CentersCenters

ModelingModeling

On-line On-line BrowseBrowse

ProductsProducts

ForecastsForecastsMapsMaps

RegionalRegionalData Data ManagementManagementSystemsSystemsInternationalInternationalData Data ManagementManagementSystemsSystems

Terrestrial and Terrestrial and AtmosphericAtmosphericData ManagementData ManagementSystemsSystemsMetadata, Data DiscoveryMetadata, Data Discovery

and Data Transportand Data TransportStandards and ProtocolsStandards and Protocols

SatellitesSatellitesShipsShips

Primary DataPrimary DataAssembly & Assembly &

QCQC

Hand Hand MeasurementsMeasurements

MooringsMoorings

FloatsFloats

June 2003June 2003

IOOS Data Management IOOS Data Management andand

Communications Communications SubsystemSubsystem

UsersUsers

Archive Archive CentersCenters

ModelingModeling

On-line On-line BrowseBrowse

ProductsProducts

ForecastsForecastsMapsMaps

RegionalRegionalData Data ManagementManagementSystemsSystemsInternationalInternationalData Data ManagementManagementSystemsSystems

Terrestrial and Terrestrial and AtmosphericAtmosphericData ManagementData ManagementSystemsSystemsMetadata, Data DiscoveryMetadata, Data Discovery

and Data Transportand Data TransportStandards and ProtocolsStandards and Protocols

SatellitesSatellitesShipsShips

Primary DataPrimary DataAssembly & Assembly &

QCQC

Hand Hand MeasurementsMeasurements

MooringsMoorings

FloatsFloats

Sept 2003Sept 2003Integrated Ocean Observing SystemIntegrated Ocean Observing System

Data Management and CommunicationsData Management and Communications

June Feb

DMAC Steering Committee Convened

Expert team

papers

1st Draft Plan

Publish DMAC Plan

DMAC Timeline – Creating the Plan

Mar

Airlie workshop

Sep Dec

Requirements, schedules, &

costs

May

“Expert”

review

Sept

2002 2003

--Metadata--Transport--Archive--Products

June

publicreview

mtg



What makes this a difficult problem?What makes this a difficult problem?

1. Good

2. Fast 3. Cheap

Like the classic “trilemma” …

We can’t have all three



1.Heterogeneous

data

2. Looselyfederated

organizations

3. Large,distributedholdings

…competingcharacteristics rule out the most straightforward solutions



The DMAC Plan is a framework for The DMAC Plan is a framework for interoperability among interoperability among

independent, heterogeneous independent, heterogeneous systems, large and smallsystems, large and small

• Participating organizations will Participating organizations will manage their data independentlymanage their data independently



The methods utilized by data suppliers to The methods utilized by data suppliers to deliver data to their “customers” need not deliver data to their “customers” need not changechange(FTP, subscription services, etc.)(FTP, subscription services, etc.)

Guiding Principles



However, a data provider may However, a data provider may choosechoose to to utilize DMAC middleware as a cost effective utilize DMAC middleware as a cost effective way to reach a wide range of users.way to reach a wide range of users.

Guiding Principles



DMAC requires participation fromDMAC requires participation from– All levels of managementAll levels of management– ScientistsScientists– Technical staffTechnical staff

Why doesn’t DMAC exist Why doesn’t DMAC exist today? today?

The greatest challenges are in the area of community outreach and

organizational behavior rather than in technology!



The DMAC PlanThe DMAC Plan• An evolutionary pathAn evolutionary path

– A software engineering strategyA software engineering strategy– An inter-agency/international outlookAn inter-agency/international outlook

• Three partsThree parts– Part I – overview for general readersPart I – overview for general readers– Part II – formal requirements, Part II – formal requirements,

milestones, tasks for technical plannersmilestones, tasks for technical planners– Part III – in-depth appendices on special Part III – in-depth appendices on special

topics topics



The DMAC PlanThe DMAC Plan

• Technology componentsTechnology components– Data Discovery (metadata)Data Discovery (metadata)– Data TransportData Transport– On-line BrowseOn-line Browse– ArchiveArchive– Metrics, feedback and fault correctionMetrics, feedback and fault correction



The DMAC PlanThe DMAC Plan• Data Discovery (metadata)Data Discovery (metadata)

– Utilize FGDC (& future ISO)Utilize FGDC (& future ISO)– A partnership of current and new metadata A partnership of current and new metadata

repositories repositories – Metadata WG to design the strategy for Metadata WG to design the strategy for

distributed searchdistributed search– A Web Portal at which users can searchA Web Portal at which users can search

• Embed in other Web pagesEmbed in other Web pages• Machine-to-machine accessibleMachine-to-machine accessible

– Seamless segue to DMAC On-Line Browse and Seamless segue to DMAC On-Line Browse and Data TransportData Transport



The DMAC PlanThe DMAC Plan• Data Transport (access)Data Transport (access)

– ““Web Services” Web Services” standardize the standardize the interfaceinterface to the datato the data

– Adopt current successesAdopt current successes•OPeNDAP (a.k.a. “DODS” or “NVODS”)OPeNDAP (a.k.a. “DODS” or “NVODS”)•Discipline neutralDiscipline neutral•Potentially interoperable with GIS componentsPotentially interoperable with GIS components•Adapts to current and future computer Adapts to current and future computer

applicationsapplications– WG must define comprehensive semantic WG must define comprehensive semantic

data model(s)data model(s)



The DMAC PlanThe DMAC Plan• On-line BrowseOn-line Browse

– Uniform access to all IOOS data from Uniform access to all IOOS data from Web browsersWeb browsers•maps (“GIS”) & basic scientific graphicsmaps (“GIS”) & basic scientific graphics•readable tablesreadable tables

– Target audience is IOOS scientific Target audience is IOOS scientific managersmanagers(also useful to many others)(also useful to many others)

– Utilize Data TransportUtilize Data Transport– Live Access Server (LAS) is an initial Live Access Server (LAS) is an initial

“pre-operational” component“pre-operational” component



The DMAC PlanThe DMAC Plan•ArchiveArchive

– Build upon partnership of National and Build upon partnership of National and Regional archive centersRegional archive centers

– Archive WG to be formedArchive WG to be formed•establish DMAC archive policiesestablish DMAC archive policies•ensure that all data have an archive planensure that all data have an archive plan

– Utilize DMAC Metadata and Transport Utilize DMAC Metadata and Transport standardsstandards

– (new)(new) Operational data plus QC to be archived Operational data plus QC to be archived (in coordination with data assembly centers such as (in coordination with data assembly centers such as US GODAE server and NOAA/NDBC)US GODAE server and NOAA/NDBC)



The DMAC PlanThe DMAC Plan• Data productsData products(*)(*)

(IOOS Modeling and Analysis (IOOS Modeling and Analysis Subsystem)Subsystem)

•State estimationState estimation•ForecastsForecasts

• Information productsInformation products(*)(*)

•Value-added products for user groups provided Value-added products for user groups provided by the private sector, academe, governmentby the private sector, academe, government

* * Not part of the DMAC PlanNot part of the DMAC Plan



The DMAC PlanThe DMAC Plan

• Feedback and Fault CorrectionFeedback and Fault Correction– DMAC Web portal will invite user DMAC Web portal will invite user

feedbackfeedback– DMAC to maintain a “Help desk”DMAC to maintain a “Help desk”– Performance metrics gathered Performance metrics gathered

automaticallyautomatically– Oversight by the DMAC Steering Oversight by the DMAC Steering

CommitteeCommittee•Continual, active outreach to usersContinual, active outreach to users



Implementation of IOOS has Implementation of IOOS has already begunalready begun

• New funding possible by FY06New funding possible by FY06• Key pieces of DMAC must be in place by that timeKey pieces of DMAC must be in place by that time

– Interim metadata standards (incl. semantic data Interim metadata standards (incl. semantic data model)model)

– Distributed data discovery architectureDistributed data discovery architecture– Voids to be filled in the data transport toolkitVoids to be filled in the data transport toolkit– Initial responsibilities clear among Archive centersInitial responsibilities clear among Archive centers

• Must fund these developments using current Must fund these developments using current mechanisms!mechanisms!

(See details in (See details in Part I, Section 4Part I, Section 4 of DMAC Plan) of DMAC Plan)

http://dmac_part_I_8.29.03.pdf/



DMAC Plan

published

DMAC Phased Implementation

Sept2003

‘04 ‘05 ‘06 ‘07

Immediate Priorities• Metadata Standards (W.G.)• GIS&OBIS Data Transport integration• Archive strategy (W.G.)• Software engineering oversight

‘08 ‘09 ‘10

$81M$2.3M

Five year implementation Plan• Standards and Tool Development ($30M)• Hardware & Software for networking, data assemb. & archive ($37M)• Pilot projects ($14M)

minimally functioning

initial systemIOC



NVODS and DMACNVODS and DMAC• Use of OPeNDAP (NVODS) was not a Use of OPeNDAP (NVODS) was not a

guaranteed outcome of the DMAC guaranteed outcome of the DMAC planning processplanning process– It was selected for DMAC Data Transport It was selected for DMAC Data Transport

because the NVODS project anticipated because the NVODS project anticipated the DMAC analysis of the IOOS “the DMAC analysis of the IOOS “trilemmatrilemma” – the need for a “framework ” – the need for a “framework for interoperability”.for interoperability”.



NVODS and DMACNVODS and DMAC

OPeNDAP must be made ready for the OPeNDAP must be made ready for the FY06 rapid expansion of IOOS.FY06 rapid expansion of IOOS.

• GIS compatibilityGIS compatibility• OBIS (biological) compatibilityOBIS (biological) compatibility• delimited files (ASCII)delimited files (ASCII)• OPeNDAP Development Kit (ODK) – an easily OPeNDAP Development Kit (ODK) – an easily

adaptable server coreadaptable server core



NVODS and DMACNVODS and DMAC• A semantic data modelA semantic data model

– need leadership from NVODSneed leadership from NVODS• A stable, published spec (v4 -- XML)A stable, published spec (v4 -- XML)• Expanded resource control frameworkExpanded resource control framework

– Restrict data volumes, cpu cycles, privs, …Restrict data volumes, cpu cycles, privs, …• Expanded “web service” infrastructureExpanded “web service” infrastructure

– Query resource limitsQuery resource limits




• Improved Sequence data supportImproved Sequence data support“pilot” “pilot” “pre-op” “pre-op”– additional client optionsadditional client options

•perl, python, “translation”, jdbc (?), …perl, python, “translation”, jdbc (?), …– usability conventions (metadata)usability conventions (metadata)

•min/max ranges, size guidelines, …min/max ranges, size guidelines, …– much more hands-on experiencemuch more hands-on experience

• further forms of aggregationfurther forms of aggregation– in-situ collections of profiles and time series, in-situ collections of profiles and time series,

5D and 6D forecast ensembles5D and 6D forecast ensembles




Hard work remains to be done,Hard work remains to be done,but with a sustained effortbut with a sustained effort

NVODS/OPeNDAP can succeedNVODS/OPeNDAP can succeed



Questions?Questions?

Click for immediate actions recommended for

IOOS data providers



Concrete actions to contribute data Concrete actions to contribute data into the IOOS/DMAC framework todayinto the IOOS/DMAC framework today• Data discoveryData discovery

1.1. Produce FGDC metadataProduce FGDC metadata2.2. Submit it to national search sitesSubmit it to national search sites

(NASA/GCMD, NOAA/NCDDC & CSC)(NASA/GCMD, NOAA/NCDDC & CSC)3.3. Create a home page and a search Create a home page and a search

portal to your data holdingsportal to your data holdings

• Interoperable data accessInteroperable data access• On-line browseOn-line browse



Concrete actions …Concrete actions …• Data discoveryData discovery• Interoperable data accessInteroperable data access

1.1. Gridded data: Gridded data: – Install OPeNDAPInstall OPeNDAP(*)(*) servers servers

2.2. Time series:Time series:– Consider partnership with NOAA/NDBCConsider partnership with NOAA/NDBC(**)(**)

3.3. In-situIn-situ data: data:– Use RDBMS or standard formatsUse RDBMS or standard formats– Contribute to development of pilot Contribute to development of pilot

IOOS/DMAC components – “DRDS”, “A.S.”IOOS/DMAC components – “DRDS”, “A.S.”• On-line browseOn-line browse(*) (*) http://www.unidata.ucar.edu/packages/dods/http://www.unidata.ucar.edu/packages/dods/(**) [email protected] , (228)-688-2840(**) [email protected] , (228)-688-2840

http://www.unidata.ucar.edu/packages/dods/



Concrete actions …Concrete actions …• Data discoveryData discovery• Interoperable data accessInteroperable data access• On-line browseOn-line browse

1.1. Install Live Access ServerInstall Live Access Server(*)(*) utilizing utilizing DMAC/OPeNDAP data transportDMAC/OPeNDAP data transport

2.2. Create custom Web sites that Create custom Web sites that demonstrate regional data demonstrate regional data integrationintegration

(*) http://www.ferret.noaa.gov/LAS/(*) http://www.ferret.noaa.gov/LAS/



Questions?Questions?

Documents

Integrated Ocean Observing System Data Management and Communications Sept 2003 The US Integrated Ocean Observing System (IOOS) Plan for Data Management