Upload
joleen-stanley
View
214
Download
1
Embed Size (px)
DESCRIPTION
Sept 2003 Integrated Ocean Observing System Data Management and Communications IOOS: One system….seven goals –Detect and Predict Change –Mitigate natural hazards –Improve safety and efficiency of marine ops –Ensure national security –Reduce public health risks –Protect and restore marine ecosystems –Sustain marine resources
Citation preview
Integrated Ocean Observing SystemIntegrated Ocean Observing SystemData Management and CommunicationsData Management and CommunicationsSept 2003Sept 2003
The US Integrated The US Integrated Ocean Observing Ocean Observing System (IOOS)System (IOOS)
Plan forPlan forData Management and Data Management and
Communications Communications (DMAC)(DMAC)
IOOS DMAC Steering IOOS DMAC Steering CommitteeCommittee
June 2003June 2003
IOOS Data Management IOOS Data Management andand
Communications Communications SubsystemSubsystem
UsersUsers
Archive Archive CentersCenters
ModelingModeling
On-line On-line BrowseBrowse
ProductsProducts
ForecastsForecastsMapsMaps
RegionalRegionalData Data ManagementManagementSystemsSystemsInternationalInternationalData Data ManagementManagementSystemsSystems
Terrestrial and Terrestrial and AtmosphericAtmosphericData ManagementData ManagementSystemsSystemsMetadata, Data DiscoveryMetadata, Data Discovery
and Data Transportand Data TransportStandards and ProtocolsStandards and Protocols
SatellitesSatellitesShipsShips
Primary DataPrimary DataAssembly & Assembly &
QCQC
Hand Hand MeasurementsMeasurements
MooringsMoorings
FloatsFloats
June 2003June 2003
IOOS Data Management IOOS Data Management andand
Communications Communications SubsystemSubsystem
UsersUsers
Archive Archive CentersCenters
ModelingModeling
On-line On-line BrowseBrowse
ProductsProducts
ForecastsForecastsMapsMaps
RegionalRegionalData Data ManagementManagementSystemsSystemsInternationalInternationalData Data ManagementManagementSystemsSystems
Terrestrial and Terrestrial and AtmosphericAtmosphericData ManagementData ManagementSystemsSystemsMetadata, Data DiscoveryMetadata, Data Discovery
and Data Transportand Data TransportStandards and ProtocolsStandards and Protocols
SatellitesSatellitesShipsShips
Primary DataPrimary DataAssembly & Assembly &
QCQC
Hand Hand MeasurementsMeasurements
MooringsMoorings
FloatsFloats
Sept 2003Sept 2003Integrated Ocean Observing SystemIntegrated Ocean Observing System
Data Management and CommunicationsData Management and Communications
June Feb
DMAC Steering Committee Convened
Expert team
papers
1st Draft Plan
Publish DMAC Plan
DMAC Timeline – Creating the Plan
Mar
Airlie workshop
Sep Dec
Requirements, schedules, &
costs
May
“Expert”
review
Sept
2002 2003
--Metadata--Transport--Archive--Products
June
publicreview
mtg
Sept 2003Sept 2003Integrated Ocean Observing SystemIntegrated Ocean Observing System
Data Management and CommunicationsData Management and Communications
What makes this a difficult problem?What makes this a difficult problem?
1. Good
2. Fast 3. Cheap
Like the classic “trilemma” …
We can’t have all three
Sept 2003Sept 2003Integrated Ocean Observing SystemIntegrated Ocean Observing System
Data Management and CommunicationsData Management and Communications
1.Heterogeneous
data
2. Looselyfederated
organizations
3. Large,distributedholdings
…competingcharacteristics rule out the most straightforward solutions
Sept 2003Sept 2003Integrated Ocean Observing SystemIntegrated Ocean Observing System
Data Management and CommunicationsData Management and Communications
The DMAC Plan is a framework for The DMAC Plan is a framework for interoperability among interoperability among
independent, heterogeneous independent, heterogeneous systems, large and smallsystems, large and small
• Participating organizations will Participating organizations will manage their data independentlymanage their data independently
Sept 2003Sept 2003Integrated Ocean Observing SystemIntegrated Ocean Observing System
Data Management and CommunicationsData Management and Communications
The methods utilized by data suppliers to The methods utilized by data suppliers to deliver data to their “customers” need not deliver data to their “customers” need not changechange(FTP, subscription services, etc.)(FTP, subscription services, etc.)
Guiding Principles
Sept 2003Sept 2003Integrated Ocean Observing SystemIntegrated Ocean Observing System
Data Management and CommunicationsData Management and Communications
However, a data provider may However, a data provider may choosechoose to to utilize DMAC middleware as a cost effective utilize DMAC middleware as a cost effective way to reach a wide range of users.way to reach a wide range of users.
Guiding Principles
Sept 2003Sept 2003Integrated Ocean Observing SystemIntegrated Ocean Observing System
Data Management and CommunicationsData Management and Communications
DMAC requires participation fromDMAC requires participation from– All levels of managementAll levels of management– ScientistsScientists– Technical staffTechnical staff
Why doesn’t DMAC exist Why doesn’t DMAC exist today? today?
The greatest challenges are in the area of community outreach and
organizational behavior rather than in technology!
Sept 2003Sept 2003Integrated Ocean Observing SystemIntegrated Ocean Observing System
Data Management and CommunicationsData Management and Communications
The DMAC PlanThe DMAC Plan• An evolutionary pathAn evolutionary path
– A software engineering strategyA software engineering strategy– An inter-agency/international outlookAn inter-agency/international outlook
• Three partsThree parts– Part I – overview for general readersPart I – overview for general readers– Part II – formal requirements, Part II – formal requirements,
milestones, tasks for technical plannersmilestones, tasks for technical planners– Part III – in-depth appendices on special Part III – in-depth appendices on special
topics topics
Sept 2003Sept 2003Integrated Ocean Observing SystemIntegrated Ocean Observing System
Data Management and CommunicationsData Management and Communications
The DMAC PlanThe DMAC Plan
• Technology componentsTechnology components– Data Discovery (metadata)Data Discovery (metadata)– Data TransportData Transport– On-line BrowseOn-line Browse– ArchiveArchive– Metrics, feedback and fault correctionMetrics, feedback and fault correction
Sept 2003Sept 2003Integrated Ocean Observing SystemIntegrated Ocean Observing System
Data Management and CommunicationsData Management and Communications
The DMAC PlanThe DMAC Plan• Data Discovery (metadata)Data Discovery (metadata)
– Utilize FGDC (& future ISO)Utilize FGDC (& future ISO)– A partnership of current and new metadata A partnership of current and new metadata
repositories repositories – Metadata WG to design the strategy for Metadata WG to design the strategy for
distributed searchdistributed search– A Web Portal at which users can searchA Web Portal at which users can search
• Embed in other Web pagesEmbed in other Web pages• Machine-to-machine accessibleMachine-to-machine accessible
– Seamless segue to DMAC On-Line Browse and Seamless segue to DMAC On-Line Browse and Data TransportData Transport
Sept 2003Sept 2003Integrated Ocean Observing SystemIntegrated Ocean Observing System
Data Management and CommunicationsData Management and Communications
The DMAC PlanThe DMAC Plan• Data Transport (access)Data Transport (access)
– ““Web Services” Web Services” standardize the standardize the interfaceinterface to the datato the data
– Adopt current successesAdopt current successes•OPeNDAP (a.k.a. “DODS” or “NVODS”)OPeNDAP (a.k.a. “DODS” or “NVODS”)•Discipline neutralDiscipline neutral•Potentially interoperable with GIS componentsPotentially interoperable with GIS components•Adapts to current and future computer Adapts to current and future computer
applicationsapplications– WG must define comprehensive semantic WG must define comprehensive semantic
data model(s)data model(s)
Sept 2003Sept 2003Integrated Ocean Observing SystemIntegrated Ocean Observing System
Data Management and CommunicationsData Management and Communications
The DMAC PlanThe DMAC Plan• On-line BrowseOn-line Browse
– Uniform access to all IOOS data from Uniform access to all IOOS data from Web browsersWeb browsers•maps (“GIS”) & basic scientific graphicsmaps (“GIS”) & basic scientific graphics•readable tablesreadable tables
– Target audience is IOOS scientific Target audience is IOOS scientific managersmanagers(also useful to many others)(also useful to many others)
– Utilize Data TransportUtilize Data Transport– Live Access Server (LAS) is an initial Live Access Server (LAS) is an initial
“pre-operational” component“pre-operational” component
Sept 2003Sept 2003Integrated Ocean Observing SystemIntegrated Ocean Observing System
Data Management and CommunicationsData Management and Communications
The DMAC PlanThe DMAC Plan•ArchiveArchive
– Build upon partnership of National and Build upon partnership of National and Regional archive centersRegional archive centers
– Archive WG to be formedArchive WG to be formed•establish DMAC archive policiesestablish DMAC archive policies•ensure that all data have an archive planensure that all data have an archive plan
– Utilize DMAC Metadata and Transport Utilize DMAC Metadata and Transport standardsstandards
– (new)(new) Operational data plus QC to be archived Operational data plus QC to be archived (in coordination with data assembly centers such as (in coordination with data assembly centers such as US GODAE server and NOAA/NDBC)US GODAE server and NOAA/NDBC)
Sept 2003Sept 2003Integrated Ocean Observing SystemIntegrated Ocean Observing System
Data Management and CommunicationsData Management and Communications
The DMAC PlanThe DMAC Plan• Data productsData products(*)(*)
(IOOS Modeling and Analysis (IOOS Modeling and Analysis Subsystem)Subsystem)
•State estimationState estimation•ForecastsForecasts
• Information productsInformation products(*)(*)
•Value-added products for user groups provided Value-added products for user groups provided by the private sector, academe, governmentby the private sector, academe, government
* * Not part of the DMAC PlanNot part of the DMAC Plan
Sept 2003Sept 2003Integrated Ocean Observing SystemIntegrated Ocean Observing System
Data Management and CommunicationsData Management and Communications
The DMAC PlanThe DMAC Plan
• Feedback and Fault CorrectionFeedback and Fault Correction– DMAC Web portal will invite user DMAC Web portal will invite user
feedbackfeedback– DMAC to maintain a “Help desk”DMAC to maintain a “Help desk”– Performance metrics gathered Performance metrics gathered
automaticallyautomatically– Oversight by the DMAC Steering Oversight by the DMAC Steering
CommitteeCommittee•Continual, active outreach to usersContinual, active outreach to users
Sept 2003Sept 2003Integrated Ocean Observing SystemIntegrated Ocean Observing System
Data Management and CommunicationsData Management and Communications
Implementation of IOOS has Implementation of IOOS has already begunalready begun
• New funding possible by FY06New funding possible by FY06• Key pieces of DMAC must be in place by that timeKey pieces of DMAC must be in place by that time
– Interim metadata standards (incl. semantic data Interim metadata standards (incl. semantic data model)model)
– Distributed data discovery architectureDistributed data discovery architecture– Voids to be filled in the data transport toolkitVoids to be filled in the data transport toolkit– Initial responsibilities clear among Archive centersInitial responsibilities clear among Archive centers
• Must fund these developments using current Must fund these developments using current mechanisms!mechanisms!
(See details in (See details in Part I, Section 4Part I, Section 4 of DMAC Plan) of DMAC Plan)
Sept 2003Sept 2003Integrated Ocean Observing SystemIntegrated Ocean Observing System
Data Management and CommunicationsData Management and Communications
DMAC Plan
published
DMAC Phased Implementation
Sept2003
‘04 ‘05 ‘06 ‘07
Immediate Priorities• Metadata Standards (W.G.)• GIS&OBIS Data Transport integration• Archive strategy (W.G.)• Software engineering oversight
‘08 ‘09 ‘10
$81M$2.3M
Five year implementation Plan• Standards and Tool Development ($30M)• Hardware & Software for networking, data assemb. & archive ($37M)• Pilot projects ($14M)
minimally functioning
initial systemIOC
Sept 2003Sept 2003Integrated Ocean Observing SystemIntegrated Ocean Observing System
Data Management and CommunicationsData Management and Communications
NVODS and DMACNVODS and DMAC• Use of OPeNDAP (NVODS) was not a Use of OPeNDAP (NVODS) was not a
guaranteed outcome of the DMAC guaranteed outcome of the DMAC planning processplanning process– It was selected for DMAC Data Transport It was selected for DMAC Data Transport
because the NVODS project anticipated because the NVODS project anticipated the DMAC analysis of the IOOS “the DMAC analysis of the IOOS “trilemmatrilemma” – the need for a “framework ” – the need for a “framework for interoperability”.for interoperability”.
Sept 2003Sept 2003Integrated Ocean Observing SystemIntegrated Ocean Observing System
Data Management and CommunicationsData Management and Communications
NVODS and DMACNVODS and DMAC
OPeNDAP must be made ready for the OPeNDAP must be made ready for the FY06 rapid expansion of IOOS.FY06 rapid expansion of IOOS.
• GIS compatibilityGIS compatibility• OBIS (biological) compatibilityOBIS (biological) compatibility• delimited files (ASCII)delimited files (ASCII)• OPeNDAP Development Kit (ODK) – an easily OPeNDAP Development Kit (ODK) – an easily
adaptable server coreadaptable server core
Sept 2003Sept 2003Integrated Ocean Observing SystemIntegrated Ocean Observing System
Data Management and CommunicationsData Management and Communications
NVODS and DMACNVODS and DMAC• A semantic data modelA semantic data model
– need leadership from NVODSneed leadership from NVODS• A stable, published spec (v4 -- XML)A stable, published spec (v4 -- XML)• Expanded resource control frameworkExpanded resource control framework
– Restrict data volumes, cpu cycles, privs, …Restrict data volumes, cpu cycles, privs, …• Expanded “web service” infrastructureExpanded “web service” infrastructure
– Query resource limitsQuery resource limits
Sept 2003Sept 2003Integrated Ocean Observing SystemIntegrated Ocean Observing System
Data Management and CommunicationsData Management and Communications
NVODS and DMACNVODS and DMAC
• Improved Sequence data supportImproved Sequence data support“pilot” “pilot” “pre-op” “pre-op”– additional client optionsadditional client options
•perl, python, “translation”, jdbc (?), …perl, python, “translation”, jdbc (?), …– usability conventions (metadata)usability conventions (metadata)
•min/max ranges, size guidelines, …min/max ranges, size guidelines, …– much more hands-on experiencemuch more hands-on experience
• further forms of aggregationfurther forms of aggregation– in-situ collections of profiles and time series, in-situ collections of profiles and time series,
5D and 6D forecast ensembles5D and 6D forecast ensembles
Sept 2003Sept 2003Integrated Ocean Observing SystemIntegrated Ocean Observing System
Data Management and CommunicationsData Management and Communications
NVODS and DMACNVODS and DMAC
Hard work remains to be done,Hard work remains to be done,but with a sustained effortbut with a sustained effort
NVODS/OPeNDAP can succeedNVODS/OPeNDAP can succeed
Sept 2003Sept 2003Integrated Ocean Observing SystemIntegrated Ocean Observing System
Data Management and CommunicationsData Management and Communications
Questions?Questions?
Click for immediate actions recommended for
IOOS data providers
Sept 2003Sept 2003Integrated Ocean Observing SystemIntegrated Ocean Observing System
Data Management and CommunicationsData Management and Communications
Concrete actions to contribute data Concrete actions to contribute data into the IOOS/DMAC framework todayinto the IOOS/DMAC framework today• Data discoveryData discovery
1.1. Produce FGDC metadataProduce FGDC metadata2.2. Submit it to national search sitesSubmit it to national search sites
(NASA/GCMD, NOAA/NCDDC & CSC)(NASA/GCMD, NOAA/NCDDC & CSC)3.3. Create a home page and a search Create a home page and a search
portal to your data holdingsportal to your data holdings
• Interoperable data accessInteroperable data access• On-line browseOn-line browse
Sept 2003Sept 2003Integrated Ocean Observing SystemIntegrated Ocean Observing System
Data Management and CommunicationsData Management and Communications
Concrete actions …Concrete actions …• Data discoveryData discovery• Interoperable data accessInteroperable data access
1.1. Gridded data: Gridded data: – Install OPeNDAPInstall OPeNDAP(*)(*) servers servers
2.2. Time series:Time series:– Consider partnership with NOAA/NDBCConsider partnership with NOAA/NDBC(**)(**)
3.3. In-situIn-situ data: data:– Use RDBMS or standard formatsUse RDBMS or standard formats– Contribute to development of pilot Contribute to development of pilot
IOOS/DMAC components – “DRDS”, “A.S.”IOOS/DMAC components – “DRDS”, “A.S.”• On-line browseOn-line browse(*) (*) http://www.unidata.ucar.edu/packages/dods/http://www.unidata.ucar.edu/packages/dods/(**) [email protected] , (228)-688-2840(**) [email protected] , (228)-688-2840
Sept 2003Sept 2003Integrated Ocean Observing SystemIntegrated Ocean Observing System
Data Management and CommunicationsData Management and Communications
Concrete actions …Concrete actions …• Data discoveryData discovery• Interoperable data accessInteroperable data access• On-line browseOn-line browse
1.1. Install Live Access ServerInstall Live Access Server(*)(*) utilizing utilizing DMAC/OPeNDAP data transportDMAC/OPeNDAP data transport
2.2. Create custom Web sites that Create custom Web sites that demonstrate regional data demonstrate regional data integrationintegration
(*) http://www.ferret.noaa.gov/LAS/(*) http://www.ferret.noaa.gov/LAS/
Sept 2003Sept 2003Integrated Ocean Observing SystemIntegrated Ocean Observing System
Data Management and CommunicationsData Management and Communications
Questions?Questions?