Upload
ralf-walton
View
222
Download
2
Tags:
Embed Size (px)
Citation preview
1NODC, Russia
SeaDataNet TTG meeting
Paris, 18-19 May
Overview and potential use of E2EDM technology for
SeaDataNet
Sergey Belov, Nick Mikhailov
2
Scope
NODC, Russia
E2EDM provides following functionality:
• Wrapping local data systems• Mapping local codes and dictionaries• Data discovery, delivery and processing (export and data
production from delivered data)• Integrated and secured online access to the distributed data
E2EDM is integrating operational and delay-mode time scale data (oceanographic and marine meteorological data) from multiple data formats and multiple data providers in different geographic regions.
3NODC, Russia
Basis
NODC, Russia
• Concept model • Namespace • Metadata/data objects specifications• Data exchange protocol• Software components
4NODC, Russia
E2EDM concept model
NODC, Russia
Providing formalization, identification, description and interpretation for main model objects:
• Data source• Resource• User (in particular cases – external application)Model is based on the following structured semantic objects:• Element• Class - set of elements• Record - set of classesThese semantic objects defines:• Granularity and description for original and transferred data• Metadata description• Codes and dictionaries unification• Data and user policy
5NODC, RussiaNODC, Russia
E2EDM namespace
NODC, Russia
Defines metadata and data structures in terms of XML scheme. Within E2EDM consists of two sub-namespaces: global and conceptual (thematic).
• Global namespace – structures for the model objects• Conceptual (thematic domain) namespace – list of thematic
elements dealing with grouping, e.g. some discipline and available for discovery and exchange
Both designed in XML Schema.1. Global XML Schema (ISO 19115 + WMO Core Metadata +
CDI)2. Concept XML Schema (DiGIR)XML Schemas are also used for validation and development
issues (XML Schema -> UML -> software design)
6NODC, RussiaNODC, Russia
E2EDM metadata objects specifications
NODC, Russia
More than 100 elements, 27 classes, 7 metadata records and 4 data records
Metadata
Information about local data source (linkage)
Unique identification of metadata object
Information about data set
Information about references and responsible party
Information about temporal, vertical extent
Information about data policy
Information about data quality
E2EObjectID
E2ELinkage E2EDataQuality
E2EDistributionInfo
E2EVerticalExtent E2ETemporalExtent
E2EObjectContact
E2EObjectReference
E2ERestrictions List of classes, definingInstruments and processing tools, projects, platforms, geographic coverage, data structure and granularity
7NODC, RussiaNODC, Russia
E2EDM data objects specifications
NODC, Russia
E2EDM data records, based on NetCDF:• E2EDataPoint – for point data• E2EDataProfile – for profile data• E2EDataGrid – for grids• E2EDataObject – object files (http-based and local)
Header (DataHeader) ListElement (list of elements)
DataContainer
«SequenceElement»
«StructureElement»
Common transport NetCDF construction
8NODC, RussiaNODC, Russia
E2EDM data objects specifications (II)
NODC, Russia
class/element NetCDF component NetCDF structure
E2EObjectHierarchy + objectHierarchyLevel
DataHeader + DataContainer
Group
objectHierarchyId DataHeader Attribute
objectHierarchyList DataContainer Variable
Classes and data constructions relationship:
<E2EObjectHierarchy><objectHierarchyLevel>1</objectHierarchyLevel><objectHierarchyList>objectSystemId, objectHierarchyId, countryId, organizationId,
dateTime, Latitude, Longitude, platfType, platformId, platformName</objectHierarchyList><objectHierarchyId>platformId</objectHierarchyId>
<instanceId>platformId</instanceId></E2EObjectHierarchy><E2EObjectHierarchy>
<objectHierarchyLevel>2</objectHierarchyLevel><objectHierarchyList>DEPH, TEMP, PSAL</objectHierarchyList><objectHierarchyId>dateTime, Latitude, Longitude</objectHierarchyId>
<instanceId>dateTime</instanceId></E2EObjectHierarchy>
9NODC, RussiaNODC, RussiaNODC, Russia
E2EDM data exchange protocol
NODC, Russia
Data exchange protocol is responsible for data requesting and data retrieving.
Request message
(XML)
Response message
(XML)
Transport data file
(NetCDF,v.2.2)
10NODC, Russia
E2EDM software components
1. E2EDM Data Provider (DP) – set of services, providing interface between local data systems and distributed system. Must be plugged on the local system. Services are using common library (E2EDM API). DP supports data bases, structured and object data files.
2. E2EDM Integration Server – set of services, providing security issues, resource catalogue maintenance and communication with DP (discovery, requesting and retrieving data)
11NODC, Russia
E2EDM ArchitectureNamespace
Global XML Schema, conceptual XML
Schema USER
Integration Server Management Service
Metadata service
Data access service
Interface service
Integration Server
Transport service
Request message to data
provider
Response message from data provider
Data Provider
Data Provider Management Service
Database Access Service (DiGIR-based)
Local database
Local database
SQL query
Result
Extended transform
API
Configuration metadata
Data format service
File Access Service (OPeNDAP based)
Local data system(files)
Transport data files
Navigation service
Request message to data
provider N
Response message from data provider N
Response from data provider
Integration Server layer
Communication layer
Data Provider layer
Local data system layer
External layer
12NODC, Russia
Integration Server. Features
• Metadata update & maintenance (automatic)• Resource catalogue management• Discovery services for data, metadata and common codes• Request/response communication service• Data sources monitoring• Downloading services (for obtaining data files, metadata, resource and
instance descriptions synchronization)• Export tools (e.g. export in XML from E2EDM resource description into
WMO Core Profile, to ASCII)• HTTP GET/POST (REST) and Web service interfaces• Security service• Schedule mechanisms• Administration tools (CRM, CMS)
13NODC, Russia
Data Provider. Features
• Data source/resources registry and metadata update• Codes and dictionaries mapping• Request handling• Access to the local data source (DBMS, files)• Data extraction based on search criteria• Transport data file generation• HTTP GET/POST (REST) and Web service interfaces• Security service• Schedule mechanisms• Administration tools (online maintenance for resource and instances)
14NODC, Russia
General data exchange scheme
User
Integration Server
Interface layer (HTTP, Web-services)request
Data ProviderData Provider
Data Provider
Request message
reply
Transport data file
Transport data file
Response message
15NODC, Russia
Current operational status of E2EDM
• E2EDM was originated as one of the JCOMM/ETDMP pilot projects in 2003
MetOffice, UK
IFREMER, France
VLIZ, Belgium
NODC, Russia
16NODC, Russia
Pilot project data sources
The following data sources are involved in the E2EDM prototype• Historical marine meteorological data• Historical ocean cruise data• Real-time GTS ocean (BATHY and TESAC) data• Real-time GTS ocean (TESAC/ARGO data)• Real-time GTS (SHIP) data• Monthly climatic fields of ocean parameters (imageries)• Ocean SST satellite data
Geographic area – North Atlantic (test-area)
17NODC, Russia
Pilot project data sources
The following data sources are involved in the E2EDM prototype• Historical marine meteorological data• Historical ocean cruise data• Real-time GTS ocean (BATHY and TESAC) data• Real-time GTS ocean (TESAC/ARGO data)• Real-time GTS (SHIP) data• Monthly climatic fields of ocean parameters (imageries)• Ocean SST satellite data
Geographic area – North Atlantic (test-area)
18NODC, Russia
User data access interface – via Integration Server
Resource catalogue
Parameters
Additional search options
19NODC, Russia
User Data Access interface (cont.)
Search status
20NODC, Russia
• E2EDM Web site http://data.meteo.ru/e2edm/
• E2EDM Integration Server - IODE Project Office (VLIZ)
http://e2edm.vliz.be - RIHMI-WDC (mirror) http://data.meteo.ru:8080/iserv/
User data access interface (cont.)
Point data grid dataprofile data
21NODC, Russia
E2EDM using for SeaDataNet
E2EDM Integration Server
E2EDM Data Provider
Data providers community
HTTP communication
Local Database system Local Data file systemSDN Data Centres
Network
Web-portal
NetCDF transport data-file
Object transport data-file
Responce message
Request message
User access
Registry DiscoveryDelivery
ConnectionEncoding Access
Metadata@Data
22NODC, Russia
E2EDM adaptation for SeaDataNet
JRA2 - Technical development of the interoperable system:E2EDM covers many of JRA2 tasks, but all actions & decisions
should be discussed. E2EDM model deepness is enough for many of the SeaDataNet positions. ( + - there is solution in E2EDM)
• Discovery servicesMetadata update & maintenance +Potential decision: CDI adoption.• Security servicesData access level +Transferred data size level +• Delivery servicesDownloading services + • Viewing servicesDeveloped user interface, libraries, external tools (based on
NetCDF) +
23NODC, Russia
Thank you!
Questions?