EMODnet Chemistry 2

Various actions

ByDick M.A. Schaap – Technical Coordinator

Rome – Italy, 20 – 21 January 2014, Steering Committee

Indicators for reporting to Secretariate

1) Volume of data made available through the portal: All gathered data are included in the CDI Data Discovery and Access service => the number of CDIs and the increase over time gives an excellent indicator to follow progress.

2) Organisations supplying each type of data: => related to CDIs we can provide the numbers divided over data centres and their countries. And thus the increase over time

3) Organisations that have been approached to supply data with no result: This has to be treated with care, because part of the remit of the projects is to identify potential additional data providers and data sets and then to encourage and/or negotiate them to make these part of the data discovery and access services in EMODNet. Also this process takes time. => we propose to report this as a management report and on an annual basis.

Indicators for reporting to Secretariate

4) Volume of each data type downloaded from each portal: Users undertake shopping for data for which they need to be registered and login. All requests and actions for downloading of data sets are registered in the SeaDataNet Request Status Manager (RSM) which allows for reporting. The RSM registers which CDIs, which users, which connected data centres and at what time. It also reports these per portal used (Chemistry, Bathymetry, Physics, ..) However analysing these in qualitative respect is beyond our scope of work and can be quite laborsome. => we propose a more concise report from the RSM: how many CDIs were requested by how many users from different data centres over a given time period from a given portal. This will give already a good indication of data downloading. In stead of reporting these on a 2-monthly basis we rather do this on a 6-monthly basis.

In addition there is browsing and downloading of data products (interpolated sea region maps for specific chemical substances)=> it is possible to report a table of downloaded data products over a given time period. In stead of reporting these on a 2-monthly basis we rather do this on a 6-monthly basis.

5) Organisations that have downloaded each data type: => For CDI from RSM we can provide per portal the names of user organisations that have requested and downloaded so many data sets. Qualitative analysis of what kind of data is a lot of effort and out of scope. The report is more quantitative. Like 4) this is proposed on a 6-monthly basis.  However we must be careful with privacy of individual users!! => For data products no tracking takes place of users, because viewing and downloading of data products is public, free and anonymus.

Indicators for reporting to Secretariate

6) Using user statistics to determine the main pages utilised and to identify data products being used: We make use of web statistics, e.g. AW Stats, to get info on the visits to the web portals and their different sections and services. Giving a summary table from these statistics services is easy, but analysing the statistics more in depth is complex and out of the scope of the planned work for the portals. => we can ararnge that the Secretariate gets online access to these statistics reporting systems so that it can make its own analyses.

7) List of what the downloaded data has been used for: The portals do not have insight in the purpose of downloading data sets by users. => it is proposed that the Secretariate will make use of the reported users (see 5) to do some following up towards specific users, asking more about their purpose, satisfaction, etc..

8) List of organisations that have downloaded data from more than one portal: This requires concertation between portals. => It is proposed that the Secretariate will make use of the reported users (see 5) to do this kind of cross analysis.

9) Interoperability of data of different types and from different portals: => It is proposed to discuss this in the SC and as part of the experiences of the Use Cases and the Checkpoint Studies.

Indicators for reporting to Secretariate

10) Monitoring levell of interaction with member states national processes for data stewardship: As discussed at the SC meeting this will not be easy. => Where it is possible and appropriate it is proposed to report this on an annual basis as part of the Annual Reports.

11) Management Budget Overview: As discussed and agreed at the SC this will NOT be supplied by the projects because the work is undertaken on a lump sum basis.  DG MARE will consider its own input and the time / money spent on the Secretariate as Overhead, rest as Research and Development.

12) EMODNet Citations: => It is proposed that the project reports on citations (articles in publications, presentations at conferences etc) on an annual basis as part of the Annual Reports.

EMODnet Chemistry portal – present set-up

SeaDataNet II is making good progress with automatic harvesting using a robot user in the CDI system in order to maintain internal data buffers for specific purposes

Robot harvester is operational and has been applied for automatically harvesting ca 1.000.000 T&S data sets as part of the joint data climatology product development SeaDataNet – MyOcean

Robot harvester can be configured for specific data buffer profiles and in agreement (SLA’s) with data providers AND specific data user communities (such as MyOcean, SeaDataNet regional dataproduct groups, EMODNet Chemistry regional product groups, WG DIKE); work is ongoing for extending the functionality of the RSM service for this purpose

Working with a central buffer to support products generation and additional visualisation

the central buffer system with set of database views will be maintained automatically by the RSM - robot user system, following new and updated CDI submissions

the access to these buffers of aggregate data sets can also be virtual for specific Workflow Processes (WPS) in which case only results and graphics of analyses are made available, not the individual data

further work ongoing for adding a controlled access to the buffer data sets for specific purposes, using the RSM administration, exploring options for hosting the buffer system in the cloudplus developing additional subsetting and visualisation services

See also SeaDataNet II Deliverables D5.6 and D8.7

Working with a central buffer to support products generation and additional visualisation

Planned architecture extension

CDI User Interfaces

CDI Robot harvester

Specific data buffers

Advanced access and

viewing services

OceanotronOpenEarth

DIVAODV

Agreed Settings

Dynamic Maintenance

the central buffer system will contain ‘raw’ data sets for specific parameters and as harvested from the distributed data centres=> further action needed for making the collection more homogeneous and validated => aggregated data sets

this can be done by using ODV software and specific expertise per region and chemical substance

use will also be made of the new P35 Vocabulary for aggregating P01 terms. The P35 pilot is already up for one P35 term:http://seadatanet.maris2.nl/v_bodc_vocab_v2/welcome.asp=> further populating of the P35 Vocabulary list requires contributions from domain experts who can judge which P01 terms scientifically can be grouped under specific P35 terms. BODC will ingest entries following experts proposal.

Converting buffer data sets to validated aggregated data sets

EMODnet Chemistry extension

Specific data buffers

Oceanotron &OpenEarth services

ODV QC + aggregation

QCd data buffers

Regional experts

DIVA + OceanBrowser service

CDI RobotHarvester

Next to already existing ODV -> Diva -> OceanBrowser service for interpolated maps

Use the robot harvester to build a central buffer of data sets with regular updating intervals

Extract relevant 'raw' data files for EMODNet Chemistry from the central buffer

QA/QC and aggregate these into a homogeneous ODV collection which sits in ODV

Evaluate the advances of Oceantron (within SeaDataNet) for having access to the ODV Collection, using the ODV API and appropriate Oceanotron plug-in, and making them available for OGC SWE and WMS services

WPS processes will take the OGC compliant output (possibly from Oceantron) as their input and generate on the fly added value graphics such as station time series, concentration plots for a given time and space window, etc

These new WPS products will be described in the SEXTANT Catalogue service

The new WPS products will be served to users via the OceanBrowser viewing service

EMODnet Chemistry planned output services

Sufficient storagePref. accessable as an external harddisk; otherwise minimally by FTPMachine to machine interfacing, such as automatic login etc.Reading of statistics

  Security Using SeaDataNet Caslogin must be possibleIp-number security desirable for setting up IP to IP connectionsPossible support for OpenID access

 How can we configure and control rights of the cloud towards accessing MARIS systemsHow safe is the system on which the cloud operates

 Possibility by MARIS to develop, include and run own software, such as SOAP (machine to machine)Support for platforms (windows en unix); for MARIS windows is 1st requirementPossible to develop and run Webservices (SOAP) (windows, php, ..) Possible to make and operate service for preparing and downloading of zipfiles.Possible to make and operate internet interface.Making use of storage and of database, eg mysql

Draft specifications for cloud hosting (prelimininary)

EMODNet Chemistry 2 scope is enlarged compared to previous Chemistry 1

MARIS uses the P02 scope in the CDI Data Discovery and Access service for subsetting the EMODNet Chemistry view from the overall SeaDataNet CDI database

The P02 – P01 tables are used for the data providers to support their data gathering and mapping for CDI and ODV preparation

MARIS has asked BODC, AWI and ICES for advise

BODC has prepared a first draft; MARIS has submitted the previous list

Needs finalisation (and possible extension of Vocabs P02 – P01 ?)

Filter P02 – P01 for EMODNet Chemistry scope

CDI Data Discovery and Access Service – Sea regions

C19

CDI Data Discovery and Access Service – Sea regions

C19 – polygons for marking up all CDI entries with C19 terms using spatial queries + extra tags via C19 BT relations

CDI Data Discovery and Access Service – Sea regions

Example – Search by North Sea in CDI User Interface

MARIS can help regional groups for gathering and collating by Robot all relevant ODV data sets and associated CDIs per region => provision of raw data sets by DVD; later in the project by downloading from the cloud OR WPS

Action TWG-10: coastal issue: developing an extra spatial query functionality in the CDI User Interface for searching based on the distance along the coastline. Also an external WMS layer with geographical features (e.g. towns, borders, river outlets) by means of OpenStreetMap will be added by MARIS (March 2014)

Action TWG-11: coastal issue: developing the visualization along (or near) the coastline of a Chemistry data product in OceanBrowser. One axis of such visualizations will correspond to the distance along the coastline from a given starting point. Currently vertical sections can be extracted along a user-specified path. Data extraction along the coastline will be simplified for providing a list of predetermined sections following the coastline. The display of the along-coast figures will be optimized to take into account the specificities of these. By ULg (March 2014)

Implication of data central buffer for product generation and new coastal visualization tools as discussed at the TWG

Plus extra products to be produced by OpenEarth via WPS into OceanBrowser:

visualise a time evolution of a selected group of measurementsshow concentration plots for a given time and space window and also along the coastshow inflows from rivers of nutrients. The user should be able to select a section of coast, a country or a region (NUTS3) and obtain time series of inflows of parameters expressed as mass or moles per unit time per river (or section of coast)

Implication of data central buffer for product generation and new coastal visualization tools as discussed at the TWG

CDI format and service upgraded to ISO 19139 and ISO compliance

CDI User Interface extended with search for P08->P03->P02Search for Sea Regions (C19)Search for time seriesCDI Details extended with EDMED, CSR, Display of time series in Results

Possible extra:

Geocoding and search for MSFD regionsCoastal searchingOpen Street View as one of the basis map layers

Links between Central Portal and EMODNet Chemistry:Use of WMS – WFS servicesSEXTANT Catalogue of products => subset to be shared in CentralDedicated use cases => specific customised queries / services?

TWG – activities for EMODNet Chemistry portal

Action TWG-1: a pilot will be set-up for exposing the P01 semantic model based upon JSON (XML) provided as service by BODC and upgraded client interface provided by MARIS including functions for searching and filtering on separate fields and searching across fields (end 2013 - early 2014 )

Action TWG-2: the set-up of the pilot and its progress to date will be presented by BODC and MARIS at the coming 2nd ODIP Workshop (December 2013) for discussion with international experts and also for further brainstorming on possible RDF application

Action TWG-3: the P35 Vocabulary list will be set-up by BODC including possibility to relate to P01 terms and MARIS will include it in the SeaDataNet client interface (asap)

Action TWG-4: populating the P35 Vocabulary list requires contributions from domain experts who can judge which P01 terms scientifically can be grouped under specific P35 terms. BODC will ingest entries following experts proposal.

Action TWG-5: incorporating the P35 Vocabulary list into the ODV aggregation software will be developed by AWI (spring 2014)

Action TWG-6: incorporating the P35 Vocabulary list into the Diva Products Discovery service (via SEXTANT Catalogue) will be analysed by ULG and IFREMER (end 2013)

ACTIONS from TWG-1

Proposed fields to exposeMeasurement (including classifier)Substance (including classifier, alternative name)Measurement Matrix RelationshipMatrixMatrix SubcomponentTaxonOrganism NameOrganism SpecificsTechnique

Example:XMETAL04Measurement: “Concentration”Substance: “cadmium {Cd}”Measurement Matrix Relationship: “per unit wet weight”Matrix: “biota”Matrix Subcomponent: “liver”Taxon: “Platichthys flesus (ITIS: 172894: WoRMS 127141)”Organism Name: “Platichthys flesus”Organism Specifics: “Sex: male Size: length 226-250mm”Technique: “inductively-coupled plasma mass spectrometry”

Semantic model for P01

Action TWG-7: formulating a plan incl proposed format (JSON?) for expanding the new P01 discovery client interface with an online form procedure for requesting and first level validating new P01 codes as combinations of fields by BODC and MARIS for discussion at ODIP 2nd Workshop (end November 2013)

Action TWG-8: implementing the online form procedure for requesting and first level validating new P01 codes as combinations of fields by BODC and MARIS (April 2014)

Action TWG-9: exploring extending the CDI discovery interface with (smart) search on chemical substances by BODC, IFREMER and MARIS (period mid 2014 - mid 2015)

Action TWG-10: coastal issue: developing an extra spatial query functionality in the CDI User Interface for searching based on the distance along the coastline. Also an external WMS layer with geographical features (e.g. towns, borders, river outlets) by means of OpenStreetMap will be added by MARIS (March 2014)

Action TWG-11: coastal issue: developing the visualization along (or near) the coastline of a Chemistry data product in OceanBrowser. One axis of such visualizations will correspond to the distance along the coastline from a given starting point. Currently vertical sections can be extracted along a user-specified path. Data extraction along the coastline will be simplified for providing a list of predetermined sections following the coastline. The display of the along-coast figures will be optimized to take into account the specificities of these. By ULg (March 2014)

ACTIONS from TWG-1

Action TWG-12: communication between IFREMER and ULg about the upgraded SEXTANT metadata format for upgrading the Diva2XML application and how to harvest Diva XML entries (asap)

Action TWG-13: communication between IFREMER and ULg about the alignment of the present content because at the moment the two sets of maps are different (asap)

Action TWG-14: planning and implementing by IFREMER and ULg mutually linking of SEXTANT and OceanBrowser; SEXTANT entries must give direct access to viewing of the selected product in OceanBrowser; Product Index in OceanBrowser must be fed by SEXTANT and include a SEXTANT link per product for detailed description (plan end November 2013; implementation March 2014)

Action TWG-15: optimising the SEXTANT discovery interface as a dedicated EMODNet Chemistry service, possibly including P35 and sea regions as extra search criteria and considering OceanBrowser as viewer by IFREMER (March 2014)

Action TWG-16: ULg will send a few ODV problem files to IFREMER to check whether NEMO can read and repair these (asap)

Action TWG-17: ULg will organise a Diva Workshop on Corsica - France 4 - 8 November 2013.

Action TWG-18: a pilot chain of services from ODV to Oceanotron to WPS will be formulated, deployed and evaluated by AWI, IFREMER and Deltares (March 2014)

ACTIONS from TWG-1

www.emodnet-chemistry.eu