Embed Size (px)
Citation preview
H2020 – EINFRA – 2015 – 1 Page 1 of 56
Technical Note on Infrastructure Deployment
Workpackage WP 6 VRE Deployment, maintenance and operations
Task Tasks 6.1, 6.2
Author (s) Ugo Di Giammatteo
Serena Avolio
Sergio de Gioia
ACS
ACS
ACS
Reviewer (s) Raffaele Guarino
Rosemarie Leone
ALMA
ESA
Approver (s) Cristiano Silvagni ESA
Authorizer Mirko Albani ESA
Document Identifier
Dissemination Level
Status
EVER-EST DEL WP6-D6.1
Public
Draft to be approved by the EC
Version 1.0
Date of issue 13 April 2017
H2020 – EINFRA – 2015 – 1 Page 2 of 56
Document Log
Date Author Changes Version Status
27/06/2016 Ugo Di Giammatteo Draft deployment plan prepared
v0.1 Draft
12/07/2016 Ugo Di Giammatteo
Serena Avolio
Methodology definition refined v0.2 Draft
05/10/2016 Ugo Di Giammatteo
Serena Avolio
Detailed activities identified. Technology choices refined
v0.3 Draft
14/10/2016 Ugo Di Giammatteo
Implementation task fully developed according to recommendation from Informal Interim Review (22 June 2016)
v0.4 Draft
28/10/2016 Ugo Di Giammatteo Comments from consortium reviewers integrated
v0.5 Draft
14/11/2016 Ugo Di Giammatteo Further comments integrated.
v0.6 Draft
15/11/2016 Ugo Di Giammatteo Final comments from formal reviewers implemented.
v0.7 Draft
27/12/2016 Ugo Di Giammatteo Update v0.8 Draft
25/01/2017 Ugo Di Giammatteo Final update V0.9 Draft
12/04/2017 Ugo Di Giammatteo Delivery according to DoA schedule
V1.0 Draft to be approved by the EC
H2020 – EINFRA – 2015 – 1 Page 3 of 56
Table of Contents
1 Introduction ........................................................................................................................................... 8
1.1 Purpose of the document ................................................................................................................. 8
1.2 Background ..................................................................................................................................... 8
1.3 Document structure ......................................................................................................................... 8
2 Infrastructure Integration and Deployment activities planning ................................................................ 9
2.1 Methodology ................................................................................................................................... 9
2.2 VRE architecture and interfaces definition ...................................................................................... 11
2.3 EVER-EST VRE services V1 development ......................................................................................... 11
2.4 Final development of EVER-EST services ......................................................................................... 12
2.5 VRE Infrastructure and solution trade-off ....................................................................................... 13
2.6 Infrastructure preparation ............................................................................................................. 14
2.7 System architecture analysis .......................................................................................................... 15
2.8 V1 Component Integration and deployment ................................................................................... 16
VRC portals (GUI) .............................................................................................................................. 19 2.8.1
ROHub ............................................................................................................................................... 19 2.8.2
Personal file storage .......................................................................................................................... 20 2.8.3
Authentication and user management ............................................................................................. 21 2.8.4
VM discovery and deployment ......................................................................................................... 22 2.8.5
E-Collaboration.................................................................................................................................. 22 2.8.6
Workflow runner ............................................................................................................................... 23 2.8.7
COCONET DB ..................................................................................................................................... 23 2.8.8
2.9 Deadlines ...................................................................................................................................... 24
2.10 Integration and deployment of final VRE components .................................................................... 25
2.11 Population ..................................................................................................................................... 25
2.12 Integration and deployment testing ............................................................................................... 27
3 Critical Activities .................................................................................................................................. 28
4 Technology Drivers ............................................................................................................................... 29
5 Deployment Risks ................................................................................................................................ 31
6 Smart Objectives and KPI Compliance ................................................................................................... 34
6.1 Smart objectives ............................................................................................................................ 34
6.2 KPIs ............................................................................................................................................... 35
7 Report on Integration and Deployment Activities .................................................................................. 36
7.1 Integration and deployment activities ............................................................................................ 36
7.2 Authentication (ACS) ..................................................................................................................... 36
7.3 Personal file storage (PSNC) ........................................................................................................... 36
7.4 VM management (T2) .................................................................................................................... 37
7.5 User management (provisioning, de-provisioning, sign-up) (ACS) .................................................... 37
H2020 – EINFRA – 2015 – 1 Page 4 of 56
7.6 Auditing (ACS) ............................................................................................................................... 37
7.7 Enterprise service bus (ACS) ........................................................................................................... 37
7.8 EO data discovery (T2) ................................................................................................................... 37
7.9 EO data storage (PSNC) .................................................................................................................. 38
7.10 E-collaboration (MEEO) .................................................................................................................. 38
7.11 Cloud services (T2/PSNC) ............................................................................................................... 38
7.12 Excution of remote services (T2) .................................................................................................... 38
7.13 RO integration in VRC portal (ACS, MEEO, PSNC) ............................................................................ 39
7.14 Remote workflow execution (PSNC) ............................................................................................... 39
7.15 COCONET (MEEO) .......................................................................................................................... 39
7.16 E-learning (T2) ............................................................................................................................... 39
7.17 Preservation (ACS) ......................................................................................................................... 39
7.18 Digital information services (MEEO) ............................................................................................... 39
7.19 ROHUB (PSNC) ............................................................................................................................... 39
8 ANNEXES ............................................................................................................................................. 40
8.1 ANNEX 1 sequence diagrams .......................................................................................................... 40
8.2 ANNEX 2 components’ delivery schedule ........................................................................................ 48
H2020 – EINFRA – 2015 – 1 Page 5 of 56
List of Figures
Figure 1 EVER-EST technical development process ................................................................................................... 9
Figure 2 High-Level Time constraints ...................................................................................................................... 10
Figure 3 Elementary activities of the technical process (limited to M22) ............................................................... 11
Figure 4 Integration Step #1 – ROHub functionalities ............................................................................................. 16
Figure 5 Integration Step #2 - Personal File Storage ............................................................................................... 17
Figure 6 Integration Step #3 - User Management .................................................................................................. 17
Figure 7 Integration Step #4 - VM management and E-Collaboration ................................................................... 18
Figure 8 Integration Step #5 - Full system ............................................................................................................... 18
Figure 9 Mapping VRC Portals Interactions with RO-APIs ....................................................................................... 19
Figure 10 CNR Scenario 1 – Habitat suitability model for CWC (D3.1 Appendix A) ................................................ 40
Figure 11 CNR Scenario 2 – The Citizen Science and jellyfish distribution (D3.1 Appendix A)................................ 41
Figure 12 CNR Scenario 3 - Trend in the evolution of invasive jellyfish distribution (D3.1 Appendix A) ................ 42
Figure 13 CNR Scenario 4 - Habitat suitability model for jellyfish species, study case Pelagia noctiluca and Velella velella (D3.1 Appendix A) ................................................................................................................................ 43
Figure 14 INGV Scenario 1 - The Mt. Etna ground deformation investigation (D3.1 Appendix A) ......................... 44
Figure 15 NHP Scenario 1 - Surface water flooding hazard impact modelling case studies (D3.1 Appendix A) ..... 45
Figure 16 SatCen Scenario 1 – Land monitoring (D3.1 Appendix A) ....................................................................... 46
Figure 17 RO Re-using Scenario (generic scenario to show a common functionality) ............................................ 47
List of Tables
Table 1 Deployment needs for system components .............................................................................................. 14
Table 2 Technical Control Points ............................................................................................................................. 24
Table 3 COTS and 3rd Party components ................................................................................................................ 30
Table 4 EVER-EST Risk Table .................................................................................................................................... 33
H2020 – EINFRA – 2015 – 1 Page 6 of 56
Definitions and Acronyms
Acronym Description
API Application Programming Interface
CMS Content Management System
DOA Description Of Action
DOI Digital Object Identifier
E2E End to End
EO Earth Observation
ESA European Space Agency
ESB Enterprise Service Bus
EVER-EST European Virtual Environment for Research - Earth Science Themes
GUI Graphical User Interface
HTML Hypertext Mark-up Language
KPI Key Performance Indicator
M2M Machine To Machine
OGC Open Geospatial Consortium
OIDC OpenID Connect
RDF Resource Description Framework
REST Representational State Transfer
RO Research Object
RODL Research Objects Digital Library
SLA Service Level Agreement
SMART Specific Measurable Achievable Relevant Timely
SQL Structured Query Language
SSO Single Sign-On
URL Uniform Resource Locator
VRC Virtual Research Community
VRE Virtual Research Environment
Applicable Documents
Document ID Document Title
[1] European Virtual Environment for Research - Earth Science Themes, Grant Agreement N° 674906 – Available on the EVER-EST Alfresco Platform under WP1 Management Folder
[2] EVER-EST Description Of Work
[3] Review Report (from Informal Interim Review held in Brussels on 22/06/2016)
H2020 – EINFRA – 2015 – 1 Page 7 of 56
[4] D1.1 EVER-EST Project Management Plan WP1
Reference Documents
Document ID Document Title
AMD-674907 Amendment to Grant Agreement 674907
EVER-EST DEL WP5-D5.1 VRE Architecture and Interfaces definition
EVER-EST DEL WP5-D5.2 Technical Note on Common Services – Intermediate Version
H2020 – EINFRA – 2015 – 1 Page 8 of 56
1 Introduction
1.1 Purpose of the document
A Technical Report on Infrastructure Deployment shall be delivered at M17 according to [AD1]. As a follow up of the Informal Interim Review held in Brussels on 22/06/2016 a preliminary version of the Technical Report was requested to “Develop a detailed implementation plan of the tasks, technology choices and critical activities to be accomplished by the time of the 1st periodic review foreseen after the end of March 2017; this should be accompanied by an appropriate risk monitoring and contingency plan [AD3]
The detailed implementation plan, critical activities and risk assessments provided by this document will be shared and used by all parties as master schedule and overall system deployment status report.
1.2 Background
The EVER-EST VRE is based on a Service Oriented Architecture that has been preliminary identified in the DOA and re-defined in the first 12 months of the project.
The architectural design is based on the powerful concept of research objects (ROs), developed by ESI/PSNC and consolidated within WP4.
A revision of the VRE architecture system component modules has been conducted to take into account the latest technical achievements in WP4 and WP5 deliverables and the more accurate definition of user needs in WP3 deliverables. Deployment strategy and physical resources needs for the VRE deployment have been defined and detailed in WP6 deliverable.
1.3 Document structure
This document is composed of the following Sections:
1. Introduction, which explains the purpose and the background of the document. 2. Infrastructure Integration and Deployment activities planning, in which the general approach is
described. In this section the integration and deployment technical activities are identified, described in detail, in terms of scope, criticality and relationships with the other activities and overall planning.
3. Critical activities, in which the critical issues of the development/integration/deployment process are described.
4. Technology drivers, which describes the main choices made during the design phase, their justification and their impact on the project.
5. Deployment risks, in which the project risks are identified, ranked and discussed. For each identified risk a mitigation strategy is indicated along with recovery actions.
6. Compliance to SMART Objectives and KPIs, which describes how the document contributes to achievement of project goals.
H2020 – EINFRA – 2015 – 1 Page 9 of 56
2 Infrastructure Integration and Deployment activities planning This section describes the detailed planning of activities related to the EVER-EST VRE integration and deployment. It identifies the main technical activities related to the system development, integration and deployment and provides a summary of the technical discussions that have been carried out during the first 12 months of the project.
2.1 Methodology
The overall process concerning the technical development is shown in the following figure:
Figure 1 EVER-EST technical development process
The following project milestones are defined:
M14 research objects and V1 Services Delivery;
M17 VRE V1 Deployment (MS3);
M18 Intermediate Project Review with EC Project Officer;
M18 Final Services Delivery;
M22 VRE Final Deployment.
Activities are organised to fully comply with these milestones ensuring the best feedback loop to take into account technical versus user communities’ feedback and constraints.
The following diagram illustrates the project time constraints, together with the reference to the 1st Interim Review:
WP5
SERVICES DESIGN AND DEVELOPMENT
WP6
INTEGRATION AND DEPLOYMENT
WP3 REQUIREMENTS FROM VRCs
WP4 RESEARCH OBJECTS
Services ready for integration
Need for rework
Usersconstraints
Technical constraints
H2020 – EINFRA – 2015 – 1 Page 10 of 56
Figure 2 High-Level Time constraints
Figure 2 highlights the critical time constraints due to the choice of releasing the complete verified system by Month 24 to WP3 for the validation activities. Thus permitting (after the verification of the system) a full year of validation and pre-operations by the pre-selected VRCs user communities starting from M24. The period from M24 to M36 will allow the project teams to prepare for full-scale operations.
The tasks that are addressed by the WP5 and WP6 teams from M13 to M22 are detailed below:
Task 5.1 - VRE - Architecture and Interfaces Definition;
Task 5.2 -> 5.5 - Services design and implementation;
Task 6.1 – VRE Core infrastructure deployment;
Task 6.2 – VRE Services integration with Core infrastructure;
Task 6.3 – VRE Population.
Components are designed, developed and made available (in WP5) as separate modules. Once these components are delivered, they are progressively integrated and deployed in the target platforms (WP6).
The following sections describe the breakdown of technical activities to allow the project to systematically track and monitor their progress.
Validation activities are outside the scope of D6.1 and addressed by D3.2.
H2020 – EINFRA – 2015 – 1 Page 11 of 56
Figure 3 Elementary activities of the technical process (limited to M22)
On the upper part of the diagram are shown the project milestones as described in AD [2]. The activities are described in detail in the following pages.
2.2 VRE architecture and interfaces definition
This activity involves the definition of the overall system architecture, the detailed identification of system components and the interfaces among them.
It has been carried out in Task 5.1, in close collaboration with Task 6.1. D5.1 - VRE Architecture and Interfaces Definition deliverable updated version constitutes the applicable basis for the development, integration and deployment activities that will take place until M22.
The VRE architecture and interface definition task completed jointly by WP5 and WP6 was a pre-requisite for planning the deployment phase since it facilitated the definition of the integration sequence and the design of the verification test procedures to assess the dynamic behaviour and functioning of the system being assembled under WP6.
2.3 EVER-EST VRE services V1 development
The development of the EVER-EST VRE services is managed within WP5 and is reported here to show the correlation and timeline of development activities and integration/deployment activities schedules, progressive resources procurements, criticalities and constraints.
H2020 – EINFRA – 2015 – 1 Page 12 of 56
V1 development involves the development of the first subset of the following classes of services according to each individual task:
Task 5.2 – Common services;
Task 5.3 – Digital Information and E-Collaboration Services;
Task 5.4 – E-Research Application Services;
Task 5.5 – E-Learning application services.
The master schedule of the subset of services to be implemented within V1 is summarised in Annex 2 of this document (§8.2) and is maintained updated on-line on the EVER-EST collaboration environment, for the incoming verification and validation activities and serves as the basis for component deployment status tracking for the project coordination.
The components and services that will be implemented in V1 are the following:
VRC Portals;
Authentication;
Personal file storage (Seafile);
Virtual Machine discovery and deployment;
User management;
E-Collaboration o Notify User;
ROHub (All except Discover and Search RO)
Workflow runner;
COCONET DB.
The detailed list of components and interfaces, along with the corresponding dates for deliverables are reported in Annex 2 (§8.2).
2.4 Final development of EVER-EST services
This activity will start immediately after the delivery of the V1 services and will be carried out until M18. The full set of EVER-EST services will be delivered at M18, ready for the successive integration and deployment.
The availability of all components for the integration of the final system will allow the verification of the EVER-EST infrastructure and the system to progressively become operational.
The components and services that will be added in the final delivery are listed hereafter:
Auditing (Data Analytics);
Authorisation
Enterprise Service Bus;
Data Discovery;
E-Collaboration o Peer Review; o Instant Messaging;
Cloud Services;
Data Storage;
ROHub o Discover and Search RO;
e-Learning;
H2020 – EINFRA – 2015 – 1 Page 13 of 56
Preservation.
2.5 VRE Infrastructure and solution trade-off
During this activity, the needs related to the physical infrastructure to host the EVER-EST services, are defined.
This activity consists of:
Analysis of EVER-EST component infrastructure resources requirements and needs;
Assessment of available HW resources within the consortium;
During the first phase of the project various scenarios were explored for the hosting of EVER-EST system.
This process ended in the Venice plenary meeting, where PSNC and INGV declared their availability to host the VRE components at their premises. No technical constraint was identified to limit the deployment of the VRE exclusively within PSNC or INGV datacentres. It will be possible, if needed to expand the system deployment and exploit other cloud resources (provided by other partners or commercial cloud services).
As a result of the trade-off and Venice plenary meeting discussion, it was decided to deploy the VRE using the available resources at one of the EVER-EST project partner datacentre. The characteristics of the PSNC platform were assessed to be more suitable for the deployment of the EVER-EST VRE than the one proposed by INGV,
Based on the above assumptions a questionnaire was prepared and sent to all technical partners to assess the resources needed by the system components. The following table summarises the results of this survey.
H2020 – EINFRA – 2015 – 1 Page 14 of 56
Table 1 Deployment needs for system components
2.6 Infrastructure preparation
In the frame of this task, in coordination with PSNC, the necessary resources according to the requirements defined in Table 1 have been deployed. This has implied the following activities:
Analysis of the results of the questionnaires inputs to identify hardware needs (see §132.5);
Set-up of the procedure, based on Jira, for virtual machine requests and provisioning;
Activation of the procedure to initiate the components deployment.
A centralised approach for the virtual machine requests and provisioning was chosen for the following reasons:
The number of virtual machines to be deployed is not expected to be significantly high.
Most of the virtual machines required by the various partners were already deployed while setting up the centralised procedure.
Partition CPUs RAM (GB) Local storage (GB) Volume (TB)
General Resources 64 128 1280 100
RO-HUB (reserved) 32 64 640 2
General Resources 64 128 1280 100
Contingency (20 %) 13 26 256 20
Per tech partner 17 34 341 27
ID Name memory_MB Disk Ephemeral Swap VCPUs RXTX_Factor IS_Public
1 m1.tiny 512 1 0 1 1 True
2 m1.small 2048 20 0 1 1 True
3 m1.medium 4096 40 0 2 1 True
4 m1.large 8192 80 0 4 1 True
5 m1.xlarge 16384 160 0 8 1 True
Partner Components VM_Type Number of needed VM CPUs RAM Local Storage Notes
ACS WSO2 ESB 3 1 2 4096 40
ACS WSO2 Identity Server 3 1 2 4096 40
ACS WSO2 Analytic Server 4 1 4 8192 80
8 16384 160 ACS OK
TERRADUE Support Center (T6.5) 3 1 2 4096 40
2 4096 40 T2 OK
MEEO VRE portal 3 1 2 4096 40
MEEO VRC 1 portal 3 1 2 4096 40
MEEO VRC 2 portal 3 1 2 4096 40
MEEO VRC 3 portal 3 1 2 4096 40
MEEO VRC 4 portal 3 1 2 4096 40
10 20480 200 MEEO OK
TOTAL 20 40960 400
NERC
DLR
ALMA
SATCEN
INGV
CNR
PSNC Service provider
ESI Reserved resources
Machine pre-defined size
H2020 – EINFRA – 2015 – 1 Page 15 of 56
Most of the partners do not know the process for the creation and configuration of a virtual machine in OpenStack, and if they will need to do it only once or twice, it is not worth for them, at this stage of the project to spend efforts in learning such process.
The process of requesting a virtual machine is managed and maintained directly by the PSNC Help Desk. In particular, the Help Desk service enables the following actions:
Creating a virtual machine with well-defined capacity and features;
Running the virtual machine in the requested time frame;
Suspending the virtual machine outside a requested time frame;
Closing or deleting a virtual machine;
Creating a snapshot of a virtual machine in a specific state;
Loading/reloading a virtual machine with a new instance of an operating system;
Loading/reloading a virtual machine from a specific snapshot;
Re-assigning resources to a virtual machine, if such resources are available.
All aspects related to the maintenance of the particular operating system running on the assigned virtual machine will be handled by an appointed member of a specific development team who will be given operating system root level privileges
During pre-operations and for future external parties that may want to integrate their applications these procedure will be re-assessed and resources assigned as defined by the sustainability model adopted.
2.7 System architecture analysis
Identification of detailed interfaces and dynamic behaviour is the step of this activity.
This activity consists of the progressive refinement of system architecture design in order to allow the integration of the various components, the correct exchange of data among them and the actual deployment of the EVER-EST VRE. This activity, carried out from M5 to M14, allows identifying the details of the dynamic behaviour of the system.
The analysis started from the Architecture Document (D5.1) and progressively defined the interfaces of the various components. This activity comprised:
1-to-1 meetings, aimed at interviewing the technical partners in order to agree on the services’ scope and interfaces;
Preparation of sequence diagrams based on D3.1 deliverable use case and user scenarios description. This task is key to identify the dynamic functioning of the system and to correct possible inconsistencies in the overall design.
The procedure is articulated over the following steps, which are applied to each user action:
1. Reading of the user action in order to identify the external actors involved; 2. Identification of the main VRE components involved in the action; 3. Identification of sub-components, where possible; 4. Identification of interfaces between components; 5. Consistency check and verification of naming and commonalities.
This procedure generated the sequence diagrams reported in Annex 1(§8.1).
The System architecture analysis helped to define the priorities of services and to decide the components to be implemented in V1 (see §2.3)
H2020 – EINFRA – 2015 – 1 Page 16 of 56
The analysis of user actions allowed to understand the logical link between the user-level functionalities and the way ROHub backend implements these functionalities. The purpose is to fully decouple the RO-Environment from the user experience.
The ROs constitute the technical means that allows sharing, preserving, editing, publishing, searching and evolving user experiments. The correspondence between each user actions and the corresponding sequence of API calls has thus been identified.
2.8 V1 Component Integration and deployment
This activity is composed of the integration of components developed in WP5 in the infrastructure and the refinement of the deployment. It is phased with development activities carried out in WP5.
This activity started at M11 and is proceeding in parallel with the V1 service development. This allows the early identification of integration issues that require re-work on development. This activity continues after the completion of V1 components until the deployment of V1 VRE at M17.
The following diagrams show the deployment sequence.
Figure 4 Integration Step #1 – ROHub functionalities
H2020 – EINFRA – 2015 – 1 Page 17 of 56
Figure 5 Integration Step #2 - Personal File Storage
Figure 6 Integration Step #3 - User Management
H2020 – EINFRA – 2015 – 1 Page 18 of 56
Figure 7 Integration Step #4 - VM management and E-Collaboration
Figure 8 Integration Step #5 - Full system
The following sections describe the main characteristics of the V1 Components/Services and the guidelines for their integration.
H2020 – EINFRA – 2015 – 1 Page 19 of 56
VRC portals (GUI) 2.8.1
These components are the user interfaces that will be used by the user communities to access the infrastructure and exploit all available functionalities.
Each VRC portal will be customised to the specific needs of the community.
The aim is to have VRC specific entry portals with a set of functionalities common to all VRCs complemented by VRC specific features for data analysis and presentation. The VRC portals are based on Django, a CMS that allows the development of specific portlets, customised to the needs of the communities. Details related to the VRC Portals are available in D5.3.
Integration methodology
The first step for integration consists of interfacing the VRC portals to ROHub, which constitutes the core of the VRE knowledge. ROHub exposes a series of APIs that are invoked by the VRC portals. In some cases a single operation on the portal side corresponds to a sequence of API calls on the ROHub side. This is being implemented by introducing a middleware that exposes a Façade API towards the portal and creates the appropriate sequences for the ROHub. Further details are available in D5.2.
The following figure depicts the high-level mapping between interactions with portal and ROHub APIs.
Figure 9 Mapping VRC Portals Interactions with RO-APIs
ROHub 2.8.2
The EVER-EST VRE is the first RO-centric native infrastructure leveraging the notion of research objects and their application in observational rather than experimental disciplines and particularly in Earth Science.
H2020 – EINFRA – 2015 – 1 Page 20 of 56
The RO functionalities can be grouped in four categories:
RO Access and Usage;
RO Data Management and Analysis;
RO Storage and Preservation;
RO Lifecycle.
The ROHub is composed of a front-end (RO-Portal) and a backend (RO-Services). The RO-Portal implements a set of features to help scientists throughout the research lifecycle to (i) create and maintain high-quality ROs that can be interpreted and reproduced in the future; (ii) to collaborate along this process; (iii) to publish and search these objects and their associated metadata; (iv) to manage their evolution; and (v) to monitor and preserve them supporting their accessibility and reusability.
ROHub backend (RODL) has a modular structure that comprises access components, long-term preservation components and the controller that manages the flow of data. ROs are stored in the access repository once created, and periodically the new and/or modified ROs are pushed to the long-term preservation repository. The access components are the storage backend and the semantic metadata triple store.
RO functionalities are realised by a set of services exposing RESTful APIs alongside client applications and the storage backend is based on dLibra. The semantic metadata are additionally parsed and stored in a triple store backed by Jena TDB (Triple Data Base) meanwhile the long-term preservation component is built on dArceo.
Integration methodology
Based on the analysis of user interactions with the system, the ROHub will be progressively integrated with the VRC Portals. The integration will proceed according to the following order:
Integration of the ROHub functionality in the VRE Portal;
Create RO;
Add resources;
Delete resources in a RO;
Delete RO;
Delete resource;
Get RO List (filtered or not);
Get RO;
List the resources in a RO;
Get a resource in a RO.
Personal file storage 2.8.3
The EVER-EST VRE users shall require space where they can store complementary or auxiliary resources (e.g., documents, scripts, papers, intermediate results, etc.) and this will be provided by a Personal Data Storage via SeaFile that is similar to Dropbox and Google Drive.
Seafile is written in C and Python. It is cross-platform software that offers a choice of MySQL/MariaDB or SQLite for database and supports file system or distributed storage as data storage. It is free and open source and can be installed by users on their own servers without limits of storage capacity or number of clients.
Seafile offers the following features:
File synchronisation;
Client-side encryption;
Per-folder access control;
Version control;
H2020 – EINFRA – 2015 – 1 Page 21 of 56
Public link-sharing;
File editing.
The instance of Seafile hosted in the PSNC environment is integrated with the EVER-EST Identity Service Provider by means of the SAML 2.0 authentication protocol. Possible future configurations/customisation are the definition of user quotas or increasing of the initial size (5TB).
Integration methodology
The integration of Seafile is straightforward. Once installed, the other services, applications and components can access Seafile's functions remotely via REST calls. All API calls must be authenticated and the offered APIs are WEB, python and PhP which are all respectively documented in:
https://manual.seafile.com/develop/web_api_v2.1.html
https://manual.seafile.com/develop/python_api.html
https://github.com/rene-s/Seafile-PHP-SDK
Authentication and user management 2.8.4
The Authentication and User Management is implemented by an open source tool, WSO2 Identity Server that makes use of Java technology and is built on top of Carbon, a SOA middleware platform based on Apache.
WSO2 Identity Server provides secure Identity Management for the EVER-EST Web Application services and APIs by managing identity and entitlements of the users securely and efficiently. It offers the following features:
Authentication, the process of identifying an individual. Authentication merely ensures that the individual is who he or she claims to be.
Authorization, the process of granting or denying access to a network resource. The authorization allows the user access to various resources based on the user's identity.
Account Provisioning, the process of providing users with access to data and technology resources. The process implies that the access rights and privileges are monitored and tracked to ensure the security of the VRE resources.
Identity Administration, the process of creating new and modifying or deleting existing identities as well as managing the security entitlements associated with those identities.
Security, the process to provide secure access to resources based on standards and model for security and user security requirements.
All these functionalities are centralized to support the coordinated usage of the different EVER-EST components and allow their interoperability. The user will be automatically authenticated on all application enabling a SSO scenario once the necessary credentials have been provided. Further details concerning the 2WSO2 Identity Server are available in D5.2.
Integration methodology
The Identity Server shall be configured and customised to integrate the different EVER-EST components, applications and services. Each of the EVER-EST components shall be registered with the Identity Server as service provider and linked to the involved authentication protocol. The Open ID Connect and the SAML 2.0 authentication protocols are involved in the integration of the current EVER-EST components.
H2020 – EINFRA – 2015 – 1 Page 22 of 56
VM discovery and deployment 2.8.5
The data processing infrastructure that Terradue brings to EVER-EST is based on a cloud computing Platform-as-a-Service (PaaS) approach, in which applications can be integrated using virtual machines with their own specifications (disk size, processor speed, operating system etc.) that can run on private (physical deployment over local hardware) or on commercial Cloud infrastructures. The virtual machines are appliances running on an infrastructure managed by a Cloud Controller, and accessible as an independent domain once instantiated. These appliances initially constitute a development environment where applications or services are integrated and then are moved to production centres when fully deployed for their exploitation.
The platform is delivered "as-a-Service" to applications developers, allowing them to:
Access a dedicated application integration environment in the cloud, with software tools and libraries available and easily exploitable from that developer environment;
Search for data collections from distributed Earth Observation Data repositories and from Open Data repositories within the platform, and retrieve results in the application integration environment;
Integrate a set of data processing chains (data pipeline), leveraging selected data programming models (e.g. MapReduce and non-MapReduce frameworks such as Distributed-Shell, graph processing, MPI, TEZ, Spark, Storm etc.)
Deploy a Cloud appliance (e.g. for on-demand Earth Observation data processing) prepared for the processing of a range or category of datasets, and running on a selected third party Cloud service provider.
Expose the resulting application through a Web Service endpoint.
The Terradue cloud platform is powered by the open source software Open Nebula and the cloud controller, in charge of the cloud resources management is an extension of the OpenNebula cloud controller.
The Developer cloud sandbox PaaS allows to integrate scientific applications written in a variety of languages (e.g. Java, C++, IDL, Python, R, scripting languages).
Integration methodology
The integration of applications will be carried out according to the following steps:
Definition of parallelisation strategy: where the service developer defines the DAG (directed acyclic graph) of the service showing the different jobs, inputs and outputs of the workflow. This graph will show the parallelization strategy to be applied on the cloud environment.
Definition of data sources: the data requirements need to be analysed and their retrieval mechanism accessed to make sure that the data is available in the system to be consumed as expected. In this assessment not only the technical aspects will be considered but also the legal (data policies enforcement) and financial ones (e.g. commercial data).
Analysis of tools and libraries: the tools and libraries necessary to execute the applications need to be analysed along with their compatibility with the computational resources that are available. Technical, legal and financial aspects for each tool shall also be taken into consideration.
Once integrated in the provided framework, the application can scale under the automatic management (job distribution and task trackers over a compute cluster). The resulting applications are exposed via the OGC Web Processing Service (WPS) interface to perform the end-to-end verification and subsequently exploit the application in a B2B environment.
E-Collaboration 2.8.6
The E-Collaboration services component facilitates the interaction, communication and collaboration between users. They provide collaboration management, group management, peer review, synchronous (e.g. 1-to-1 and 1-to-N instant messaging, audio-visual conferencing, etc.) and asynchronous (e.g. forum, blog, wiki pages, send
H2020 – EINFRA – 2015 – 1 Page 23 of 56
and receive e-mail, etc.) services. E-Collaboration instant messaging services are based on the eJabberd server. It runs on several Unix-like operating systems such as Mac OS X, GNU/Linux, FreeBSD, NetBSD, OpenBSD and OpenSolaris. Additionally, ejabberd can run under Microsoft Windows. Other e-collaboration services have been integrated as JavaScript extensions.
Django, the Content Management System (CMS) on which the EVER-EST gateway portal and the VRC portals are based, allows the integration of most of the e-collaboration capabilities through third-parity plugins.
The CMS technology allows publishing, editing, modifying, organising, deleting, and maintaining content from a central interface.
Integration methodology
The client part of the E-Collaboration services and the following add-ons have been installed during the EVER-EST gateway initial configuration:
Internal mailing and notification;
Forum;
Instant Messaging.
The eJabberd server was installed and configured on the same machine that hosts the EVER-EST Portal.
Workflow runner 2.8.7
Scientific workflows play an important role for sharing, exchanging and reusing scientific methods. In order to assess if a workflow is functional, it is generally useful to be able to re-execute it. However different workflow systems have different ways of running a workflow. For instance, Taverna has the Taverna Server, while Wings has a portal and a Pegasus/Condor Engine in the backend.
Workflow Runner component, developed in the Wf4Ever project, provides a common lightweight interface for addressing the needs of the community. Moreover it allows managing workflows in an agnostic and independent way from WFMS. These services offer an API where users can count with features such as “Run this workflow please” and “Show me the data from that workflow run”.
The Workflow Runner HTTP REST API mirrors the RODL (ROHub backend) API, but each RO exposed by this service represent a particular workflow run structured to show inputs, outputs, console logs, provenance and annotations containing wfprov and wfdesc mappings.
To access the root of the service, the system should redirect to a default server runs resource. From here the client may either:
POST a new workflow run, providing as a minimum the workflow definition;
GET a list of existing workflow runs;
DELETE existing workflow runs.
Integration methodology
In V1 the integration of Workflow Runner consists in the launch of the POST of a new workflow run from the client.
COCONET DB 2.8.8
In the frame of the EVER-EST project the first catalogue to be interfaced will be the CoCoNet GIS catalogue, which is based upon the ESRI Geoportal Server 1.2.6 and managed directly by CNR-ISMAR. This catalogue has been included in the scope of V1 because it is the central point of the CNR Use Case, which is particularly useful to demonstrate many functionalities of the EVER-EST system.
H2020 – EINFRA – 2015 – 1 Page 24 of 56
CoCoNet integrates and provides access to the datasets in EMODnet that are considered most relevant for the use cases deployed in EVER-EST.
CoCoNet makes data available freely, without restriction and in a timely and easy way to users through the WebGIS platform, but CoCoNet data remains dependent on data contributions.
Integration methodology
The current way of using CoCoNet is through a direct interaction of the user with the WebGIS. In the EVER-EST integration the OpenSearch APIs offered by the Geoportal will be used.
2.9 Deadlines
Control points, which are listed in the following table, are necessary to check the timing of integration process with respect to the expected project schedule.
Date Control Point Objective Comment
17 Oct 2016
Verification of:
- Integration of the ROHub functionalities with the VRC portals (i.e. authorization code flow for oauth2 - status of the activity);
- Integration of the Data Discovery and Data storage functionalities with the VRCs portal;
- Possible approaches for the user de-provisioning;
- Possibility to set the group in Seafile;
- Collaboration spheres as OIDC client;
- Actions contained in the shared technical spreadsheet.
Completed
30 Nov 2016 Availability of all components/services foreseen by the technical planning (see §8.2)
Completed
2 Jan 2017
Verification of development status of full VRE components as foreseen for that date. Verification of the integration status of V1 components (basic end-to-end sessions must be possible).
Completed
6 Feb 2017
Verification of the development status of the full VRE components as foreseen for that date. Verification of the integration status of the V1 components (some complex end-to-end sessions must be possible)
Completed during Technical Meeting at ESRIN (7-8 February 2017)
28 Feb 2017 Verification of development status of the full VRE components as foreseen for that date. Verification of full integration of V1 components.
Completed
31 Mar 2017 Verification of development completion of the full VRE components. Completed
Table 2 Technical Control Points
H2020 – EINFRA – 2015 – 1 Page 25 of 56
The scope of the above control points will be refined and enriched as new information becomes available, according to the rolling-wave planning paradigm.
It has to be highlighted that, besides the control points indicated above, the WP6 technical meetings held every two weeks will allow the timely detection of critical situations.
2.10 Integration and deployment of final VRE components
This activity is due to start on M17. Its detailed description will be provided as the necessary information becomes available.
2.11 Population
This activity consists of the insertion of VRC data within the VRE and the integration/linking of data catalogues of the different user communities. It is articulated in successive phases, according to the successive versions of the VRE and to the availability of data and metadata from users’ communities.
The topic of VRE population has been extensively discussed in the plenary meeting that was held in Edinburgh on September 2016.
The VRCs have been requested to provide the list of their data of interest. For each dataset the full description of data, size, type, access mode and contact person have been identified.
The updated list of dataset at this point in time is:
INGV
GPS GPS data catalogue for Italian geodetic network
GPS data from Etna geodetic network catalogue for the IPVW Algorithm
ENVISAT SLC ASAR data
SENTINEL-1 TOPSAR acquisition mode (250kmx250km ground) - Sentinel 1A and 1B
MSG SEVIRI MSG-SEVIRI dataset download from EUMETCast or other similar archive services
MODIS Terra/Aqua-MODIS dataset download from NASA-LAADS or other similar archive services
IPVW MODIS data
Atmospheric Atmospheric Profiles from WMO stations and/or from NCEP and ECMWF.
GRIB GRIB data from NOAA catalogue
CNR-ISMAR
CWC Geographical points where cold water corals have been sighted
Geographical points where cold water corals are not present
Bathymetry Digital terrain model of the bathymetry (high resolution data)
H2020 – EINFRA – 2015 – 1 Page 26 of 56
EVG Environmental variables derived from the bathymetry (e.g. Slope and Aspect)
Jellyfish Geographical points where jellyfish have been sighted from citizens
Environmental Environmental variables from Copernicus (sst, ssh, current u and v, sss)
NHP
NPD Point based GIS dataset that combines location information with population information, including sensitive populations e.g. schools, hospitals etc.
NRD Residential and non-residential properties – location, type, use, value and size
OSMM Ordnance Survey MasterMap Integrated Transport Network layer
OSBM Various background mapping datasets providing contextual geography information
SWIL Database of pre-calculated SWF impact information used for creating impact scenarios for the SWF HIM
Flood Maps Updated flood map for based on output from SWF modelling using G2G
SWF hazard SWF hazard footprint produce by CEH using G2G modelling in house
County areas Flood Forecasting Centre reporting areas used to generate county summaries of impact output
MetOffice Observed rainfall data readings from rain gauges
Observed radar rainfall data
Rainfall forecasts – 24 members (files) for each ensemble forecast
Met Office streamed data received twice daily in NIMROD proprietary format. Needs to be converted to ASCII format before use
Met Office streamed data received twice daily in NIMROD proprietary format. Needs to be converted to ASCII format before use
BGS Slope BGS licensed landslide susceptibility model based on 25m DTM and 5 categories
BGS DigMap BGS licensed geology layers – bedrock, superficial deposits and mass movement
Landslides Regional characterisation of landslides across GB based on dominant failure type, style and geomorphology
17 500 records of landslide events and associated information for Great Britain; it excludes Northern Ireland, Isle of Man and the Channel Islands
LexisNexis DB Database of news articles used to collate information on observed impacts. Collated by KCL.
SATCEN
Sentinel-1 Sentinel 1 Level 1 SLC
Sentinel 1 Level 1 GRD
Sentinel-2 Sentinel 2 Level 1C
H2020 – EINFRA – 2015 – 1 Page 27 of 56
2.12 Integration and deployment testing
Integration and Deployment System testing activities are necessary to check the proper functioning of the system and its suitability for the users’ objectives. In the frame of EVER-EST project, the following user requirements verification and validation definition apply:
Verification: the evaluation of whether or not a product, service, or system complies with a regulation, requirement, specification, or imposed condition. It is often an internal process. Implies answering the following question: "Are we building the product right?";
Validation: the assurance that a product, service, or system meets the needs of the customer and other identified stakeholders. It often involves acceptance and suitability with external customers. Implies answering the following question: "Are we building the right product?"
Verification testing is carried out by technical partners within Task 6.4 and will follow well-documented test procedures (manual or automated by means of scripts). This activity will take place from M16 to M34 and have two peaks in M17-M18 (for VRE first release) and in M22-M24 (for VRE final release). The results will be collected and inserted in the deliverable D6.2 that will document the details on the test case specification, the results obtained and the possible deviations with respect to the test expectations and success criteria.
Validation testing is carried out with the direct involvement of users’ communities in WP3. This activity will be part of Task 3.6 and will demonstrate that the system is completed satisfactorily, satisfies users’ needs and is thus accepted.
The users will then use operationally the system from M24 to M36. During this period they will be able to identify bugs or shortcomings that will be corrected in the framework of evolutionary maintenance (Task 6.5).
During this operational phase will be provided an indication of waived/superseded/new user requirements and new use cases which have been identified by the assessment as potential new requirements to be implemented in subsequent infrastructure updates or possible follow on of the project.
H2020 – EINFRA – 2015 – 1 Page 28 of 56
3 Critical Activities The development and integration planning depicted in Figure 3 shows two finish-to-start relationships:
The link between Integration of V1 components and Integration of final VRE components to be verified by WP6;
The link between V1 service developments and final services developments to be validated by the pre-selected communities under WP3.
In this light the proper completion of V1 is critical to the implementation and deployment of the whole system.
At the current stage of development this risk is classified as low. The project requirement to develop a SOA architecture for the VRE mitigates the criticality of V1 successful verification and deployment versus to assure progressing and completion of the final VRE system deployment, according to schedule. The capability and flexibility to easily accommodate integration of SW components in the infrastructure reduces the impact of delay for components that might be not ready for integration as originally planned.
A low risk with high impact has been assigned to the risk that users will not be able to validate functionalities and to provide timely feedback to the project due to delay in components delivery and services availability. This risk has been mitigated by involving the WP3 in all the technical discussion to properly manage and adjust the schedule of the validation test to avoid impact on the system component integration due to feedback delay in the validation activities (described in §2.12).
H2020 – EINFRA – 2015 – 1 Page 29 of 56
4 Technology Drivers The main technology choice stated in the DOA as one of the project objective and recalled in §3, is the adoption of Service Oriented Architecture. The VRE is fully compliant to this paradigm. This allows the insertion of existing modules and the system expandability as new services become available.
Another technology choice (mentioned in the DOA) is the concept of research objects and the consequent adoption of the ROHub as the tool to manage ROs, which are the technical pivot of EVER-EST VRE. The architecture is thus fully RO-centric. However the design is structured in such a way that the user can use the system without necessarily knowing all the inner details of RO management.
In addition to the above technical choices some other decision have been made during the first year of the project. They are reported as follows:
The system is based on a classical 3-tier structure (presentation layer, service layer and data layer);
Though RO-centric, the system will show a front-end (4 VRC portals + a generic Earth Science portal), which shields the inner mechanism of RO management;
WSO-2 is integrated within the VRE. It will allow: o The implementation of a complex authentication mechanism and the SSO (via the Identity
Server); o The measurement of real time status of system components (via the data analytics); o The decoupling/mediation of communications between components and the creation of blocks
of instructions to access ROHub back-end (via the Enterprise Service Bus).
The Seafile system is used as a Dropbox-like personal storage system for end-users;
The EVER-EST user interface will leverage on the multi-sensor evolution analysis developed by MEEO;
Django has been selected as a suitable system to provide e-collaboration services through native CMS plugins or third-party plugins;
E-learning services will be based on Jupyter notebook tool.
Full detail on technical choices can be found in the technical documents produced by WP5 (D5.1 for general architecture and D5.2 through D5.11 for specific services).
The EVER-EST system strongly relies on COTS software, libraries and other 3rd Party tools. The following table summarises these components.
COTS/3rd Party components Version Comment
WSO2 Identity Server 5.3.0 WSO2 products are 100% open source and released under the Apache License Version 2.0.
WSO2 Enterprise Service Bus 6.1.0 Same as above. WSO2 Data Analytics Server 3.1.0 Same as above.
Seafile 6.0.9 Open Source (GPLv3)
Django-machina 0.5.3 Forum capabilities plugin. Open Source.
Django-postman 3.5.1 Internal messaging and notification capabilities. Open Source.
Django-todo 1.6.1 Calendar/scheduling capabilities
ejabberd 17.04 Chat capabilities. Open Source
Taverna Server 2.5.4 Workflow Management. Taverna 2.x is licenced under the Lesser General Public License (LGPL) Version 2.1. Taverna 2.x was released by University of Manchester
H2020 – EINFRA – 2015 – 1 Page 30 of 56
Jupyter 5.0 Jupyter Notebook supports a web notebook for e-learning. Jupyter is licensed under the BSD 3-clause "New" or "Revised" License A permissive license similar to the BSD 2-Clause License, but with a 3rd clause that prohibits others from using the name of the project or its contributors to promote derived products without written consent.
Australian Geoscience Data Cube
1.3.0 E-learning tool for organising and analysing vast quantities of Earth Observation data. Licensed under the Apache License, Version 2.0.
Table 3 COTS and 3rd Party components
Management of COTS and 3rd Party Components takes into account the possible risks of obsolescence, interrupted maintenance and compatibility (See Table 4). The EVER-EST architecture allows the easy replacement of components that could prove not usable anymore.
H2020 – EINFRA – 2015 – 1 Page 31 of 56
5 Deployment Risks Risks that may be encountered throughout the project lifecycle can encompass many areas: technical, financial, management and staffing. The following table gathers the risks related to the technical development as foreseen at the moment. As a starting point the technical risks addressed in DOA are reported and updated according to the current status of the project.
The table describes the preventive actions put in place to minimise the risks and the contingency plan in case a risk becomes a reality.
.
ID Description Probability Impact Time Response
TR1 Software fails to deliver expected functionalities.
Low High M12-M36
The development team works in close cooperation with the WP3 and provides constant updates on progress of work. Specific validation activities are foreseen to ensure the early checking of system behaviour by target users. In case of discrepancy between expected and actual functionalities the technical team is engaged during the whole operations phase in bug fixing and evolution.
TR2 Difficulties in accessing external data/tools
Low High M12-M36
Selection of external data/tools carried out with strong focus on feasibility and reliability of sources.
Constant monitoring on accessibility.
Technical alternatives will be pursued in case of problems.
TR3 Technology changes require redesign
High Medium M12-M36
The SOA paradigm contributes to the minimisation of consequences, since the architecture is based on services more than on specific components. Re-design would mostly imply the use of alternative tools to implement the same service.
TR4
Time for development, integration and deployment is underestimated.
High Medium M12-M36
WP3 has been involved since the beginning in all technical discussion to minimise this risk.
The technical team performs constant monitoring of technical results achieved.
In case of delay activities are prioritised and proper recovery activities (crashing and/or fast-tracking) will be put in place.
TR5 Tools cannot interoperate
Low High M12-M36 The technical choice make this possibility very unlikely. ESB should be used to overcome extreme cases.
TR6 Difficulty in adapting and integrating existing components
Low High M12-M24 Techncial difficulties were encountered in ROHub Notifications and Workflow Runner API. Ad-hoc technical meetings were put in
H2020 – EINFRA – 2015 – 1 Page 32 of 56
ID Description Probability Impact Time Response
place to address the issues.
TR7 Security issues related to M2M communications
Low High M12-M14
Specific study has been carried out. The outcome shows that the adoption of the Oauth-2 protocol grants full security for M2M communications. However, should problems arise, an alternate solution based on a technical patch has been identified.
The problem has been fixed and closed.
TR8
Resources made available by PSNC for deployment prove not sufficient
Low Medium M9-M36
Resources have been secured since early phases of the project. In case of further needs (that cannot be satisfied by PSNC resources) alternate solutions can be identified within or outside the consortium.
TR9 Seafile not working with enough performance
Low Medium M9-M36
Experimentation carried out so far shows high reliability and good performances. Other open source storage solutions can be deployed in case of failure.
TR10
Development of ROHub backend takes longer than expected
Low Medium M9-M24
Development is closely monitored and is following the foreseen roadmap that put at the beginning the APIs with higher priority. Main APIs are already developed and the last ones can be neglected or replaced by other tools.
TR11
ESB usage as a middleware to ROHub back-end proves not possible
Medium Low M12-M24
A specific study is on-going. Alternate technical solutions have been assessed as feasible.
TR12
Problems in integration of workflow management tools
Medium Medium M12-M24
Accurate experimentation during development and integration phases are undertaken to identify possible shortcomings. Several alternative solutions have been identified in D5.1. The current assumption is to use Taverna, which is largely known by many of the user communities involved, and should be put in a VM. Other solutions can be put in place if Taverna proves not performing or not reliable in a virtualised environment. The use of Taverna as a standalone tool can be considered as the ultimate backup solution.
This solution will be put in place if the nominal Taverna solution is not feasible.
TR13
Problems in interfacing non-OpenSearch catalogues
Medium Medium M12-M24
The current assumption is to use OpenSearch as the protocol to search geo-spatial catalogue. Some of these catalogues could be not compatible. If this proves true
H2020 – EINFRA – 2015 – 1 Page 33 of 56
ID Description Probability Impact Time Response
for fundamental catalogues several responses are possible (in decreasing order of preference:
1. Use ESB as a translator/adapter 2. Create a converter 3. Modify the catalogue
Avoid using that catalogue
TR14 Compatibility of co-existing components
Low Medium M12-M24
Continuous monitoring is carried out to highlight possible compatibility issues. Alternate equivalent components or adaptation of the components will be done, as needed.
TR15 Possible evolution of licensed software
Low High M12-M24
External SW components are mainly Open Source. In case of interrupted maintenance of a proprietary SW, we will evaluate the adoption of an equivalent open version or other workarounds.
Table 4 EVER-EST Risk Table
H2020 – EINFRA – 2015 – 1 Page 34 of 56
6 Smart Objectives and KPI Compliance
6.1 Smart objectives
The following table contains the SMART Objectives that are addressed by this deliverable. Beyond the description taken from the applicable DOA, the compliance status and the reference to the involved section of the document are reported.
SM_OB#1.1 Deploy a VRE based on a Service Oriented Architecture (SOA) tailored for Earth Science
Measured by Adherence to SOA standards; services usability outside the VRE for custom clients.
Achievable Take-up and adaptation of web services components developed in previous projects and
development of new ones.
Relevant SOA architecture is crucial for a dynamic and scalable system
Timely VRE architecture and interfaces will be defined by M4 and deployed at M22
Compliance The document complies with the objective because all implementation and integration
activities are carried out in Service Oriented Architecture. Several components come
from previous projects and are easily inserted in the EVER-EST system. All interfaces have
been defined in the design phase and applied in integration activities.
Traceability §2.7, §2.8
SM_OB#1.2 Deploy services to discover, access & process heterogeneous Earth Science (ES)
datasets
Measured by VRCs capacity to discover, access and process ES datasets reported in WP3 deliverables.
Achievable Query/Access/Process components for heterogeneous ES data available through EVER-
EST building blocks components
Relevant Heterogeneous data query, access and processing is vital for the VRE adoption by ES VRCs
Timely Services deployed in their final version in M22
Compliance The system, as described in this deliverable, is designed to manage datasets from 4
different ES use cases. The VRCs Portals are specifically designed for these different
communities.
Traceability §2.8.1, §2.8.2
SM_OB#1.3 Deploy services to capture and store research activities workflows, processes and
results
Measured by VRCs capability to manage the full research activity workflow lifecycle
Achievable Using the innovative research objects paradigm adapted to Earth Science
Relevant Research workflow lifecycle mgt. is key to create new knowledge re-using pre-existing
H2020 – EINFRA – 2015 – 1 Page 35 of 56
research results and knowledge
Timely research object and ROHub finalized in M12; integrated in M22
Compliance The system, as described in this deliverable, allows to store and retrieve workflows and
data of the different VRCs. The RO paradigm is at the heart of the system.
Traceability §2.7, §2.8.2, §2.8.7
SM_OB#1.4 Deploy services to share data, models, algorithms, results within and across
communities
Measured by Capacity to share models, data and knowledge across communities measured by
feedback
Received from VRCs and reported in WP3 deliverables
Achievable research objects and Preservation services to allow resource sharing (
Relevant The improvement of data and knowledge sharing capabilities is the core objective of the
EVER-EST VRE concept and design
Timely research object study finalized in M12; integrated in VRE final version in M22
Compliance Data and information sharing across communities is one of the features showed by the
system, as described in this deliverable. Specific sharing capabilities (for live data) are
addressed by the E-Collaboration tools. ROs support the longer term sharing of
experiments and results.
Traceability §2.8.2, §2.8.6
6.2 KPIs
KPIs require the system to be in operation to be measured. The compliance to KPIs is thus postponed to a successive phase.
H2020 – EINFRA – 2015 – 1 Page 36 of 56
7 Report on Integration and Deployment Activities This section describes the integration and deployment activities carried out so far (see § 2) and the current status of deployment.
7.1 Integration and deployment activities
Activities regarding VRE Architecture and Interfaces Definition has been successfully completed.
V1 Service Development has been completed with some exceptions (see details in the following sections about services moved to V2).
V2 Service Development is ongoing and is foreseen to finish within the nominal timeframe.
VRE Deployment needs have been completely identified.
PSNC has made available the HW infrastructure foreseen for the project. This platform is active and used by technical partners. The process of requesting, providing and managing Virtual Machines is running properly.
The analysis of system architecture has been completed, in close cooperation with WP5.
The Integration of V1 components is ongoing.
The integration of some V2 components has started.
VRE population has started in December 2016 and is ongoing.
VRE End-to-End Testing has started in January 2017 and is currently limited to some simple scenarios.
7.2 Authentication (ACS)
Identity Server has been deployed on PSNC machine “VM3” (VM Type 3, see Table 1 Deployment needs for system components).
It has been configured to authenticate:
VRE Portal
ROHub Portal
Collaboration Spheres
Seafile
Terradue Cloud Platform
Authentication for the above applications works properly implementing a SSO.
7.3 Personal file storage (PSNC)
For the personal file storage, PSNC has deployed a Seafile instance for EVER-EST. This instance is running on the same infrastructure used for the provisioning of VMs to EVER-EST, both for the services/applications and for the end-users (VRCs). The service following configurations have been carried out:
Installation and configuration of Shibboleth Service Provider (SP) to enable the integration of the Identity Service deployed for EVER-EST. Hence, EVER-EST users can have access to Seafile with their same credentials.
The initial space allocated for Seafile is 5TB, which can be increased if necessary.
EVER-EST instance of Seafile can be accessed through the service Web Interface, available at http://box.everest.psnc.pl/ , and via the provided Web API available at https://box.everest.psnc.pl/api2 . Additionally, the users can install the Seafile desktop client and get access to their personal file storage space.
Seafile has been integrated with the VRC portals through its API
H2020 – EINFRA – 2015 – 1 Page 37 of 56
[MEEO] solved Seafile access issue
[MEEO] implemented Seafile VRC Visual Components: seafilecontent and seafileuploader
[MEEO] integrated Seafile VCs within the EVER-EST environment and other Visual Components.
7.4 VM management (T2)
Build an EVER-EST reference baseline template, with a common security stack and policies, based on CentOS 7 leveraged by configuration management.
Deployment and configuration of INGV software stack as golden image VM.
7.5 User management (provisioning, de-provisioning, sign-up) (ACS)
Implemented sign-up managed by IS administrator .
Provisioning implemented within Identity Server by the self-sign up functionality and deployed on PSNC machine “VM3” (VM Type 3, see Table 1 Deployment needs for system components).
Provisioning implemented within EVER-EST applications via JIT user provisioning.
De-provisioning not started.
7.6 Auditing (ACS)
Under design.
7.7 Enterprise service bus (ACS)
Enterprise Service Bus has been deployed on PSNC machine “VM2” (VM Type 3, see Table 1 Deployment needs for system components).
Deployed the following EVER-EST APIs:
o Create RO (with/without internal/external resources)
o Add Resources (internal/external)
o Delete Resources
o Get List of RO
o Get List of Resources
o Get List of Resource Metadata
o Delete RO
o Delete Resources
o Publish RO (Snapshot/Archive)
o Add Annotation
7.8 EO data discovery (T2)
Deployment of Sentinel-1 catalogue and store.
Created Sentinel-1 series for selected Volcanoes (in Hawaii, Iceland, Campi Flegrei, Etna).
H2020 – EINFRA – 2015 – 1 Page 38 of 56
7.9 EO data storage (PSNC)
Analysis of the available storage APIs in PSNC: i. S3, provided by the Swift technology
ii. POSIX (provided by the NFS technology, or Ceph technology) 1. Select POSIX using the Ceph technology, with write performance scaling with the file
size
Installation of the storage management on PSNC.
Setup and configuration of the Terradue Data Gateway for the storage management on PSNC.
7.10 E-collaboration (MEEO)
eJabberd XMPP server and JSXC XMPP client updates.
eJabberd XMPP server configured on vm1.everest.psnc.pl.
django-postman, django-machina updates.
integrated django-postman and django-machina into EVER-EST environment.
7.11 Cloud services (T2/PSNC)
PNSC integration with the Terradue Cloud Controller through Openstack within the jclouds driver.
Build an Ever-EST Developer Cloud Sandbox template, leveraged by configuration management.
PSNC manages three pools of resources o One for project services and applications
This pool of resources includes:
64 VCPU (30 used)
128 GB RAM (60 used)
1280 GB disk space
15 TB external disk space (volume) (7 used) Currently we have 7 VMs (CentOS, Ubuntu, W7pro)
o One for T2 cloud controller for the provisioning of VM to VRCs. This pool of resources includes:
20 VCPU
50 GB RAM
500 GB disk space
1 TB external disk space (volume) This pool of resources is managed by T2 cloud controller Currently we have golden images for INGV, CNR, and a couple of test images
o One dedicated for ROHUB This pool of resources includes:
32 VCPU
64 GB RAM
640 GB Disk Space
2TB external disk space (volume)
7.12 Excution of remote services (T2)
Support the implementation of a "change detection" processing chain using SNAP GPT in the Terradue Cloud Sandbox.
H2020 – EINFRA – 2015 – 1 Page 39 of 56
Deployment of the Change detection Service on PSNC ICT resources.
7.13 RO integration in VRC portal (ACS, MEEO, PSNC)
Fully integrated the EVER-EST APIs listed in §7.7
[MEEO] integration of RO Middleware APIs into the EVER-EST environment
[MEEO] implemented RO Visual Components: addtoro, removefromro, lastros
[MEEO] extended RO Visual Components functionalities: createro (added annotations)
7.14 Remote workflow execution (PSNC)
Taverna server v2.5.4 has been deployed for the remote execution of workflows in EVER-EST. The server is available at http://sandbox.rohub.org/taverna-server. It is accessible via a Restful API, described at http://dev.mygrid.org.uk/wiki/display/tav250/REST+API
The service is currently being integrated in VRC portals, and later it will also be integrated in ROHUB
7.15 COCONET (MEEO)
Patched EVER-EST OpenSearch interface to manage GeoNetwork OpenSearch incompatibilities.
Tested OpenSearch interface of GET-IT tool (developed in CNR), as GeoNetwork alternative.
Tested an alternative GeoNetwork installation (based on jar) that allow OpenSearch configuration file customization.
7.16 E-learning (T2)
Definition of a base VM template of e-learning services.
Package e-learning modules and create e-learning VM template.
7.17 Preservation (ACS)
Not started
7.18 Digital information services (MEEO)
New template for generic VRE: landing page and generic demo VRC.
VRC portals setup: integration and configuration of Visual Components (catalogues, VRC layouts, …)
7.19 ROHUB (PSNC)
ROHUB backend is accessible from sandbox.rohub.org/rodl/. The APIs exposed by ROHUB are used the middleware RO API, which is used by the VRC portals. The original ROHUB portal is accessible from http://www.rohub.org/, while the new ROHUB portal that is under development is available at http://beta.rohub.org/. ROHUB portal is implementing a wide range of visual components for the creation and management of research objects and exposes the full set of research object capabilities with a more advanced view than the VRC portals. ROHUB index server has been upgraded to v6.4.1
H2020 – EINFRA – 2015 – 1 Page 40 of 56
8 ANNEXES
8.1 ANNEX 1 sequence diagrams
Sequence diagrams used to support the system architecture definition are reported in the following.
Figure 10 CNR Scenario 1 – Habitat suitability model for CWC (D3.1 Appendix A)
H2020 – EINFRA – 2015 – 1 Page 41 of 56
Figure 11 CNR Scenario 2 – The Citizen Science and jellyfish distribution (D3.1 Appendix A)
H2020 – EINFRA – 2015 – 1 Page 42 of 56
Figure 12 CNR Scenario 3 - Trend in the evolution of invasive jellyfish distribution (D3.1 Appendix A)
H2020 – EINFRA – 2015 – 1 Page 43 of 56
Figure 13 CNR Scenario 4 - Habitat suitability model for jellyfish species, study case Pelagia noctiluca and Velella velella (D3.1 Appendix A)
H2020 – EINFRA – 2015 – 1 Page 44 of 56
Figure 14 INGV Scenario 1 - The Mt. Etna ground deformation investigation (D3.1 Appendix A)
H2020 – EINFRA – 2015 – 1 Page 45 of 56
Figure 15 NHP Scenario 1 - Surface water flooding hazard impact modelling case studies (D3.1 Appendix A)
H2020 – EINFRA – 2015 – 1 Page 46 of 56
Figure 16 SatCen Scenario 1 – Land monitoring (D3.1 Appendix A)
H2020 – EINFRA – 2015 – 1 Page 47 of 56
Figure 17 RO Re-using Scenario (generic scenario to show a common functionality)
H2020 – EINFRA – 2015 – 1 Page 48 of 56
8.2 ANNEX 2 components’ delivery schedule
The rows with background in grey indicate components delivered in the final VRE.
Component
Interfaces
Owner
(server
side)
Consumers
(client side) Description Deadline for delivery
VRC Portals
Authenticatio
n Demo
ACS
Login_request ACS MEEO,
ROHub
T2, Seafile
and
CollabSphe
re (new
clients)
User inserts
credentials
(username/passw
ord) and login in
the community
portal
01/09/2016
Completed
(Server)
T2 and CollabSphere
to be verified
Logout_Request ACS MEEO,
ROHub
T2, Seafile
and
CollabSphe
re (new
clients)
User explicitly
ends its session
on the portal
01/09/2016
Completed
, endpoint to be
provided to clients.
Non-standard API
under evaluation.
Personal File
Storage
PSNC
Store_Data PSNC ACS
MEEO
User wants to
store file on
personal storage
space
13/02/2017
Retrieve_Data PSNC ACS
MEEO
User wants to
retrieve file from
personal storage
space
13/02/2017
H2020 – EINFRA – 2015 – 1 Page 49 of 56
Component
Interfaces
Owner
(server
side)
Consumers
(client side) Description Deadline for delivery
Access_Data_Discovery PSNC ACS
MEEO
User wants to
search files
stored on their
personal storage
space
13/02/2017
VM Discovery
and
Deployment
T2
List_Virtualised_Applicatio
n
PSNC T2 User wants to list
all the virtualised
applications
available for their
profile
07/02/2017
Completed
Start_Virtualised_Applicat
ion
PSNC T2 User wants to
start a virtualised
application
07/02/2017
Completed
Stop_Virtualised_Applicati
on
PSNC T2 User wants to
stop a virtualised
application
07/02/2017
Completed
Interactive_Session PSNC T2 User wants to
have an
interactive
graphic session in
the virtualised
application
07/02/2017
Completed
Link_RO-Portal PSNC MEEO User want to
open link of RO in
the ROHub portal
from the VRC
portal
Completed
Identity Mgt.
H2020 – EINFRA – 2015 – 1 Page 50 of 56
Component
Interfaces
Owner
(server
side)
Consumers
(client side) Description Deadline for delivery
User
Management
ACS
User_Provisioning ACS PSNC,
MEEO, T2
Administrator
wants to create a
User, this means
that an user
register by ACS
will be able to
login in ROHub
01/09/2016
Provisioning by
Administrator
Completed.
User_De-Provisioning ACS PSNC,
MEEO, T2
Administrator
wants to remove
a User
28/04/2017
Self_signup ACS MEEO User wants to
register
07/02/2017
Completed
Auditing
(Data
Analytics)
ACS Start a process to
understand how
Analytic Server of
WSO2 can be
used.
01/01/2017
Process started.
Aggregating data ACS MEEO Administrator
wants to collect
data streams for
further analysis.
28/04/2017
Analyzing data ACS MEEO Statistical
analysis (to be
applied to KPIs)
28/04/2017
Presenting data (show
dashboard)
ACS MEEO Administrator
wants to show
analysis results
28/04/2017
Enterprise
Service Bus
Transport protocol
management
ACS ALL 15/12/2016
Completed
Format management ACS ALL 15/12/2016
Completed
Routing ACS ALL 15/12/2016
Completed
H2020 – EINFRA – 2015 – 1 Page 51 of 56
Component
Interfaces
Owner
(server
side)
Consumers
(client side) Description Deadline for delivery
Mediation ACS ALL 15/12/2016
Completed
Transformation ACS ALL 15/12/2016
Completed
Data Discovery and Storage
Discover_EO T2 MEEO User select
search
parameters,
executes
requests and
visualizes results
(on a map) and
select products
28/11/2016
Completed
EO Data
Discovery
T2
Discover_EO_Extended T2 MEEO ok 31/01/2017
Completed
Environmental data (TBC) T2 MEEO TBD For now
download from
3rd party and
store_data
VERIFY STATUS
19/05/2017
In-Situ (TBC) T2 MEEO TBD for now
download from
3rd party and
store_data
VERIFY STATUS
19/05/2017
Data Storage Store_Data (EO data
cache)
PSNC ACS repeated from
VRC portal
8.2.1.1 19/05/2017
Retrieve_Data (EO data cache)
PSNC ACS repeated from VRC portal
19/05/2017
8.2.1.2
Access_Data_Discovery (EO data cache)
PSNC ACS repeated from VRC portal
19/05/2017
8.2.1.3
e-Collaboration Tools
H2020 – EINFRA – 2015 – 1 Page 52 of 56
Component
Interfaces
Owner
(server
side)
Consumers
(client side) Description Deadline for delivery
e-
Collaboration
Notify_Users MEEO MEEO User wants to be
notify of some
event (TBC)
19/05/2017
Peer_Review MEEO MEEO Priority to be re-
defined.
19/05/2017
Instant_Messaging MEEO MEEO 01/12/2016
Completed
8.2.1.4 Cloud Management
Cloud_Platfor
m
Define_Cloud_Service T2 Communiti
es
Developer wants
to integrate
application in the
developer cloud
sandbox
07/02/2017
Completed
(Server)
Clients are testing
28/04/2017
Integration of
Cloud Service
T2
Update_Cloud_Service T2 Communiti
es
Developer wants
to change/alter
application in the
developer cloud
sandbox
07/02/2017
Completed
(Server)
Clients are testing
28/04/2017
Execute_Cloud_Service T2 MEEO User wants to
execute
application with a
given set of
parameters,
obtain live status
and retrieve
results
07/02/2017
28/04/2017
RO Management
RO
integration in
VRC portal
MEEO
Create_Live_RO PSNC MEEO, ACS,
ESI
User wants to
create a Live RO
01/09/2016
Completed
Update_Live_RO PSNC MEEO, ACS,
ESI
User want to
update a Live RO
but doesn't want
to release it yet
28/04/2017
H2020 – EINFRA – 2015 – 1 Page 53 of 56
Component
Interfaces
Owner
(server
side)
Consumers
(client side) Description Deadline for delivery
Take_snapshot
PSNC MEEO, ACS,
ESI
Produce an
intermediary
release of the
experiment to
share within
restricted group
01/09/2016
Completed
Publish_RO ESI,
PSNC
MEEO, ACS Release the
current state of a
scientific
investigation for
other community
members to
reuse and cite. It
involves
generating an RO
snapshot,
generating a DOI
which replaces
the temporary
identifier of the
RO and updating
the metadata
related to the RO
lifecycle state
through the
ROEVO model
01/09/2016
Completed
Load_RO PSNC,
ESI
MEEO, ACS Get a research
object details
01/09/2016
Completed
Show_RO_history PSNC,
ESI
MEEO, ACS Get the evolution
information.
Provenance info
may be involved
here
28/02/2017
Completed
H2020 – EINFRA – 2015 – 1 Page 54 of 56
Component
Interfaces
Owner
(server
side)
Consumers
(client side) Description Deadline for delivery
Define_RO_Access_Rights PSNC,
ESI
MEEO, ACS Specify who has
read or write
access, and the
research object
access mode
(open/public/priv
ate)
01/09/2016
Completed
Save RO state (client side,
Archive functionality and
Snapshot functionality)
PSNC,
ESI
MEEO, ACS During a
research,
scientist wants to
"save" the steps
performed. ---
Isn't this the
same as update
RO? What kind of
steps you mean
and in which
form?
28/02/2017
Completed
Save RO state (client side,
Branch functionality)
PSNC,
ESI
MEEO, ACS During a
research,
scientist wants to
"save" the steps
performed. ---
Isn't this the
same as update
RO? What kind of
steps you mean
and in which
form?
28/04/2017
Discover and search RO ESI,
PSNC
MEEO Intuitive and rich
interfaces for
finding research
objects based on
different criteria
28/11/2016
Completed
H2020 – EINFRA – 2015 – 1 Page 55 of 56
Component
Interfaces
Owner
(server
side)
Consumers
(client side) Description Deadline for delivery
Monitor RO quality PSNC,
ESI
MEEO Definition of
checklists for
measuring the
quality and the
requirements
that need to be
addressed by a
specific
investigation,
generating
notifications
upon changes
and providing the
user with wizards
to produce these
checklists. Typical
interaction with
user interface will
display current
quality status and
notification panel
19/05/2017
Workflow Management
Start_Workflow ESI PSNC,
MEEO,
ACS
New workflow
execution.
Workflow Runner
superseded.
19/05/2017
COCONET-DB CNR-
ISMAR
MEEO COCONET
reachable from
VRC in
OpenSearch
01/03/2017
Completed
E-Learning Management
E-Learning Develop_eLearning_Modu
le
GitHub T2 Developer a new
eLearning
Module
01/12/2016
Completed
H2020 – EINFRA – 2015 – 1 Page 56 of 56
Component
Interfaces
Owner
(server
side)
Consumers
(client side) Description Deadline for delivery
Register_eLearning_Modu
le
GitHub PSNC Developer wants
to register an
eLearning
Module (e.g. one
github repository
with several
notebooks) as a
RO.
01/12/2016
Completed
List_eLearning_Module PSNC MEEO
(TBC)
User wants to
discover available
eLearning
Modules
registered as RO
19/05/2017
Instantiate_eLearning_Mo
dule
T2 MEEO
(TBC)
User wants to
instantiate a VM
with a eLearning
module
containing a set
of Notebooks and
associated
resources (e.g.
data)
19/05/2017
Update_eLearning_Modul
e
GitHub T2 Developer wants
to
improve/maintai
n the eLearning
Module
01/12/2016
Completed
Preservation Management
Preservation Save RO state
(preservation side)
PSNC MEEO, ACS
+ ESI +
Communiti
es
Knowledge data
provided by
researchers shall
be properly
linked and
compressed in a
RO
29/05/2017
![Ugo Santamaria [ePub]](https://img.dokumen.tips/doc/110x75/617819796b211c0c2f56a846/ugo-santamaria-epub.jpg)
