Collaborating with e infrastructures for innovative data management solutions

BlueBRIDGE receives funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 675680 www.bluebridge-vres.euBlueBRIDGE receives funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 675680 www.bluebridge-vres.eu

Collaborating with e-infrastructures for innovative data management solutions

Pasquale Pagano (CNR-ISTI)Project technical [email protected]

Understanding how ecosystems of EU e-infrastructures can boost Blue Growth

11 January 2017Brussels, Belgium

2

BlueBRIDGE Services Offer

Tailored services addressing three heterogeneous domainsOne common infrastructure

BlueBRIDGE Infrastructure

"Understanding how ecosystems of EU e-Infrastructures can boost Blue Growth", 11 January 2017, Brussels

3

BlueBRIDGE Infrastructure Services Offer

Data services addressing the whole data management value chainInfrastructure services addressing the access and management of infrastructure resources


4

BlueBRIDGE hides ICT complexities

Storage Database GIS NoSQL

ComputingD4Science EGI SeaDataNet

DataOSCAR Copernicus

BlueBRIDGE


5

BlueBRIDGE hides ICT complexities

Data and Services

Hosted by different Organizations

Accessible through different Protocols

Described with different Metadata

Policies

Different Licenses

Different Terms of Use

Different approaches for Credits

Heterogeneity

Provide Discovery, Access, Use

Support Validation, Curation,

Harmonization

Trace Provenance

BlueBRIDGE provides solutions for dealing with modern science increasingly global, multi-disciplinary and networked


6

BlueBRIDGE realizes an open scientific infrastructure ecosystem

• >800 data providers• >19k temporal datasets• >50k spatial datasets• >350 physical and chemical

variables with global coverage spanning the last ten years

• 40k data analysis/month

Data

• 5 EGI sites• 4 D4Science sites• 4 Organization sites• 1000 CPUs• 400 TB storages• 4 TB Memory

Computing

• >250 services• >180 analytical models• >500 software components

integrated

Services

BlueBRIDGE provides access to Data, Computing, and Services

BlueBRIDGE simplifies the integration of new community servicesthe exploitation of community-specific analytical modelsThe analysis of community/user data "Understanding how ecosystems of EU e-Infrastructures can boost Blue Growth", 11 January 2017, Brussels

7

BlueBRIDGE supports data analytics

• are multidisciplinary, involve members belonging to diverse organisations • require to access data and services that are spread among many providers

Dynamic groups addressing specific questions/problems

• Without relying on pre-organised and costly supporting environments managed by dedicated organizations

BlueBRIDGE builds and operates their own VRE

• BlueBRIDGE allows to operate on remote resources via dedicated interfaces

With limited or no additional costs for the resource providers

Not performed by individuals but group of data analysts


8

BlueBRIDGE supports data analyticsvia sharing of data and workflows

Support collaborative data analysis and experimentation via a virtual workspace

Enable secure and controlled data sharingManage heterogeneous data access policiesImplement Traceability and Reproducibility-Repeatability-Reusability

Information objects can be • files, dataset, workflows, experiments, etc. • organized into folders

Users can • share with selected users• disseminate via persistent public URLs


9"Understanding how ecosystems of EU e-Infrastructures can boost Blue Growth", 11 January 2017, Brussels

Software Integration

Download the (python, R, Java, …) script and the user’s data

Execute script

Collect output

Destroy local copies of I/O and script

Save Output on the User’s Workspace, with provenance info

Scientist provides a method to analyse data

User’s data

BB Infrastructure

10

Collaborative Experiments

WS

Shared online folders

Inputs

Outputs

Results

Computational system

In the e-Infrastructure

Through third party software


11

Enabling New Workflow

Script provider

Updates the script on his private Workspace

The BB service downloadsthe script on-the-fly

A user executes an experiment on his/her data

The output, the input and the parameters can be shared with another user

This user can execute the experiment againand share the computation with other users

1

2

3

4

5

6

7

8

9

5

VRE Social



Access, integrate, process and analyse very heterogeneous resources

• BlueBRIDGE is close to community needs

BB

standard

secure collaboration

Easy-to-useSimplify

Integration

applications

methods

data

PromoteCollaboration

Sharing

Publication

Provenance

Re-use

13

Very different purposes

• BlueBRIDGE delivers tailored environments• Same capabilities, different access policies• Same capabilities, access to different datasets• Different capabilities

• BlueBRIDGE supports all the scientific phases• Testing of new methods and algorithms• Validation of methods and algorithms• Provision of production services



Very different purposesBlueBRIDGE Common Facilities

BlueBRIDGE for Ecosystem Approach to Fisheries

BlueBRIDGE for Aquaculture

BlueBRIDGE for Education


Conclusions• BlueBRIDGE simplifies the integration of user

applications into e-infrastructure

• BlueBRIDGE promotes access to an integrated environment where personal applications are managed as enabling technologies

• BlueBRIDGE delivers a community-cloud where virtual services hide cloud and related technologies

• BlueBRIDGE delivers an open Infrastructure Ecosystem contributing to Blue Growth

BB manages

BB supports

BB operates

Users exploits

PoliciesAuthzToU

TrainingAdoption

Use

MonitoringAccountingProvisionSecurity

Discover AccessRe-useShare

16

Thank YouQuestions?

Technology

Collaborating with e infrastructures for innovative data management solutions