Embed Size (px)
Citation preview
Get Data to Computation
eudat.eu/b2stagewww.eudat.eu
B2STAGEHow to shift large amounts of data
Version 4February 2016
This work is licensed under the Creative Commons CC-BY 4.0 licence.Attribution: EUDAT – www.eudat.eu
eudat.eu/b2stage
B2STAGE is…
a reliable, efficient, light-weight and easy-to-use service to transfer research data sets
between EUDAT storage resources and high-performance
computing (HPC) workspaces
2
eudat.eu/b2stage
A truly pan-European Infrastructure
3
EUDAT offers common data services to both research communities and individuals through a network of 35 European organisations.
EUDAT wants to enable European researchers from any discipline to preserve, find, access, and process data in a trusted environment, as part of a Collaborative Data Infrastructure. European infrastructures
Technology ProvidersResearch Communities
eudat.eu/b2stage
Community-Driven Solutions
4
PHYSICAL SCIENCES & ENGINEERING
SOCIAL SCIENCES
& HUMANITIES
MATERIALS & ANALYTICAL FACILITIES
ENVIRONMENTAL SCIENCES
MAPPER
BIOMEDICAL & MEDICAL SCIENCES
EUDAT services are designed, built and implemented based on user community requirements.
eudat.eu/b2stagemove large amounts of data
between data stores and high-performance compute resourcesre-ingest computational results back into EUDATdeposit large data sets into EUDAT resources for long-term preservation
Facilitating communities to:
Features:high-speed transferreliable and light-weightmanages permanent PIDs
6
B2STAGE Features
eudat.eu/b2stage
Why use B2STAGE?
7
Research challenges are getting larger and more complex:
E.g. full-Earth climate simulation, coupled simulations of multiple organs in the human body, seismic analyses of earthquakes at continental scale
High level benefits
Researcher data and compute demands are rising fast
Efficient transfer of data to high performance computing (HPC) workspaces is essential especially in distributed computing, where resources are geographically dispersed
eudat.eu/b2stage
Why use B2STAGE?
8
Facilitates transfer of large data collections from EUDAT storage resources to HPC facilities.
Specific User Requirements
Provides the means to re-ingest computational results back into the EUDAT infrastructure.
Ingests data sets into EUDAT resources for long-term preservation.
Offers reliable, efficient, easy-to-use tools to manage data transfers.
The Data Staging Script is the only tool handling data transfer using PIDs.
9
eudat.eu/b2stage
Who can use B2STAGE?
Researchers can transfer large data collections from EUDAT storage resources to HPC facilities for processing.
Community Managers can replicate community data through a lightweight service and ingest data sets to EUDAT storage resources for long term preservation.
10
eudat.eu/b2stage
How can you use B2STAGE?
EUDAT offers B2STAGE to all registered researchers and interested communities, enabling them to make use of the service to stage data out of EUDAT, and ingest computational results back.
Access to remote HPC facilities should be negotiated and arranged by individual users in parallel.
To help researchers use the B2STAGE service, EUDAT offers documentation, training material and a service helpdesk.
For more information please email: [email protected]
eudat.eu/b2stage
How does B2STAGE work?
12
GridFTP server
iRODS-DSI
User desktop
GridFTP client
data
control
PID Registry
PID
control
HPCGridFTP server
eudat.eu/b2stageUser desktop
How does B2STAGE work?
13
GridFTP client
File systemGridFTP server
iRODS-DSI
PID Registry
PID
data
control
14
eudat.eu/b2stage
B2STAGE User communities
VPH Community ingesting data onto EUDAT resourcesApproximately 12TB will be ingested through this serviceVPH data also replicated between RZG and PSNC sites
B2STAGE will foster the collaboration with EGI and PRACE to develop cross-infrastructure usage:
B2STAGE will be the main service to enable the interoperability of these infrastructures.
Numerous new communities to adopt it as part of the 2015 and 2016 Calls for Collaboration
15
eudat.eu/b2stage
B2STAGE summary
B2STAGE offers:data staging functionalities to easily and efficiently transfer data from EUDAT storage resources to HPC facilitiesa powerful mechanism to ingest data onto EUDAT resourcesa script to facilitate the staging, ingest and retrieval of PID information of transferred data
B2STAGE is unique in handling PIDs for the data
16
eudat.eu/b2stage
Future features
The Data Staging Script will be replaced by a modular and extensible python library which will furnish the users with a programmable interface towards most of the EUDAT services.
eudat.eu/b2stage
17
For more info: http://eudat.eu/services/b2stageUser documentation: http://eudat.eu/services/userdoc/b2stage
Thank you
