SCI-BUS is supported by the FP7 Capacities Programme under contract no. RI-283481

Science Gateways for improving the uptake of Grid and Cloud Computing Infrastructures

Tamas Kiss, Stephen Winter, Gabor Terstyanszky, Thierry Delaitre University of Westminster

Wibke Sudholt, CloudBroker GmbH

Peter Kacsuk, Zoltan Farkas, SZTAKI

Jens Krüger, Sandra Gesing, University of Tübingen

Using Distributed Computing Infrastructures is not so easy

gsissh -p 2222 grid-compute.oesc.ox.ac.uk globus-job-submit grid-compute.oesc.ox.ac.uk/jobmanager-

pbs  -q RXXXXX /bin/hostname -f globus-job-status <uid> globus-job-get-output <uid> globus-job-clean <uid>

?!or is it?

• SCI-BUS is a 3-year EU FP7 project• Started on 1st October 2011• Build around 30 science gateways to various user

communities in Europe• 10 academic and 5 commercial partners from 10

European countries.

SCIentific Gateway Based User Support

Main objectives of SCI-BUS

• Create a generic-purpose gateway technology • Elaborate an application-specific gateway building technology

and customisation methodology• Establish production gateway services both for NGIs and

various user communities• Provide seamless access to major computing, data and

networking DCIs and services including supercomputers, clusters, grids and clouds

• Create and maintain a Liferay portlet repository that enables the quick creation of user specific customised gateways

• Provide gateway development and application development support

• Develop business models to guarantee the sustainability of the gateway services and to enable the commercial exploitation

Value brought by the project

SCI-BUSGeneric-Purposegateway

Helioport EPOS MoSGrid BioGrid

Blender ETICS e-Group SimSoft

AMChospital

PireGrid

SHIWAWF

repository

EGI/EDGIAppl.

repositories

SCI-BUSAppl.

repositoryHU

UK

D

TU

B

CH

BG

IR

SP

NGI GatewaysLiferayPortlet

repository

SimSoft

Customized gateways

SCI-BUS will serve large number of user communities by providing generic-purpose and customized scientific gateways for them.SCI-BUS will also create and provide access to many different portlet and application repositories to help user communities in sharing and running these applications via the SCI-BUS gateways

Project partners:• International seismology community • Helio-physics community• Swiss systems biology community of the SystemsX.ch project• German MoSGrid computational chemistry and bioinformatics community • Biomedical researchers community of the Academic Medical Centre of the

University of Amsterdam• Astrophysics community• PireGrid SMEs community• Business process modelling community involving a wide range of areas such as

finance, healthcare, government, production, robotics and emergency• Blender rendering community• Citizen web-2 community• Public application developer community

Subcontractors:• 6 sub-contractors to be selected representing additional user communities• Hugely competitive – over 3 times as much applications

Additional user communities supported by the project

The SCI-BUS user communities

The gateway services will enable unified access to all the major DCI infrastructures used in Europe:• Local and remote clusters accessed by queuing systems (PBS,

LSF, Condor, SGE)• Local and remote supercomputers• Local and volunteer Desktop Grids (BOINC, XtremWeb,

OurGrid)• Grids (ARC, gLite, Globus, UNICORE)• Academic Public and Private Clouds (OpenNebula,

Eucalyptus, etc.)• Public Commercial Clouds (Amazon EC2, IBM Cloud, etc.)

The SCI-BUS architecture

8

The SCI-BUS Infrastructure

CloudBroker Platform

SCI-BUSScienceGateways

API

App Storefor Cloud

Platformto accessapps inClouds

CloudProvider

AraGrid (Spain), Armenian Grid, Baltic Grid, Belgian GridBIFI Desktop Grid, Bulgarian Grid, ClGrid (Chile), COMPCHEM VO of EGEE

Croatian Grid, EGRID (Economy Grid, Italy)GILDA training grid,Grid Ireland, HunGrid (Hungarian National Grid)

IberGrid (Portugal and Spain), MathGrid (Spain)MoSGrid (Molecule Simulation Grid of D-Grid), KnowledgeGrid Malaysia

PireGrid (Spain and France), See-Grid (South-East European Grid)SwissGrid, Turkish Grid, UK NGS, UK White Rose Grid

VOCE (Central European Grid), Westminster Desktop Grid

Clusters: PBS, LSF, Condor, SGE

...

Liferay and P-Grade

DesktopGrids:

BOINC, Xtrem-Web, OurGrid

Super-computers

WebService / API

Eucalyptus OpenNebula

gLiteMiddleware

UNICOREMiddleware

ARCMiddleware

App 1 App 2 App 3

SGIAmazo

nEC2

IBM

...

...Private Clouds

API API API API

Commercial Components

SCI-BUSgenericgateway

Germ

an M

osGr

id

Com

mun

ity

Stati

stica

l Sei

smol

ogy

Com

mun

ity

Blen

der R

ende

ring

Com

mun

ity

Amst

erda

m M

edica

l Ce

nter

Com

mun

ity

Swiss

Pro

teom

ics

Com

mun

ity

Astr

ophy

sics

Com

mun

ity

Helio

phy

sics

Com

mun

ity

Busin

ess P

roce

ss

Com

mun

ity

Softw

are b

uild

and

test

Com

mun

ity

Citiz

en W

eb

Com

mun

ity

Pire

Grid

Com

mer

cial

Com

mun

ity

The baseline technologyP-GRADE Portals around the globe

– National Grid portals (UK, Ireland, Croatia, Turkey, Spain, Belgium, Malaysia , Kazakhstan, Switzerland, Australia, ClGrid, etc.)

– HP-SEE/SEE-GRID-SCI, BalticGrid– US Open Science Grid, TeraGrid– IDGF (International Desktop Grid Federation)– SHIWA Simulation Platform– Economy-Grid, Swiss BioGrid, Bio és

Biomed Virtual Organisations, MathGrid, etc.

HP-SEE

P-GRADE Portal software family

P-GRADE Portal2.4

NGS P-GRADE portal

P-GRADE Portal2.5

P-GRADE Portal2.8

P-GRADE Portal2.9.1

WS-PGRADE Portal 3.1

WS-PGRADE Portal 3.2

GEMLCAGrid Legacy Code Arch.

GEMLCA, storage concepts

Basic concepts

2008

2009

2010 WS-PGRADE Portal 3.3

P-GRADE Portal2.10

Open source since 2008

Open source since 2011

2011 WS-PGRADE Portal 3.4

Motivations of creating WS- PGRADE/gUSE

• To overcome (most of) the limitations of P-GRADE portal:• To provide better modularity to replace any service• To improve scalability to millions of jobs• To enable advanced dataflow patterns• To interface with wider range of resources• To separate Application Developer view from Application

User viewWS-PGRADE (Web Services Parallel Grid Runtime and Developer

Environment) and

gUSE (Grid User Support Environment) architecture

WS P-GRADE in a nutshell• General purpose, workflow-oriented portal. Supports the

development and execution of workflow-based applications• Based on Liferay• Services supported by the portal:• New functionalities

– Web services– DB connectors– Embedded workflows– Job level PS– Conditional jobs– Recursive graph– Multi-generator– Multi-collector– CROSS product PS– DOT product PS

Basic Services EGEE grids

(LCG2,GLite)

Globus grids

(GT2,GT4)

Desktop

grids

clusters

Job executionFile storageCertificate managementInformation systemBrokeringJob monitoringWorkflow & job visualization

Inside of the System gUSE tires

WFGraph editor

WEB-UI(HTML)

Liferay

WS-PGRADE portal

Information System

WF Storage

File Storage

Application Repository

WF Interpreter

Backend

local submitter

GT4 submitter

Glite submitter

LCG submitter

GT2 submitter

GEMLCA submitter

WS (Axis) submitter

Glite GridGlite

Grid

GMLCA GridGMLCA

Grid

GT2 GridGT2 Grid

GT4 GridGT4 Grid

LCG GridLCG

Grid

WSWS

DCI-B

ridge

WS P-GRADE workflow concept

Concrete WorkflowAlgorithms,Resource references,Inputs

Graph

Jobs,Edges,Ports

Template

Constraints,Comments,Form Generators

Workflow Instance

Running state,Outputs

Repository Item

Application ORProject OR,Workflow part(G,T,CW)

Legend:a b a must reference ba b a may reference b

ASM Application Specific Module

• A new module called “Application Specific Module” has been developed for gUSE that ease the creation of Application Specified Portlets – The module provides programming API to control applications developed

using gUSE– The module provides programming API to access inner gUSE services

• Any kind of Graphical interface can be developed on top of this module – GUI written in JSP, JSTL (Java Servlet Tag Library ) and Java– GUI can be customized for the grid application– GUI can be customized for end users

ASM - Application Specific Module

DCI systems

Grid Application Developer creates own Grid Application via ws-PGrade and export it to the local repository

gUSE

Repository

ws-PGradeGrid Application

Submitter

Workflow Interpreter

Workflow Storage

Storage

Information User spaceGrid Application

Then the Grid Portal Developer creates a new web-application (mainly an web-interface) which fits to the user community's needs, and connects it to the gUSE using ASM API

Developed web-interface

ASM

Finally the end-users create own application from the exported one, parameterize and execute it using the developed portlet

<< import >>

Grid Application

CloudBroker Platform• Easy, scalable, secure, integrable and pay-per-use access to scientific

and technical applications in the cloud• High performance computing application store and marketplace

with accounting and billing• Deployment and execution of applications in the cloud with the

same parameters and files as for local execution• Accessible through the internet via a web browser UI• Integration possibilities into third party tools via web service-based

APIs• Public version available under https://platform.cloudbroker.com

• Using infrastructure as a service (IaaS) from cloud providers• Offering platform as a service (PaaS) for software vendors• Providing software as a service (SaaS) to end users

CloudBroker GmbH

Platform Architecture

CloudBroker GmbH

CloudBroker Platform

AmazonCloud

IBMCloud

…Cloud

ChemistryAppli-

cations

BiologyAppli-

cations

MedicineAppli-

cations

WebBrowser

UI

Engineering Appli-

cations

Web Service API

GenericWorkbenches CloudBroker Integration

Domain-Specific Gateways

R&D End Users and Software Vendors

CLI

…Appli-

cations

CloudBroker GmbH

CloudBroker Business Model• Free access to the platform

after registration• Each organization manages

its own users• Prepayment or monthly

billing

• Platform usage costs =resource usage costs +software usage costs + CloudBroker usage surcharges

CloudBroker

EndUsers

SoftwareVendors

CloudProviders

Resources Applications

Usage $

$ $

CloudBroker GmbH

Clouds and Applications

• Currently supported cloud infrastructures:– Amazon Web Services– IBM SmartCloud

Enterprise• Adapters to further

clouds planned• Open source-based

private cloud tools will be interfaced within SCI-BUS

• Current applications:– Chemistry (GAMESS,

Gromacs)– Biology (BLAST, X! Tandem,

Rosetta)– Medicine (AutoDock)– Engineering (OpenFOAM)

• Further applications in preparation

• Own applications can be easily added by users

CloudBroker Platform Integration in Liferay/WS-PGRADE/gUSE

Gamess GromacsAutodock Open FoamBlast RosettaXtandem ...

API

gUSE

WS-PGRADE Job Configuration Portlet CloudBroker UI Portlet

UI

DCI-BRIDGE

SaaS (Softwares as a Service)

CloudBroker plugin

Expose the CloudBroker UI through WS-PGRADE interface by proxy.

User Mapping Portlet

CB Java API

• Integrate the SaaS capabilities of CB into the gUSE/WS-PGRADE system.• Enable WS-PGRADE users to run softwares (SaaS) exposed by the CB platform.

MosGrid Science gateway for molecular simulations

• Requirement: Workflow enabled grid portal for UNICORE based on Liferay

• Solution:– Adaptation of WS-PGRADE from

GridSphere to Liferay– Development of UNICORE

Submitter– Support for SAML– Development of ASM

(Application Specific Module) support

MosGrid Science gateway - architecture

MosGrid Science gateway - architecture

MosGrid Science gateway – application domains

• Quantum Chemical Calculations– based on approximated solutions of the Schrödinger equation– Average scalability– Gaussian, NWChem, Turbomole, …

• Molecular Dynamics– based on forcefields describing molecular interactions– Good scalability– Gromacs, NWChem, Amber, …

• Docking– Based on simplified forcefields– Excellent scalability– CADDSuite, FlexX, …

MosGrid Science gateway – application domains – molecular dynamics

SCI-BUS is supported by the FP7 Capacities Programme under contract no. RI-283481

Thank you for your attention!

Any questions?

http://www.sci-bus.eu/