Introduction to Grid Technologies - westgrid.ca · Command line tools: globusrun-ws. WestGrid...

Introduction to

Grid Technologies

Cameron Kiddle

Grid Systems Architect, WestGrid

Research Fellow, Grid Research Centre (GRC)

WestGrid Seminar Series Feb. 6, 2008

Introduction to Grid Technologies - 2

Presentation Outline/Goals

� Provide an overview of grid computing

� Introduce various grid computing

technologies

� Identify the grid computing technologies

currently supported by WestGrid

� Demonstrate how researchers can benefit

from use of these technologies via examples

What is Grid Computing?

� Many different definitions/uses� computational grids, data grids, desktop grids, campus

grids, sensor grids, access grids

� Coordinated sharing of resources that can span multiple administrative domains

� Related terms� utility computing, computing on demand, cloud computing,

e-Science, e-Infrastructure, cyberinfrastructure

Grid Computing Goals

� Accessibility� Providing users with easier access to more resources

� Collaboration� Enabling large scale collaborations

� Utility� Providing on demand access to computing resources

similar to public utilities such as electricity or water

� Transparency� Providing users access to computing resources without the

need to know how or where computations are taking place

Virtual Organization (VO)

� a group of people, typically spanning institutional

and regional boundaries, that share resources to

collaborate on a common project

Resources Shared by

Virtual Organization X

Resources Shared by

Virtual Organization Y

Domain A

Domain B Domain C

Example Grid Projects

development of standards and infrastructure to share and

analyze astronomical archives from around the world

International Virtual

Observatory Alliance (IVOA)

http://www.ivoa.net/

development of a grid-enabled biomolecular simulation

database to make results more accessible to the biological

community

BioSimGrid

http://www.biosimgrid.org/

a US national network of 15 facilities to study the impact of

earthquakes on buildings, bridges, etc.

Network for Earthquake

Engineering Simulation (NEES)

http://www.nees.org/

data storage and analysis infrastructure for the high energy

physics community using the Large Hadron Collider (LHC) at

CERN (ATLAS Tier-1 site at TRIUMF in British Columbia)

LHC Computing Grid

http://lcg.web.cern.ch/

DescriptionName

Grid Technologies

� Grid Middleware

� Security

� VO Management Services

� Information Services

� Data Management Services

� Execution Management Services

� Web Portals/Scientific Gateways

� Virtualization

Grid Middleware

� The layer between users/applications and grid resources that glues everything together

� Example grid middleware� Globus Toolkit (GT)

� GT2 – pre-standards

� GT4 – standards based (Web Services)

� UNICORE

� gLite

� ARC

� NAREGI

Security

� Authentication� X.509 certificates (IETF)

� Used to identify and authenticate users and services

� Based on public key cryptography

� Issued and signed by a certificate authority

� Provides global name space

� Enables single sign-on

� Authorization� grid-mapfile

� Maps distinguished names (found in certificates) of authorized users to local user names (e.g., unix login)

VO Management Services

� Services for managing membership and roles

within a Virtual Organization

� Helps simplify user account management

� Examples

� VOMS, GUMS, PRIMA

� Shibboleth, GridShib, myVocs

� Other – CAS, Akenti, PERMIS, SHEBANGS

Information Services� Provide information about resources, policy, services and

applications to tools and users

� Information models� DMTF Common Information Model (CIM)

� GLUE Schema

� GRC Model Schema

� Example services� Monitoring and Discovery System (MDS)

� MDS 2 – LDAP based

� MDS 4 (WS MDS) – Web Service based

� Relational Grid Monitoring Architecture (R-GMA)

� Berkeley Database Information Index (BDII)

� Universal Description, Discovery, and Integration (UDDI)

Data Management Services� Data transfer

� GridFTP

� Reliable File Transfer (RFT)

� Data replication � Replica Location Service (RLS)

� Metadata management� Metadata Catalog Service (MCS)

� Higher level data management services� Data Replication Service (DRS)

� Storage Resource Broker (SRB)

� i Rule Oriented Data Systems (iRODS)

� Proactive Data Management System (PDMS)

Execution Management Services� Handle placement, provisioning and lifetime management of

� Job submission� Submission of jobs to different types of resources

� Grid Resource Allocation and Management (GRAM)

� Meta-schedulers� Higher level schedulers that manage and distribute jobs between

different local schedulers

� Examples: Condor-G, CSF, GridWay, Moab Grid Scheduler (Silver)

� Workflow managers� Automate the management and submission of a set of jobs that have

various ordering dependencies

� Examples: DAGMan, Kepler, Triana, Taverna, Pegasus

Web Portals/Scientific Gateways

� Provide Web-based access to computing resources for communities of users

� Web portal development software� WebSphere

� GridSphere

� Web 2.0 technologies� Social networking (Facebook) , wikis (Wikipedia), blogs, …

� Example portals� nanoHUB

� myExperiment

Virtualization

� Can transform a single physical machine into multiple virtual machines (VMs) each with their own OS and software stack

� Virtualization software� Xen, VMWare

� Support allocation, deallocation, suspension and migration of VMs

� Benefits� custom environments (root access), resource

consolidation, system maintenance without disruption

WestGrid and Grid

� Is WestGrid a computational grid?

� Provides grid enabled resources

� Security services

� Data transfer tools

� Job submission services

� WestGrid resources can be part of

computational grids

GT4-based Grid Environment

Grid Services Status

Grid Services Supported in WestGrid

� Security Services

� GSI (Grid Security Infrastructure), X.509 certificates,

GSI-OpenSSH, MyProxy

� Information Services

� WS MDS, WebMDS

� Data Management Services

� GridFTP, RFT

� Execution Management Services

� WS GRAM, Condor-G (by request), DAGMan (by request)

Certificates in WestGrid

� A user automatically receives a certificate

when applying for an account

� Certificate and password protected private

key is stored in users $HOME/.globus/

directory on their home site

� Certificates are issued by Grid Canada

� Certificate must be renewed annually

(users will receive 60 day notice by e-mail)

GSI-OpenSSH

� GSI enabled version of OpenSSH

� Provides a single sign-on remote login and

file transfer service

� Command line tools:

� gsissh – GSI enabled ssh

� gsiscp – GSI enabled scp

� gsisftp – GSI enabled sftp

MyProxy

� Developed by NCSA (National Center for

Supercomputing Applications)

� Credential repository

� Allows proxy credential to be received from any

machine

� Can allow trusted services to renew proxy

credentials

� WestGrid MyProxy Server - myproxy.westgrid.ca

WS MDS

� Web Services version of the Monitoring and

Discovery System

� Index Service� Collects data and provides a query/subscription

interface to the data

� Can create hierarchy of index services

� Trigger Service� Collects data and takes actions based on the data

GRC Model Schema

� Models developed to describe systems, applications and scheduler policy

System Model Class Diagram

WebMDS� Customizable Web based interface for WS MDS information

GridFTP

� Based on FTP (File Transfer Protocol)

� GSI security on control and data channels

� Supports third-party transfers

� Improved efficiency of transfers

� Modification of TCP buffer sizes

� Parallel transfers (multiple TCP streams)

� Striped transfers

� Command line tools: globus-url-copy, gcp

GridFTP Performance

Reliable File Transfer (RFT)

� Manages a set of third-party GridFTP transfers

� Uses a database to checkpoint transfer state

� Recovers from� Source/destination server failures

� Network failures

� Container failures

� Transfers retried with exponential backoff

� Resumes transfers where they left off

� Command line tools: rft, rft_delete

WS GRAM

� Web Services version of the Grid Resource

Allocation and Management protocol

� Provides a single standard interface for remote job

submission and resource management

� Requires users and application developers to learn

only one method to gain access to a large variety of

local management systems

� Command line tools: globusrun-ws

Condor-G

� Developed at the University of Wisconsin-Madison

� An extension of Condor that makes use of Globus

services to submit jobs to different sites

� Matchmaking functionality to match jobs with

appropriate resources

� Available for use in WestGrid by request

DAGMan

� Part of the Condor software

� It manages workflows which are directed acyclic

graphs (DAGs), ensuring that jobs with

dependencies are executed in the correct order

� Available for use in WestGrid by request

Use Case: Life3D Simulations

� Example to illustrate the

benefits of workflow

management

� 3-dimensional version

of The Game of Life

� Workflow includes

simulation, rendering

and animation

Life3D - Workflow - I

Life3D

Simulation

Rendering

AnimationStage

gridstore gridstorelattice

octarine

Life3D - Workflow - II

WestGridGrid Research Center

gridstore (SFU)

lattice (UofC)

octarine

2. Life3D Simulation

4. Rendering5.

6. Animation

Data Storage

Life3D - Technologies Used

� Grid Middleware

� GT2

� Data Management

� GridFTP

� Execution Management

� Job Submission - GRAM

� Meta-scheduler – Condor-G

� Workflow Manager - DAGMan

Use Case: Confederation Bridge ICE

Force Monitoring Project

� Monitoring of forces on the Confederation Bridge

� Data analyzed by civil engineering groups at University of Calgary and Carleton University

� GRC developed solution to automate data management as part of a CANARIE AAP project

(http://www.confederationbridge.com) (http://www.confederationbridge.com)

ICE Force - Technologies Used

� Grid Middleware

� GT4

� Data Management

� PDMS

� Data Transfer - GridFTP, RFT

� Replication Management – RLS

� Metadata Management - MCS

Use Case:

Molecular Dynamics Simulations

� GROMACS

� Parallel molecular dynamics

simulation application

� Can simulate hundreds to

millions of particles

� Simulation runs can take days,

weeks or months

� Issues with long running jobs

� Fault tolerance

� Scheduler policy constraints

(http://moose.bio.ucalgary.ca/)

GROMACS: Grid Enabled Solution

� Automated grid enabled solution developed by

GRC to manage GROMACS simulations as part

of a CANARIE AAP project

� Long jobs split into a series of shorter jobs

� Automates checkpointing, migration and

reconfiguration of jobs

GROMACS: Portal

GROMACS - Technologies Used� Grid Middleware

� GT4

� Information Services� WS MDS

� Data Management� PDMS (GridFTP, RFT, RLS, MCS)

� Execution Management� Custom system (Condor-G, GRAM)

� Portal� GridSphere

Use Case: Fire Simulation� Developed a comprehensive

environment for the Fire Dynamics Simulator (FDS) as part of a collaborative project between GRC and HP Labs

� Deployed on HP Labs Data Centre at University of Calgary

� Initial focus of project� Leverage Web 2.0 technologies

� Explore use of virtualization in a utility computing environment

Fire Simulation - Technologies Used

� User level� Web 2.0 interface (Facebook)

� Service provider level� LAMP environment (Linux, Apache, MySQL,

Perl/Python/PHP)

� Simulation (FDS, Condor)

� Visualization (Smokeview, VNC)

� Resource (utility) provider level� Virtualization (Xen)

Use Case: Ecosystem Modelling

� ecosys

� An application that models ecosystems (agriculture,

forests, savannah, grassland, tundra, desert)

� Used to study ecosystem behavior under different

environmental conditions (Dr. Robert Grant – UofA)

� Individual experiments consist of several hundred

simulations with common and run specific input files

� GRC is developing a portal/experiment engine to

automate execution of experiments as part of a

Alberta Cyberinfrastructure (Cybera) pilot project

ecosys - Technologies Used� Grid Middleware

� GT4

� Information Services� WS MDS

� Data Management� GridFTP, Stork, SRB

� Execution Management� Custom system (Condor-G, GRAM)

� Portal� Web 2.0 based portal (with GSI authentication)

� Virtualization� Xen (in HP Labs Data Centre)

Summary� Grid computing technologies enable sharing of resources

across administrative domains

� They are aimed at improving accessibility to resources, enabling large scale collaborations, providing computing on demand and making access to resources transparent

� There are a large variety of technologies supporting security, VO management, information services, data management, execution management, portals and virtualization

� WestGrid supports various technologies based on the GT4 grid middleware

� Developing grid solutions is not easy but there can be substantial benefits

Contact Information

Cameron Kiddle

kiddlec@cpsc.ucalgary.ca

support@westgrid.ca

(WestGrid) http://www.westgrid.ca/

(GRC) http://grid.ucalgary.ca/

Introduction to Grid Technologies - westgrid.ca · Command line tools: globusrun-ws. WestGrid...

Documents

FlávioSchenkelsbmaonline.org.br/sbma2015/palestras/flavio_schenkel.pdf · Colaboração com o projeto “Genoma de 1000 touros” Processamento Dados brutos de sequência WestGrid

Intro To WestGrid

GRID Leadership_01_The Grid Concept

Job Submission on WestGrid Feb 15 2005 on Access Grid

Introduction to Batch Visualization - WestGrid · Introduction to Batch Visualization Alex Razoumov alex.razoumov@westgrid.ca ... In Python’s matplotlib can script the entire workﬂow

GRID · GRID ... grid

3D graphs with NetworkX, VTK, and ParaView - Compute Canada · Intro Tools Graphs/layouts Selection Statistics 3D graphs with NetworkX, VTK, and ParaView Alex Razoumov alex.razoumov@westgrid.ca

2011-2012 Stakeholder Recap - WestGrid...(ORAN) partners: MRnet (Manitoba), SRnet (Saskatchewan), Cybera (Alberta) and BCNET (British Columbia); and CANARIE, Canada’s Research and

Network Power Users The Case for Jumbo Packets with WestGrid Examples

WestGrid Overview Dr. Rob Simmonds Distributed Systems Architect

Network Power Users The Case for Jumbo Packets with WestGrid Examples

INTRODUCTION TO OPENMP - WestGrid · INTRODUCTION TO OPENMP Hossein Pourreza hossein.pourreza@umanitoba.ca March 26, 2015 Acknowledgement: Examples used in …

2015 Resource Allocation Competition Awards - WestGrid file2015 Resource Allocation Competition Awards Institution Researcher Project title Total allocation core years Total allocation

Grid Smart. Grid Tough

WestGrid / Compute Canada · Introduction to MD simulations Outline: Introduction Basic concepts of Molecular Dynamics Simulations. Examples of Simulations using Molecular Dynamics

AVC: The Access Grid Venue Customizer A joint R&D project by NRC, SFU/Westgrid, CRC Andrew Patrick, NRC Canada Brian Corrie, SFU/Westgrid

WestGrid Handbook for University of Manitoba Researchers

How to Use WestGrid fileWindows Mac Linux OpenSSH . Windows Mac Linux PuTTY ... cluster.srv.ualberta.ca/doc/ Title: PowerPoint Presentation

GRID · PDF fileGRID ... grid

Compute Canada/WestGrid Plans and Updates - BC · PDF fileCompute Canada/WestGrid Plans and Updates Patrick Mann, ... (RPP) a. Up to 3 years. 3. ... TSM: Replication service