Basic Grid Projects - Basic Grid Projects - GlobusGlobus

Sathish VadhiyarSathish Vadhiyar

Sources/Credits: Project web pages, publications available at Globus site.

Some of the figures were also taken from the same

GlobusGlobus

Open source toolkit used for building GridsOpen source toolkit used for building GridsSoftware forSoftware for Security (GSI)Security (GSI) Information infrastructure (MDS)Information infrastructure (MDS) Resource management (GRAM, job manager, Resource management (GRAM, job manager,

gatekeeper)gatekeeper) Data management (GridFTP, DataGrid)Data management (GridFTP, DataGrid) Communication (Nexus)Communication (Nexus) Fault detection, andFault detection, and Portability Portability

Now moving to web services - OGSANow moving to web services - OGSA

TimelineTimeline

I-WAY experiment – 1994I-WAY experiment – 1994

Formal beginning - 1996Formal beginning - 1996

11stst version – 1997 version – 1997

Version 1.0 – 1998Version 1.0 – 1998

2.0 – 20022.0 – 2002

3.0 – latest3.0 – latest

Show GT2 history powerpointShow GT2 history powerpoint

GT4 Planned architectureGT4 Planned architecture

Grid Security Infrastructure (GSI)Grid Security Infrastructure (GSI)

Supports security across organizations. Supports security across organizations.

Single sign-onSingle sign-on

Delegation of credentialsDelegation of credentials

Digital signatures based on public key Digital signatures based on public key cryptography for verification of messagescryptography for verification of messages

Globus/Grid Security Infrastructure Globus/Grid Security Infrastructure (GSI) based on PKI(GSI) based on PKI

GSI is:GSI is:

PKI(CAs and

Certificates)

SSL/TLS

Proxies and Delegation

PKI forcredentials

SSL forAuthenticationAnd message protection

Proxies and delegation (GSIExtensions) for secure singleSign-on

PKI: Public Key Infrastructure, SSH: Secure Socket LayerTLS: Transport Level Security

Credits: Globus course material

Verification of messages / digital Verification of messages / digital certificatescertificates

Message

Hash(message)

Encyrpted hash

Encypted hash + message

Hash1 = hash(Message)

Hash2 = decrypt hash

If Hash1 = Hash2 ?

GSIGSI

Every resource identified by a certificate.Every resource identified by a certificate.

Certificate provided and signed by CA.Certificate provided and signed by CA.

Certificate = resource identity + public key Certificate = resource identity + public key of resource + certificate authority + digital of resource + certificate authority + digital signature of CAsignature of CA

Uses SSL for mutual authenticationUses SSL for mutual authentication

Parties trust CA’s – possess CA’s public Parties trust CA’s – possess CA’s public keyskeys

Mutual AuthenticationMutual Authentication

I want to communicate. This is my certificate

AB

CA

Did CA sign the certificate or is the certificate tempered? Verify digital signature

OK. CA signed the certificate.

Are you really A or did you steal the certificate from A? Send a random message

Authentication with Proxy and Authentication with Proxy and delegationdelegation

Encrypted file for storing private keys. Needs Encrypted file for storing private keys. Needs passphrasepassphraseProxy and delegation - More convenience and Proxy and delegation - More convenience and less securityless securityAlso for dynamic delegation for third-party Also for dynamic delegation for third-party services and dynamic entitiesservices and dynamic entitiesOwner signs proxy certificateOwner signs proxy certificateProxy’s private key are stored in unencrypted Proxy’s private key are stored in unencrypted files since proxies are for short durationsfiles since proxies are for short durationsChain of trust is establishedChain of trust is established

Mutual Authentication with ProxyMutual Authentication with Proxy

Proxy’s certificate. A’s certificate

A’s proxyB

First validate proxy’s certificate and then owner’s certificate

GSS APIGSS API

GSI implemented using GSS-GSI implemented using GSS-APIAPIGSS API provides both GSS API provides both transport and mechanism transport and mechanism independence.independence.Provides functions for Provides functions for obtaining credentials, obtaining credentials, performing authentication, performing authentication, signing messages and signing messages and encrypting messagesencrypting messagesGSI – X.509 public key GSI – X.509 public key certification, public key certification, public key infrastructure, SSL protocol, infrastructure, SSL protocol, X.509 proxy certificatesX.509 proxy certificates

X.509 Proxy CertificatesX.509 Proxy Certificates

To allow users to:To allow users to: Create identities for new entities dynamically and Create identities for new entities dynamically and

light-weightlight-weight Delegate privileges to those entities dynamicallyDelegate privileges to those entities dynamically Perform single sign-onPerform single sign-on

Proxy certificateProxy certificate Subject name (identity) – scoped by the subject name Subject name (identity) – scoped by the subject name

of the issuer – subject name of the issuer + RDN of the issuer – subject name of the issuer + RDN (Relative Distinguished Name) + serial number(Relative Distinguished Name) + serial number

Public key – different from subject’s public keyPublic key – different from subject’s public key PCI – Proxy Certificate Information – policy method PCI – Proxy Certificate Information – policy method

identifier + policy fieldidentifier + policy field

ProxiesProxies

Single sign-on and ProxiesSingle sign-on and Proxies

Delegation over NetworkDelegation over Network

Globus Resource Allocation Globus Resource Allocation and Managementand Management

Globus Resource Management Globus Resource Management ArchitectureArchitecture

For remote job submission and resource For remote job submission and resource managementmanagement

Designed to address following problems in Designed to address following problems in metacomputing:metacomputing: Site autonomy (resource managers)Site autonomy (resource managers) Heterogeneous substrate (resource managers)Heterogeneous substrate (resource managers) Co-allocation (co-allocators)Co-allocation (co-allocators) Online control (RSL and resource brokers)Online control (RSL and resource brokers)

Resource Management Resource Management ArchitectureArchitecture

Resource Management Resource Management ArchitectureArchitecture

DUROCDUROC

Dynamically-Updated Request Online Dynamically-Updated Request Online CoallocatorCoallocator

coallocatorcoallocator is used to coordinate is used to coordinate transactions with each of the RMs and transactions with each of the RMs and bring up the distributed pieces of the jobbring up the distributed pieces of the job

RSL spec.RSL spec.

E.g.:E.g.:

Multi-request

Local resource Management - Local resource Management - GRAMGRAM

GRAM simplifies the use of remote systems by GRAM simplifies the use of remote systems by providing a single standard interface for providing a single standard interface for requesting and using remote system resources requesting and using remote system resources for the execution of "jobs".for the execution of "jobs".

3 main functions:3 main functions: Processes RSL specificationsProcesses RSL specifications Enables resource monitoring and managementEnables resource monitoring and management Periodically updates MDSPeriodically updates MDS

GRAMGRAM

Provides interfaces to local job scheduling mechanismsProvides interfaces to local job scheduling mechanismsProvides mechanisms to map GSI identities to local user Provides mechanisms to map GSI identities to local user accountsaccountsProcesses the requests for resources for remote Processes the requests for resources for remote application execution, allocates the required resources, application execution, allocates the required resources, and manages the active jobs.and manages the active jobs.Also returns updated information regarding the Also returns updated information regarding the capabilities and availability of the computing resources to capabilities and availability of the computing resources to the Metacomputing Directory Service (MDS).the Metacomputing Directory Service (MDS).Provides an API for submitting and canceling a job Provides an API for submitting and canceling a job request, as well as checking the status of a submitted request, as well as checking the status of a submitted job. job.

GRAMGRAM

A Gatekeeper runs on the A Gatekeeper runs on the remote hostremote hostCreates jobmanager for the jobCreates jobmanager for the jobGatekeeper:Gatekeeper:

mutually authenticates with mutually authenticates with the client, the client,

maps the requestor to a local maps the requestor to a local user, user,

starts a job manager on the starts a job manager on the local host as the local user, local host as the local user, and and

passes the allocation passes the allocation arguments to the newly arguments to the newly created job manager. created job manager.

Jobmanager:Jobmanager: Common componentCommon component Machine-specific componentMachine-specific component

GRAMGRAM

Advanced reservation and co-Advanced reservation and co-allocation - GARAallocation - GARA

Globus References / sources / Globus References / sources / creditscredits

A Resource Management Architecture for Metacomputing A Resource Management Architecture for Metacomputing SystemsSystems. K. Czajkowski, I. Foster, N. Karonis, C. Kesselman, S. . K. Czajkowski, I. Foster, N. Karonis, C. Kesselman, S. Martin, W. Smith, S. Tuecke. Martin, W. Smith, S. Tuecke. Proc. IPPS/SPDP '98 Workshop on Proc. IPPS/SPDP '98 Workshop on Job Scheduling Strategies for Parallel ProcessingJob Scheduling Strategies for Parallel Processing, pg. 62-82, 1998., pg. 62-82, 1998.Describes the resource management architecture implemented as Describes the resource management architecture implemented as part of the Globus system.part of the Globus system.A Distributed Resource Management Architecture that A Distributed Resource Management Architecture that Supports Advance Reservations and Co-AllocationSupports Advance Reservations and Co-Allocation. I. Foster, C. . I. Foster, C. Kesselman, C. Lee, R. Lindell, K. Nahrstedt, A. Roy. Kesselman, C. Lee, R. Lindell, K. Nahrstedt, A. Roy. Intl Workshop Intl Workshop on Quality of Serviceon Quality of Service, 1999., 1999.Describes the new Globus Architecture for Reservation and Describes the new Globus Architecture for Reservation and Allocation, which integrates CPU and network QoS.Allocation, which integrates CPU and network QoS.

Globus References / sources / Globus References / sources / creditscredits

A Security Architecture for Computational GridsA Security Architecture for Computational Grids. I. . I. Foster, C. Kesselman, G. Tsudik, S. Tuecke. Foster, C. Kesselman, G. Tsudik, S. Tuecke. Proc. 5th Proc. 5th ACM Conference on Computer and Communications ACM Conference on Computer and Communications Security ConferenceSecurity Conference, pp. 83-92, 1998., pp. 83-92, 1998.Describes techniques for authentication in wide area Describes techniques for authentication in wide area computing environments.computing environments.http://www.globus.org/Security/papers/pki04-welch-proxyhttp://www.globus.org/Security/papers/pki04-welch-proxy-cert-final.pdf-cert-final.pdfA National-Scale Authentication InfrastructureA National-Scale Authentication Infrastructure. R. . R. Butler, D. Engert, I. Foster, C. Kesselman, S. Tuecke, J. Butler, D. Engert, I. Foster, C. Kesselman, S. Tuecke, J. Volmer, V. Welch. Volmer, V. Welch. IEEE ComputerIEEE Computer, 33(12):60-66, 2000., 33(12):60-66, 2000.Describes our experience designing, developing, and Describes our experience designing, developing, and deploying the Grid Security Infrastructure.deploying the Grid Security Infrastructure.

JUNK !JUNK !

GRAMGRAM

The most common use (and the best The most common use (and the best supported use) of GRAM is remote job supported use) of GRAM is remote job submission and control. This is typically submission and control. This is typically used to support distributed computing used to support distributed computing applicationsapplications

For remote job submission and resource For remote job submission and resource managementmanagement

GRAM RSL attributesGRAM RSL attributes

The specifications are written by the user in the The specifications are written by the user in the Resource Specification Language (RSL), and is Resource Specification Language (RSL), and is processed by GRAM as part of the job request.processed by GRAM as part of the job request.(directory=(directory=valuevalue)) (executable=(executable=valuevalue))(arguments=(arguments=valuevalue [ [valuevalue] [] [valuevalue] ...)] ...)(jobType=single|multiple|mpi|condor)(jobType=single|multiple|mpi|condor)(count=(count=valuevalue)) (hostCount=(hostCount=valuevalue))(two_phase=(two_phase=<int><int>)) (restart=(restart=<old JM contact><old JM contact>))

DUROC RSL attributesDUROC RSL attributes

LabelLabel

resourceManagerContactresourceManagerContact

subjobCommsTypesubjobCommsType

subjobStartTypesubjobStartType

ExampleExample

(executable = a.out)(executable = a.out)

(directory = /home/nobody )(directory = /home/nobody )

(arguments = arg1 "arg 2")(arguments = arg1 "arg 2")

(count = 1) (count = 1)

WS GRAMWS GRAM

A set of OGSI compliant services that provide remote job A set of OGSI compliant services that provide remote job execution execution

(Master) Managed Job Factory Service (MJFS) (Master) Managed Job Factory Service (MJFS) Managed Job Service (MJS) Managed Job Service (MJS) File Stream Factory Service (FSFS) File Stream Factory Service (FSFS) File Stream Service (FSS) File Stream Service (FSS)

Resource Specification Language (RSL-2) schema is Resource Specification Language (RSL-2) schema is used to communicate job requirements used to communicate job requirements Remote jobs run under local users account Remote jobs run under local users account Client to service credential delegation is done user to Client to service credential delegation is done user to user, *not* through a third partyuser, *not* through a third party

RSL-2 ExampleRSL-2 Example

GNSGNS = “http://www.globus.org/namespaces“ <?xml = “http://www.globus.org/namespaces“ <?xml version="1.0" encoding="UTF-8"?> <rsl:rsl version="1.0" encoding="UTF-8"?> <rsl:rsl xmlns:rsl="xmlns:rsl="GNSGNS/2003/04/rsl" /2003/04/rsl" xmlns:gram="xmlns:gram="GNSGNS/2003/04/rsl/gram"/2003/04/rsl/gram" xmlns:xsi="http://www.w3.org/2001/XMLSchema- xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"instance" xsi:schemaLocation=" xsi:schemaLocation=" GNSGNS/2003/04/rsl/2003/04/rsl ./schema/base/gram/rsl.xsd ./schema/base/gram/rsl.xsd GNSGNS/2003/04/rsl/gram/2003/04/rsl/gram ./schema/base/gram/gram_rsl.xsd"> ./schema/base/gram/gram_rsl.xsd"><gram:job> <gram:executable><rsl:path> <gram:job> <gram:executable><rsl:path> <rsl:stringElement value="/bin/ls"/> <rsl:stringElement value="/bin/ls"/> </rsl:path></gram:executable> </gram:job> </rsl:rsl></rsl:path></gram:executable> </gram:job> </rsl:rsl>

Managed Job (Factory) Service Managed Job (Factory) Service Defines an OGSI/GWSDL interface for Defines an OGSI/GWSDL interface for

submitting, monitoring and controlling a job submitting, monitoring and controlling a job MJS uses the File Stream Factory Service to MJS uses the File Stream Factory Service to

manage the job’s stdout and stderr file manage the job’s stdout and stderr file streaming streaming

MJS exposes the stdout and stderr File MJS exposes the stdout and stderr File Stream Factory Grid Service Handles (GSH) Stream Factory Grid Service Handles (GSH) in Service Data Elementin Service Data Element

The MJS instances can monitor jobs in two ways: The MJS instances can monitor jobs in two ways: Resource Information Provider Service (RIPS) Resource Information Provider Service (RIPS) A specialized notification service A specialized notification service Maintains job information from the scheduler Maintains job information from the scheduler Scheduler info provider outputs queue and job data in XML Scheduler info provider outputs queue and job data in XML

Poll the scheduler directly Poll the scheduler directly Only option for FORKOnly option for FORK

MJS to Resource Interface: can support custom-MJS to Resource Interface: can support custom-schedulers through well defined templatesschedulers through well defined templates

WS GRAM ArchitectureWS GRAM Architecture

OGSA and WS MDS Index serviceOGSA and WS MDS Index service

Standard interfaces for Grid services in the form of Standard interfaces for Grid services in the form of WSDL porttypesWSDL porttypesGridService porttype for querying and updating GridService porttype for querying and updating GridService dataGridService dataMDX index service consists of following interfaces:MDX index service consists of following interfaces:

Factory – for creating a grid service instance and return GSHFactory – for creating a grid service instance and return GSH GSH – to refer to a grid service instanceGSH – to refer to a grid service instance GSR – describes how a client can communicate with a grid GSR – describes how a client can communicate with a grid

serviceservice Query – query language supportQuery – query language support Registry - Supports discovery by returning the GSHs of a set of Registry - Supports discovery by returning the GSHs of a set of

Grid servicesGrid services Notification – for registering interest in a serviceNotification – for registering interest in a service