Seminar Grid Computing ‘06

Hui Li

Sep 18, 2006

Overview

• Brief Introduction• Presentations

– Architecture– Functionality/Middleware– Applications

• Projects

Grid Definition

• a Grid is "a set of information resources (computers, databases, networks, instruments, etc.) that are integrated to provide users with tools and applications that treat those resources as components within a 'virtual' system". Grid software solutions provide the underlying mechanisms necessary to create such systems, including authentication and authorization, resource discovery, resource management, communications, and information services, etc.

• Keywords: Virtualization, Middleware

Historically Speaking …

NetworkingARPANET

Communications and Data Sharing:Email, ftp, telnet, TCP/IP

Information Sharing:WWW, HTTP, HTML

Resource Sharing:P2P, Web Services, Grids

Why *?

• Why Grids?– Think beyond only information– Next step in networked computing

• Why now?– CPU, storage, networking– Academic, Commercial, Governmental, Personal– $funding$

Grid Checklist

• Coordinates resources that are not subject to centralized control

• Using standard, open, general-purpose protocols and interfaces (Architecture)

• To deliver nontrivial qualities of service (Performance)

• Security is a *serious* concern

The Evolution of Grid Software (Globus)

Pre-WSAuthenticationAuthorization

GridFTP

GridResource

Allocation Mgmt(Pre-WS GRAM)

Monitoring& Discovery

System(MDS2)

C CommonLibraries

GT2

WSAuthenticationAuthorization

ReliableFile

Transfer

OGSA-DAI[Tech Preview]

GridResource

Allocation Mgmt(WS GRAM)

Monitoring& Discovery

System(MDS4)

Java WS Core

CommunityAuthorization

ServiceGT3

ReplicaLocationService

XIO

GT3

Data Management

SecurityCommonRuntime

Execution Management

Information Services

Web Services

Components

Non-WS

Components

CredentialManagement

GT4

Python WS Core[contribution]

C WS Core

CommunitySchedulerFramework

[contribution]

DelegationService

GT4

Reality -> Vision

• Heterogeneity -> Virtualization

• Diversity -> Standards

• Isolated -> Interoperable

• Tightly-coupled -> Loosely-coupled

• Manual -> Automated

• … …

Toolkit based? Service Oriented!

State of the Art and Beyond:Service Oriented Architecture (SOA)

IBM

IBM

GRAM GridFTPHost EnvUser Svc

ReliableFile

Transfer

MyProxy

Uniform interfaces,security mechanisms,Web service transport,

monitoring

Host EnvUser Svc

Computers StorageSpecialized resource

UserApplication

UserApplication

UserApplication

IBM

IBM

DAIS

Database

MDS-Index

ToolTool

The Evolution of the Grid Seminar

• 2001 - first seminar, p&p structure, parallel applications

• 2003 - continuation• 2004 - p&p structure, system centric -

LUCGrid• 2005 - p&p structure, development &

research

Goal: Group learning, interaction & discussion, R & D

Presentations

• 3 presentations each class• ~30 minutes per presentation• 20-25 minutes talk, 5-10 minutes

discussion• Participation and discussion are highly

promoted, and they will be counted in grading (15%)

• “Non-trivial” questions

Topics at a Glance

Data Management

Security

Resource Management

Information Services

Architecture

Applications

Presentation Topics

• Resource Management– Superscheduling and Resource Brokering– Workload and Resource Management

Systems– State Estimation and Performance

Predictions– Fabric and Local Resource Management

VOUser

Embedded Resource Management:E.g., EGEE & OSG

ClusterResourceManager

GRAM

ClusterResourceManager

GRAM

• VO admin delegates credentials to be used by downstream VO services.• VO admin starts the required services.• VO jobs comes in directly from the upstream VO Users• VO job gets forwarded to the appropriate resource using the VO credentials• Computational job started for VO

Client-side

VO Scheduler Other Services

VO Admin

. . .

Monitoring and control

HeadnodeResourceManager

GRAM

Deleg Deleg

Deleg

VOUser

VO Job

VO Job

Presentation Topics (Cont’d)

• Information Services– Grid Information Services and Systems– Information Retrieval, Dissemination, and

Search– Cluster Resource Monitoring– Network Measurement and Monitoring

Presentation Topics (cont’d)

• Security– Authentication and GSI– Authorization and Virtual Organizations– WS-Security– Firewall Issues

Evolution of Grid Security & Policy• 1) Grid security infrastructure

– Public key authentication & delegation– Access control lists (“gridmap” files)– Limited set of policies can be expressed

• 2) Utilities to simplify operational use, e.g.– MyProxy: online credential repository– VOMS, ACL/gridmap management– Broader set of policies, but still ad-hoc

• 3) General, standards-based framework for authorization & attribute management

Security Services for VO Policy

• Attribute Authority (ATA)– Issue signed attribute assertions

(incl. identity, delegation & mapping)• Authorization Authority (AZA)

– Decisions based on assertions & policy

VO AService

VOATA

VOAZA

MappingATA

VO BService

VOUser A

Delegation AssertionUser B can use Service A

VO-A Attr VO-B Attr

VOUser B

Resource AdminAttribute

VO MemberAttribute

VO Member Attribute

Presentation Topics (cont’d)

• Data Management– Data Transport and Access– Data Storage and Replica Management– High Performance Networking

Presentation Topics (cont’d)

• Architecture– Open Grid Services Architecture (OGSA)– Web Services and WSRF– P2P and Grid

A Two-Dimensional Problem

• Decompose across network

• Clients integrate dynamically

– Select & compose services

– Select “best of breed” providers

– Publish result as new services

• Decouple resource & service providers

Function

Resource

Data Archives

Analysis tools

Discovery toolsUsers

Fig: S. G. Djorgovski

SOA

• Distributed Computing Technology: DCOM, CORBA • Web Services (SOAP, UDDI, WSDL, XML, XACML, etc)

Presentation Topics (cont’d)

• Applications– Grids and Application Scenarios– Common Runtime– Programming Environments– Grid Portals

System-Level Science

Problems too large &/or complex to tackle alone …

Summary Presentations• Resource Management

– Fabric and Local Resource Management (Oct 9th)– Superscheduling and Resource Brokering (Oct 9th) (stafleu)

• Security– Authentication and GSI (Oct 16th) (stoppa)– Authorization and Virtual Organizations (Oct 16th) (damico)

• Information Services– Grid Information Systems and Services (Oct 16th) (puglierin)– Network and Cluster Monitoring (Oct 9th) (Moerkerk)

• Data Management– Data Transport and Access (Oct 23rd) (Krsek)– Data Storage and Replica Management (Oct 23rd) (Sobolewska)

• Architecture– Open Grid Services Architeture (OGSA) (Oct 23rd) (bardini)– Web Services and WSRF (?) (geene)

• Applications – (?)

Projects

• Deployment and Maintenance

• Development/Software

• Research

• A FCFS basis

Project 1

• Maintaining and Extending the LUCGrid (deployment)

Project 2

• A Resource Monitor for non-dedicated LAN environments

Project 3

• Performance Data Miner (PDM)– An Investigation on Real-Time Properties

of Dynamic M-Tree Nearest Neighbor Search (Research)

– Deployable Performance Prediction Services (Software)

Project 4

• Parallel/distributed Computation on DAS-2/3– Resource-aware and application level

scheduling– HIRLAM numerical forecasting

(Communication/computation overlap)

Project 5

• Applications– Programming GT4 Java web services