Upload
connor-coyle
View
221
Download
0
Tags:
Embed Size (px)
Citation preview
INFSO-RI-508833
Enabling Grids for E-sciencE
www.eu-egee.org
EGEE and gLite
Slides by: Erwin Laure
EGEE Deputy Middleware Manager
Enabling Grids for E-sciencE
EGEE
• Objectives– consistent, robust and secure service
grid infrastructure
– improving and maintaining the middleware
– attracting new resources and users from industry as well as science
• Structure
– 70 leading institutions in 27 countries, federated in regional Grids
– leveraging national and regional grid activities worldwide
– funded by the EU with ~32 M Euros for first 2 years starting 1st April 2004
– proposal for 2nd phase (April 2006-March 2008) currently being negotiated
Enabling Grids for E-sciencE
Grid Middleware
Key success factors for production quality software:
• Strict software process– Use industry standard software engineering methods
Software configuration management, version control, defect tracking, automatic build system, …
• Conservative in what software to use– Avoid “cutting-edge” software
Deployment on over 100 sites cannot assume a homogenous environment – middleware needs to work with many underlying software flavors
– Avoid evolving standards Evolving standards change quickly (and sometime significantly cf. OGSI vs.
WSRF) – impossible to keep pace on over 100 sites
You will not develop and deploy your PhD project on a production Grid infrastructure
There is a long (and tedious) path from prototypes to production
Enabling Grids for E-sciencE
EGEE Middleware: gLite
• gLite– Exploit experience and existing
components from VDT (Condor, Globus), EDG/LCG, AliEn, and others
– Develop a lightweight stack of generic middleware useful to EGEE applications (HEP and Biomedics are pilot applications). Should eventually deploy dynamically (e.g. as a globus job) Pluggable components – cater for different implementations
– Focus is on re-engineering and hardening
Enabling Grids for E-sciencE
gLite Software ProcessDevelopment Integration Testing
Software Code
Deployment Packages
Integration Tests
Fail Pass
Fix
Functional Tests
Testbed Deployment
Fail
Pass
Installation Guide, Release Notes, etc
Enabling Grids for E-sciencE
gLite Grid Middleware Services
API
Access
Workload Mgmt Services
ComputingElement
WorkloadManagement
MetadataCatalog
Data Management
StorageElement
DataMovement
File & ReplicaCatalog
Authorization
Security Services
Authentication
Information &Monitoring
Information & Monitoring Services
Application
Monitoring
Connectivity
Accounting
Auditing
JobProvenance
PackageManager
CLI
Enabling Grids for E-sciencE
gLite Key Concepts
• Centered around VOs– It’s ultimately the VO who gets resources allocated and need to
decide how to best use them (share them among the VO users)
• Distinguish between infrastructure and VO services
• Infrastructure services– Operated and trusted by the resource administrator– Implement site policies
Including what share of the resources are allocated to a VO
– Provide the required security, auditing, and accounting – Grid and standard services
E.g. batch system, gatekeeper, gridFTP, …
Enabling Grids for E-sciencE
gLite Key Concepts
• VO services– Implement intra-VO policies
Scheduling, priorities, etc.
– Managed and operated by a VO Typically by sites on behalf of VOs A service instance may serve multiple VOs
– Currently mostly higher level services Resource brokers, catalogs, …
– There is the need of deploying VO services closer to the resource Better information about the resource and better control
about the resource Downside: more and more services to be deployed at the
sites – see discussion later on
Enabling Grids for E-sciencE
ServiceService Oriented Architecture Oriented Architecture
Guiding Principles
InteroperabilityInteroperability
PortabilityPortability
ModularityModularity
ScalabilityScalability
Web ServicesWeb ServicesBuilding on existingBuilding on existing
components in acomponents in alightweight mannerlightweight manner
AliEn LCG Condor
Globus SRM ...
Enabling Grids for E-sciencE
Security: Basic Concepts
• Grid security is based on X.509 PKI infrastructure– Certificate Authorities (CA) issue (long lived) certificates
identifying individuals (much like a passport)– Trust between CAs and sites is established (offline)– User identification is done by using (short lived) proxies of their
certificates
• Proxies can– Be delegated to a service such that it can act on the user’s
behalf– Include additional attributes (like VO information via the VO
Membership Service VOMS)– Be stored in an external proxy store (myProxy) – Be renewed (in case they are about to expire)
• Standard TLS (like ssh) and MLS (like WS-Sec) is used– MLS has performance and support issues!
Enabling Grids for E-sciencE
Middleware Challenges: Data Management
• Heterogeneity– Data is stored on different storage
systems using different access technologies
• Distribution– Data is stored in different locations –
in most cases there is no shared file system or common namespace
– Data needs to be moved between different locations
• Different Administrative Domains– Data is stored at places you would
normally have no access to– Security and auditing implications
– Need common interface to storage resources
Storage Resource Manager (SRM)
– Need to keep track where data is stored
File and Replica Catalogs
– Need scheduled, reliable file transfer
File transfer and placement services
– Need a common security model
ACLs enforcement based on Grid identities – DNs
Enabling Grids for E-sciencE
Data Management Architecture
File andReplica Catalog
StorageIndex
Fireman
Database
WMS
Storage Element
SRM
Storage
gLite I/O gridFTP
File Transfer andPlacement Service FTS
FPS Transfer Agent
Database
VOMS
MyProxy
Getcredential
Storecredential
File I/O
File namespace
and Metadata mgmt
File replication
Proxy renewal ReplicaLocation
WSDL
API
Enabling Grids for E-sciencE
gLite Transfer services
• File Transfer/Placement Service (FTS,FPS)– Transfer Job Database– Exposes the Transfer Web Service Interface
to which user clients talk (submit, cancel, status capability)
– Has a Web Interface– Manages Catalog updates if necessary
• Transfer Agent– Basic Actions
Get transfer jobs from Transfer Job Database Manages transfer over many channels Monitors transfer status and updates Transfer
Job Database
– Extensible with user-defined custom actions Retry Policy
• Transfer Service (glite-url-copy)– Actually performs transfer: SRM – SRM,
gsiftp – SRM, gsiftp – gsiftp – Monitor capability, including gsiftp
performance markers
Job DB
FTS/FPSWebService
Transfer Agent
Actions
Channel Channelglite-url-copy
glite-url-copy
glite-url-copy
glite-url-copy
glite-url-copy
glite-url-copy
Web Monitor
Enabling Grids for E-sciencE
Middleware Challenges: Workload Management
• Computational tasks of thousands of users need to be scheduled on the available Grid resources
• Grid (Meta)Scheduling consists of:– Resource Discovery/Brokering
Find suitable resources
– Matchmaking Assign a job to a resource that satisfies job requirements
– Job execution Reliably execute the jobs and retrieve output Deal with error management
• Job execution requires to find the “right” Computing Element (computing resource)
– with maybe boundary conditions (architecture, software installed, data accessible, etc.)
Enabling Grids for E-sciencE
Workload Management Architecture
Enabling Grids for E-sciencE
CEs & Managed Services
• CEs provide also a framework for deploying VO specific services
• VO services need to be Managed Services
• Ensure they don’t consume more resources as allocated• Provide persistency and management functions (start, stop,
suspend, resume)• Adhere to site security, auditing, and accounting policies
• All that could be done by site admins but it would be favorable to have infrastructure services taking care of that
Enabling Grids for E-sciencE
Managed Services Architecture
Authorization & configuration policies
Managed Service Factory
Resource Manager
Man
agem
ent
Cli
ents
Managed service factory functions Resource
manager commands
(Managed services)
Monitoring & enforcement
Man
aged
S
ervice C
lients
Enabling Grids for E-sciencE
More Information
• The EGEE Project– http://www.eu-egee.org
• The LCG Project– http://cern.ch/lcg
• The gLite middleware– http://www.glite.org
• The Condor Project– http://www.cs.wisc.edu/condor
• The Globus Project– http://www.globus.org