ww

w.ics

y.d

e GRID

Neue Netzdienste

ZKI Arbeitskreis NetzdiensteZKI Arbeitskreis Netzdienste

WürzburgWürzburg, den 08. März 2004

Dipl.-Inform. Markus HillenbrandTU Kaiserslautern - Kaiserslautern University of Technology,

Dept. of Computer ScienceICSY – Integrated Communication Systems

hillenbr@informatik.uni-kl.de Tel.+49 631 205 4173, Fax. +49 631 205 3056

ww

w.ics

y.d

eEvolution of the Internet

1st generation (since the beginning of the Internet):• permanent IP addresses, always connected• static domain name system (DNS) mapping• limited specialized applications, protocols: Telnet, FTP, Gopher, World Wide Access

2nd generation (since 90ties):• WWW & graphical browsers• dynamic IP addresses / NAT* (network address translation) /

roaming users• heterogeneous applications, asymmetric server based services• protocol: HTTP, ..World Wide Web

3rd generation (since 2000):• more collaboration and personalized applications• powerful edge devices (peers), instant networking• protocols/applications:

• Napster, Gnutella,…• Grid

World Wide Peering

ww

w.ics

y.d

eGrid Vision

• Ideas (and name) started with power grid generation

• I don’t care where the power used to run my fridge is generated• unless You live in California

• I just consume • and pay for my consumption

• I could have a computer terminal Plug it in and pay for the computation as I go

• But:• what application do I want to use? where is it?• what data is input? where is it? what results are produced?• where are they and how do I see them?• Are these transactions all really secure?

So it is not that simple

ww

w.ics

y.d

eGrid Definition

• Recall the following definition:

The transparent, systematic and effective utilisation of geographically distributed heterogeneous resources (both hardware and software) for applications in science and commerce

ww

w.ics

y.d

eThe Grid: Another Definition

“Dependable, consistent, pervasive access to

[high-end] resources”

• Dependable: Can provide performance and functionality guarantees

• Consistent: Uniform interfaces to a wide variety of resources

• Pervasive: Ability to “plug in” from anywhere

ww

w.ics

y.d

eGrid Resources

• Computational hardware• Storage (secondary, tertiary)• Networks • Databases, large datasets• Remote instruments• Applications• Libraries• Compilers and tools

ww

w.ics

y.d

eSo what´s a Grid?

• Grids are large• in terms of potentially available resources

• Grids are distributed• substantial latencies in moving data, may dominate

application runtime• Grids are dynamic

• resources may change during the lifespan of an application

• Grids are heterogeneous• form and properties of sites (nodes) may differ

significantly• Grids are across boundaries of organizations

• access policies differ at different sites

ww

w.ics

y.d

eWhy Grid ?

Why Grid? large-scale, flexible, secure, coordinated

resource sharing among dynamic collections of high-performance computers

Characteristics of Grid Inherited from Client/Server model unrestricted registration and exit are NOT

allowed QoS ensured service name the Grid node as Server Entity

ww

w.ics

y.d

eGrid Applications

ww

w.ics

y.d

eGrid Computing

GRID Objectives• Resource sharing among individuals, institutions and resources• Bandwidth, processing capacity, storage

Globally distributed heterogeneous resources

Grid• “Infrastructure that provides dependable, consistent, pervasive, and

inexpensive access to high-end computational capabilities” (1998)• “A system that coordinates resources not subject to centralized control,

using open, general-purpose protocols to deliver nontrivial QoS” (2002)good legal applications without communities

P2P• “Applications that takes advantage of resources at the edges of the

Internet” (2000)• “Decentralized, self-organizing distributed systems, in which all or most

communication is symmetric” (2002)clever designs without good, legal applications

ww

w.ics

y.d

eGrid and P2P

P2P GRID

History Sharing MP3 files & often illegal

content

Saving costs for data processing centers

Users Mainly private users

Mainly Scientists

Typical Transfer Volume

Small (MP3) to medium (video)

Huge (often terabytes)

Typical Service File Sharing Processing Sharing

Typical Problems

Hugh number of users cause

scalability issues

Transferring huge amounts of data

ww

w.ics

y.d

e1) Both are concerned with the same general problem

• Resource sharing within virtual communities

2) Both take the same general approach• Creation of overlays that need not correspond to underlying

structures• Networks• Organizations

3) Each has made genuine technical advances, but in complementary directions• “Grid addresses infrastructure but not yet failure”• “P2P addresses failure but not yet infrastructure”

4) Complementary strengths and weaknesses => room for collaboration (Ian Foster)

Grid and P2P

ww

w.ics

y.d

eFrom Web-Services to Grid-Services

Definition:Definition: • "An interface that describes a collection of operations

that are network accessible through standardized XML messaging."

• Self-contained, modular applications that can be described, published, located, and invoked over the Web.

• Use open standards and common infrastructure for description, discovery and invocation (XML over HTTP plus: SOAP, WSDL, UDDI)

• In the web…• The key was a universal server-to-clientserver-to-client model based on

standards• In Web Services …

• The key will be a universal program-to-programprogram-to-program communication model based on standards and industry support

ww

w.ics

y.d

eWeb-Services

ww

w.ics

y.d

eWeb-Services / Grid-Services

• Web Services address discovery and invocation of persistent services• Interface to persistent state of entire enterprise

Definition: A Web service that adheres to OGSI standards is called a Grid service.

• Grid Services must also support transient service instances that are dynamically created/destroyed

• need interfaces to the states of distributed activities

• Significant implications for how services are managed, named, discovered, and used!

OGSA: Open Grid Service Architecture

• Within OGSA, everything is represented as a Grid service, that is, a (potentially transient) service that conforms to a set of conventions (expressed using WSDL) for such purposes as lifetime management, discovery of characteristics, and notification.

• An important merit of this service-oriented model is that all components of the environment can be virtualized (and some of them can be created by a service factory on demand)

ww

w.ics

y.d

eInfrastructure and Architecture

Global Grid Forum GGF • community-initiated forum of 5000+ individual researchers and

practitioners • working on distributed computing ("grid" technologies). • primary objective

• to promote and support the development, deployment, and implementation of Grid technologies and applications

• via the creation and documentation of "best practices" - technical specifications, user experiences, and implementation guidelines.

Grid service• OGSI = Open Grid Service Infrastructure• Web services interfaces and behaviors

• address key distributed system issues

Standard service specifications• OGSA = Open Grid Services Architecture• Resource mgt, db, workflow, security, …• OGSA defines “OGSA compliance”

… and arbitrary application-specific services based on these & other definitions

ww

w.ics

y.d

eOGSI and Stateful Services

• Sometimes you can send a message to a service, get a result and

that’s the end• This is a statefree service

• However most non-trivial services need state to allow persistent

asynchronous interactions• OGSI is designed to support Stateful services through two

mechanisms• Information Port: where you can query for SDE (Service

Definition Elements)• “Factories” that allow one to view a Service as a “class” (in an

object-oriented language sense) and create separate instances for each Service invocation

• There are several interesting issues here• Difference between Stateful interactions and Stateful services• System or Service managed instances

ww

w.ics

y.d

eOGSI = Open Grid Service Infrastructure

Implementation

Servicedata

element

Other standard interfaces:factory,

notification,collections

Hosting environment/runtime(“C”, J2EE, .NET, …)

Servicedata

element

Servicedata

element

GridService(required)

Dataaccess

Lifetime management• Explicit destruction• Soft-state lifetime

Introspection:• What port types?• What policy?• What state?

Client

Grid ServiceHandle

Grid ServiceReference

handleresolution

ww

w.ics

y.d

eOpen GRID Service Architecture - OGSA

OGSA Objectives • Integrating

• GRID Technology• Web Services

• Creation of• Virtual infrastructure • Service based architecture

• Support of virtual organisations

OGSA Characteristics• Openness

• Service interaction via published interfaces• Platform independent

• Virtual service definition• Multiple interworking implementations

• Portability • Design portability & code portability on hosting environments

ww

w.ics

y.d

eOGSA / Web-Services

What is OGSA:• „One of the major distinctions between a Grid service and an

ordinary Web-Service is the underlying assumption that Grid-Services may be transient, created as part of the normal operation of the infrastructure.“ [Ian Foster]

This leads to • Factories creating services • Lifetime management of services • Service groups

ww

w.ics

y.d

eOGSA Services

OGSA defines broadly applicable services upon OGSI:

• Core Services• Name resolution and discovery• Service domains• Security• Policy• Messaging, queuing, and logging• Events• Metering and accounting

• Data and Information Services• Data naming and access• Replication• Metadata and provenance

• Resource and Service Management• Provisioning and resource management• Service orchestration• Transactions• Administration and deployment

ww

w.ics

y.d

eOGSA Future directions

Services and tools:• the set of OGSA services needs to be expanded concerning

security, resource management, data and knowledge management, instrumentation and troubleshooting

Implementation:• high performance protocols, lightweight service instances,

effective sandboxing techniques

Semantics:• formal mechanisms for expressing the behavior of individual

services

Scalability:• multiple dimensions: large numbers of entities, service

economies, embedded and mobile systems

ww

w.ics

y.d

eWS-Resource Framework

WSRF is a family of Web services specifications (proposals) that clarify:• how stateful resources are addressed• how to access resource properties using XML• how to organize groups of resources and services

WSRF and WS-Notification are an evolution of OGSIIan Foster:

• “The definition of WSRF means that Grid and Web communities move forward to a common base.”

Why is WSRF important?• WSRF completes Grid/Web

convergence

How does WSRF relate to OGSI?• WSRF restates OGSI concepts in WS

terms

How does WSRF relate to OGSA?• WSRF mechanisms will enable OGSA

What will Globus Alliance do with WSRF?• WSRF-based GT4.0 planned for Q3

2004

What does WSRF mean for GT3.0 users?• For the most only minor changes

ww

w.ics

y.d

eApplication Areas

Targeted Application Areas• eScience

• Distributed supercomputing• Resource sharing, e.g. CPU, bandwidth• Data intensive computing• Data sharing & collaborative computing

• eLearning • eBusiness• eHealth

Globus Toolkit • Large set of tools

• High level, pragmatic implementations

OGSA• Universal services architecture concept• Combining GRID and Web services

ww

w.ics

y.d

eImportant Things

Single sign-on: collective job set should require once-only user authentication

Mapping to local security mechanisms: some sites use Kerberos, others using Unix

Delegation: credentials to access resources inherited by subcomputations, e.g., job 0 to job 1

Community authorization: e.g., third-party authentication

ww

w.ics

y.d

eSources of Grid Technology

Grids support distributed collaboratories or virtual organizations integrating concepts from• Web-Services, Semantic Web

• W3C standard WSDL defines IDL (Interface standard) for Web Services

• Agents• Distributed Objects (CORBA Java/Jini COM)• High Performance Computing activities• Peer-to-peer Networks

With perhaps the Web and P2P networks being the most important for “Information Grids” and Globus for “Compute Grids”

ww

w.ics

y.d

eActivities

• Look at different styles of Grids such as Autonomic (Robust Reliable Resilient)

• New Grid architectures hard due to investment required

• Critical Services Such as• Security – e-security• Notification – event services• Metadata – Use Semantic Web• Databases and repositories – instruments, sensors• Computing – Submit job, scheduling, distributed file systems• Visualization, Computational Steering• Fabric and Service Management• Network performance

• Program the Grid – Workflow• Access the Grid – Portals, Grid Computing

Environments

ww

w.ics

y.d

eConclusions

• Grids are inevitable and pervasive

• Can expect Web Services and Grids to merge with a common set of general principles but different implementations with different scaling and functionality trade-offs

• We will be flooded with data, information and purported knowledge

• One should be preparing Grid strategies; understanding relevant Web and Grid standards and developing new domain specific standards

• Note many existing (standards) efforts assume client-server and not a brokered service model; these will need to change!

ww

w.ics

y.d

eD-Grid Initiative

ww

w.ics

y.d

eGRID Konzept

• Komplexe, verteilte Arbeitsform in variablen, virtuellen Organisationen

• Nutzung von im Netz verteilt verfügbaren Ressourcen wie Bandbreite, Rechner, Speicher, Information, Werkzeuge

• Kopplung heterogener technischer und organisatorischer Umgebung über Netze

Anforderungen an Netze Anforderungen an Middleware administrative Anforderungen

ww

w.ics

y.d

eGRID Aktivitäten

Bündelung der GRID Aktivitäten in:• USA: Cyberinfrastructure• UK: e-Science• NL: Virtual Lab• GGF• EU: 6th Framework Projekte (EGEE)

GRID Aktivitäten in Deutschland:• in verschiedenen Wissenschaftscommunities erste

Ansätze: Teilchenphysik, Klimaforschung, Biowissenschaften

• unkoordinierte, sporadische Förderung dieser Aktivitäten durch unterschiedliche Referate im BMBF

• erste Koordinierungsversuche durch DFN (G3-Group) bis 2002

• unkoordinierte Mitarbeit in internationalen Gremien (z.B. GGF) nur durch Einzelaktivitäten

ww

w.ics

y.d

eD-GRID: Entstehung

• 2 Workshops (auf Initiative des FZ KA u. der HGF)

• beteiligt alle relevanten Einrichtungen aus dem Wissenschaftsbereich + DFN + BMBF + Industrie

• Konsens über Bündelung der GRID Aktivitäten • D-GRID: Ziele der Initiative

• Bündeln der Aktivitäten, um Synergiepotenziale für globale,verteilte und neuartige Wissenschaftskollaborationen auf der Basis internetgestützter Dienste freizusetzen

• Etablierung eines e-Science-Frameworks• D-GRID: Organisation• Bildung eines Lenkungsausschusses (LA)

• Wahl der Herren Hegering (LRZ), Hiller (AWI), Maschuw (FZK, GridKa), Reinefeld (ZIB), Resch (HLRZS)

• Aufgabe: Strategiepapier erstellen

ww

w.ics

y.d

eD-GRID: Weiteres Vorgehen

• Einrichtung von 5 AKs (horiz. Struktur) Unterzeichner der Grundsatzerklärung können mitarbeiten

• Aktuelles Ziel aller AKs: Mitarbeit an der Erstellung des Arbeitsprogrammes der Initiative

• Für jeden AK gilt: Einhaltung der terminlichen und inhaltlichen Vorgaben des LA ist zwingend für den Erfolg

• Arbeitskreise:• AK1: Kooperationsmodelle für den Betrieb• AK2: Middleware und Services• AK3: Management Methoden und Autonomic Computing• AK4: Netz• AK5: Datenmanagement

ww

w.ics

y.d

eArbeitskreise

AK1: Kooperationsmodelle für den Betrieb

• Bestandsaufnahme verfügbarer Ressourcen• Abrechnungsmodelle für Nutzung von Ressourcen,

Ressourcenbewertung• Rechte für Ressourcennutzung• Weitere rechtliche und org. Fragestellungen• Bestandsaufnahme von Organisationsformen in Communities (im

Sinne VO)

AK2: Middleware und Services• Bestandsaufnahme vorhandener Middleware• Interop.anforderungen an Middleware Komponenten• Bestimmung des Bedarfs (Security, AAA, VC, Directory, Roaming)• Ressourcebroker und weitere Dienste• Beschreibungssprachen, Modelling

ww

w.ics

y.d

eArbeitskreise

AK3: Management Methoden und AutonomicComputing• Rahmenbetriebsstrukturen der Middleware- und Serviceebene (SLAs,

Support- und Operationszentren, Einbindung RZs), auch Forschungsaufgaben dazu

• Qualitätsmanagement• Methoden für Self-Management, Self Healing... von Ressourcen und

Diensten im Netz

AK4: Netz• Bestandsaufnahme existierender Netz-Dienste und -Protokolle (G-WiN

u.a.)• Bestimmung neuer, Grid spezifischer Netzdienste• Bestimmung der Volumenbedarfe (Grids)• Performance Monitoring

AK5: Daten-und Informationsmanagement• Langzeitarchivierung• Metadaten, semantische Methoden und Informationsdienste• verteilte gemeinsame Dateisysteme• Massendatenverwaltung

ww

w.ics

y.d

eD-GRID und DFN-Verein

Aufgaben des DFN-Vereins:• Bereitstellung der Netzinfrastruktur G-WiN• Aufbau und Betrieb von Basisdiensten der Middleware-

Ebene, wie z.B. PKI, AAA

Aufgaben für Rechenzentren:• Aufbau von GRID-Know How vor Ort• Support für Nutzer bei Einführung von Middlewarediensten• Bereitstellen von Bausteinen für Middlewaredienste

(z.B.)Nutzerverzeichnisse

Stärkung der Rolle der RZ als Anbieter für

zentrale IT-Dienste