Legacy code support for commercial production Grids

www.cpc.wmin.ac.uk/GEMLCAwww.cpc.wmin.ac.uk/GEMLCA

Legacy code support for commercial production Grids

G.Terstyanszky, T. Kiss, T. Delaitre, S. WinterSchool of Informatics, University of Westminster

P. Kacsuk, G KecskemetiMTA SZTAKI Laboratory of Parallel and Distributed Systems,

Hungary

Extending GEMLCA to fulfil requirements of business grids


Presentation Outline1. GEMLCA (Grid Execution Management for

Legacy Code Applications) concept

2. GEMLCA requirements towards commercial production Grids

3. GEMLCA Extensions• Dynamic User Management• Resource & Service Monitoring• Accounting and Charging

4. Conclusion and future work


GEMLCA Concept ComputeServers

Resource manager deploys:

LCGEMLCA

GEMLCAresource

Client 1: to register

legacy code as a Grid service

Client 2: to invoke

legacy code Grid service

Send custom inputSend custom inputInvoke Invoke legacy legacy codecode

Produce Produce resultresult

Return resultReturn result


The GEMLCA-view of a legacy code• Any code that correspond to the following model

can be exposed as Grid service by GEMLCA:

Legacy codeLegacy codeInput data Input data

(files, command (files, command line params, env. line params, env.

vars)vars)

Output data Output data (files)(files)

Call shared librariesCall shared libraries

Query databasesQuery databasesCommunicate over the networkCommunicate over the network

Perform computationPerform computation

. . .. . .


What’s behind the GEMLCA resource…

Client

GEMLCA Service

ComputeServers

Local jobmanager

e.g. Condor/Fork

GEMLCAProcess

LegacyCode Job

Job submission and file transfer

services

Gridmiddleware

layerGrid Service

Client

Consequences:Consequences:

1.1. Not only the legacy code and the Not only the legacy code and the GEMLCA service, but also GEMLCA service, but also a local job a local job manager and a Grid middleware layer manager and a Grid middleware layer must be installed on the hosting must be installed on the hosting system!system!

2.2. The aim of the code registration The aim of the code registration process is to tell GEMLCA how to process is to tell GEMLCA how to submit the legacy code to the grid submit the legacy code to the grid middleware layermiddleware layer


3rd generation Grids: (OGSA: GT4, gLite)

Legacy applications

Legacy applications

Grid Site 1

Grid Site 2

. . .. . .

The GEMLCA P-GRADE Portal

GEMLCAP-GRADE Portal Server

Desktop 1

Desktop N

Web browser

Web browser

LC 1LC 1

LC 2LC 2

ResultResult

InputInput


GEMLCA on the UK NGSThe NGS P-GRADE GEMLCA Portal

• Runs as a service available for all NGS users

• Portal Website: http://www.cpc.wmin.ac.uk/ngsportal/

WestminsterWestminsterWestminster


Requirements

– dynamic user management users’ accounts created automaticallygrid-mapfile modified automatically

– monitoring of GEMLCA & Grid resources information about previous and current state of

Grid resources & services

- accounting/charging of resource/service usage resource and service usage should be metered and charged accordingly

GEMLCA and business Grids


Dynamic User Management • need for dynamic user management

• dynamic and scalable management of several hundred or thousand users without manual interactions

• dynamic user management solutions• identity mapping (i.e. mapping Grid certificates) to local

accounts• providing local accounts on demand

• user management implementation• Local Credential Mapping Service (LCMAPS) to map

credentials to local account• Workspace Management Service (WMS) to allocate

workspace (or local account) to users


Dynamic User Management Service

WorkspaceManagement

ServiceP-GRADE

Portal ServerDesktop 1

Web browser

Desktop n

Web browser abstract workflow Legacy applications

+ default input

Grid Site

account pool

Identity Mapping

(LCMAPS)returning a workspace


Grid Monitoring Service

• Need for Monitoring• checking availability of resources/services• generating historical availability data • classifying availability of resources/services & making this

data available for brokers

• Monitoring scenarios• MDS initiated regular monitoring• user initiated monitoring

• Monitoring Implementation• probes are implemented by scripts and their outputs are

displayed in a monitoring portlet


Grid resource probes: network connectivity of a remote host access to a remote MyProxy server file transfer using a remote host GridFTP server Globus job submission (WS-GRAM) availability of the local information system

(MDS service) local job manager (Condor, PBS, SGE etc.)

GEMLCA services probes: GLCAdmin,

GLCList GLCProcess

Grid Monitoring Service


Grid Monitoring Service Architecture

Grid site with legacy code resources


MDS4GMT

(GEMLCA Monitoring

Toolkit)

Production Grid(e.g. EGEE, NGS)

Run GMT probes and collect data

3rd party service providers

User

Portal

Map execution


On-demand execution as a prototype

Grid Monitoring Service in the P-GRADE portal

Automatic execution

Already works as a service on the NGS P-GRADE portal


GMT classifier



MDS4GMT

(GEMLCA Monitoring

Toolkit)

Grid broker(e.g. LCG

broker)

Production Grid(e.g. EGEE, NGS)

Run GMT probes and collect data

Historical test data

Store historical

data

Prediction based on

current and historical

data

Resource availability

3rd party service providers

User

Portal

Current work - Resource testing and availability prediction

Submit workflow

Map execution

Map execution


Grid Accounting & Charging Service

• Need for accounting/charging• automatic resource/service usage accounting

• managing multiple charging policies & implementations

• creating a database to store both accounting & charging data

• Accounting/charging architecture• create a Grid Bank to manage all transactions related to

accounting & charging

• Grid Accounting & Charging service (GAC) based on the GGF’s RUR specification


resource data user data

host name / IP address Host name/IP address

certificate name certificate name

host type

local job ID

wall clock time + price

user CPU time + price job data

system CPU time + price job ID

main memory + price application name

secondary storage+ price job start date

I/O channels + price job end date

Resource Usage Record (RUR)

user certificate name

resource certificate name

job ID

resource business model

resource usage date

resource usage time

amount to be charged

Resource Charge Record (RCR)



P-GRADE

Portal ServerDesktop 1

Web browser

Desktop n

Web browser

Broker

AccountingService

Abstract workflow

Legacy applications + default input

Grid Site 1

Legacy applications + default input

Grid Site m

collecting usage data

ChargingService

RURRUR

RCR

database



Conclusions and Future Work

dynamic user management• installed on the GT4-based

WestFocus Grid • will be used in the NGS GT4

testbed

resource/service monitoring• automatic execution installed on

the NGS P-GRADE portal • On-demand execution under

development & testing

accounting/charging - under development & testing


Email:[email protected]:www.cpc.wmin.ac.uk/gemlcawww.lpds.sztaki.hu/pgportalwww.cpc.wmin.ac.uk/ngsportal

Thank you for your attention …Any

questions?

GEMLCA/P-GRADE portal demonstrations on the GOSC booth during the AHM!

Documents

Legacy code support for commercial production Grids