A module to customize CREAM A module to customize CREAM jobs according to site policiesjobs according to site policies

Tsukuba, KEK, 21 December 2010

Sylvain Reynaud

JWGEN :JWGEN :

2

The problemThe problem

Adapt the job…– to user needs

• set user job environment– middleware version, operating

system, environment variable

• set LRMS queue• set hardware resources

– CPU, memory, disk

• set logical resources– hpss, xrootd, dCache, oracle

• set permissions (sw manager)

– to site policy• accept/deny job submission• enforcement of security policy• options and information for :

– quota, priorities, accounting

According to the information associated with this job

– Computing Element (CE)

– CE Queue

– User's virtual organization

– User's role

– JDL requirements (if specified)• CPU• memory• disk

3

A solution : JWGENA solution : JWGEN

JWGEN : a module to adapt grid job to…– user needs

– site policy

…according to information associated with this job– CE, queue, VO, requirements…

CREAM-CECREAM-CE

BLAHBLAH GIPGIP APELAPEL

jobcontrol

jobmonitor

info.provider

accoun-ting

BQSBQS

JWGEN

4

A solution : JWGENA solution : JWGEN

CREAM-CECREAM-CE CREAM-CECREAM-CE

Pado

vaPa

dova

CN

AF

CN

AF

BLAHBLAH GIPGIP APELAPEL BLAHBLAH GIPGIP APELAPEL

jobcontrol

jobmonitor

info.provider

accoun-ting

jobcontrol

jobmonitor

info.provider

accoun-ting

BQSBQS Grid EngineGrid Engine

JWGEN JWGEN

5

A solution : JWGENA solution : JWGEN

CREAM-CECREAM-CE

Pado

vaPa

dova

CN

AF

CN

AF

BLAHBLAH GIPGIP APELAPEL

jobcontrol

jobmonitor

info.provider

accoun-ting

Grid EngineGrid Engine

JWGEN

CREAM-CECREAM-CE

BLAHBLAH GIPGIP APELAPEL

jobcontrol

jobmonitor

info.provider

accoun-ting

BQSBQS

JWGEN

JDL

JSDL

options+

wrapper

6

InputInput : JSDL : JSDL

7

OutputOutput : SGE options + job wrapper : SGE options + job wrapper

$ ./libexec/jwgen-core/generate_wrapper.sh /tmp/753627923.jsdl

#!/bin/bash

#$ -q admin

#$ -v SITE_NAME=Test

#$ -A "GRID=EGI SITE=Test TIER=tier3 VO=dteam ROLEVOMS=&2Fdteam&2FRole=NULL&2FCapability=NULL DN=&2FO=GRID-FR&2FC=FR&2FO=CNRS&2FOU=CC-LYON&2FCN=Sylvain&20Reynaud"

#$ -l os=sl5

#$ -N ccsyre_crm01_753627923

#$ -l s_cpu=00:01:40

#$ -l s_vmem=20M

#$ -l s_fsize=5G

#$ -l dcache

[ -r /afs/in2p3.fr/grid/profiles/glite/3.1.25-0/SL4_64/WN32/lcg-prologue.sh ] && . /afs/in2p3.fr/grid/profiles/glite/3.1.25-0/…

… etc …

8

Configuration fileConfiguration file

CREAM-CECREAM-CE

Pado

vaPa

dova

CN

AF

CN

AF

BLAHBLAH GIPGIP APELAPEL

jobcontrol

jobmonitor

info.provider

accoun-ting

Grid EngineGrid Engine

JWGEN

configfile

CREAM-CECREAM-CE

BLAHBLAH GIPGIP APELAPEL

jobcontrol

jobmonitor

info.provider

accoun-ting

BQSBQS

JWGEN

9

Configuration fileConfiguration file

Need to factorize information– thanks to CE profiles

configfile

10

Configuration fileConfiguration file

Need to factorize information– thanks to CE profiles

CC-IN2P3 siteCC-IN2P3 site

CC-IN2P3 Tier 1CC-IN2P3 Tier 1

CC-IN2P3 Tier 2CC-IN2P3 Tier 2 TestTest

EGIEGI LCGLCGprofile A profile B

LCG-CE 04LCG-CE 04CREAM

02

CREAM 02LCG-CE 08LCG-CE 08

CREAM VM-07

CREAM VM-07CREAM 05

CREAM 05

LCG-CE 11

LCG-CE 11

ROC-FRROC-FR

LCG-CE 02LCG-CE 02

LCG-CE 07LCG-CE 07

LCG-CE 01LCG-CE 01

LCG-CE 03LCG-CE 03

LCG-CE 05LCG-CE 05

LCG-CE 06LCG-CE 06

CREAM 03

CREAM 03

11

Configuration fileConfiguration file

Need to factorize information– thanks to CE profiles– thanks to operations on sets

Sets are used for– queues, groups, users, logical resources

Sets can be– created : ignore ancestor sets– subtracted : exclude some items– refined : without adding new items– extended : with adding new items

12

Query configuration fileQuery configuration file

$ ./sbin/jwgen-ce-query.sh -h

./sbin/jwgen-ce-query.sh: option requires an argument -- h

Usage: ./sbin/jwgen-ce-query.sh [-h <hostname>] [-q <queue>] [-g <group>] [-u <user>] [-x]

Where:

-h : hostname of the CE (default value: 'ccsyre.in2p3.fr')

-q : CE queue (default value: '_' = any value)

-g : unix group (default value: '_' = any value)

-u : unix user account (default value: '_default' = non-special users)

-x : print output as XML

13

Query configuration fileQuery configuration file

$ ./sbin/jwgen-ce-query.sh -h cclcgceli05.in2p3.fr -u atlas099

CEInstance 'cclcgceli05.in2p3.fr'

CEQueue 'long'

Group 'atlas'

User 'atlas099'

LogicalResourceSet = {dcache,xrootd}

OperatingSystem = sl5

ProfilesDirectory=/afs/in2p3.fr/grid/profiles/glite/3.2.2-0/SL5_64/WN64

Project = P_atlas_T2ana

ResourceCPUMax = max

ResourceFileSizeMax = max

ResourceMemoryMax = max

ResourceQueue = long

SubmissionPolicy = ACCEPT

14

Core package independent ofCore package independent of

grid middleware– based on JSDL– can be used with CREAM job control plug-in (BLAH) or Globus job

manager for example

LRMS– developed, tested and used with BQS– will now be used with Grid Engine

site policy– configuration keywords and templates are provided separately

15

ConclusionConclusion

JWGEN has proven to be useful with BQS, now with GE It can be reused with other grid middlewares, LRMS and

site policies

But…– configuration is of course more complex that the BLAH flat file– customization is not so easy to do (even if customizable files are

well identified)

We plan to propose JWGEN as an optional module for grid sites that need this kind of advanced features

16

Questions ?Questions ?