Wolfgang Friebel, 15.10.2001 HEPiX Meeting Berkeley Installing and Running SGE at DESY (Zeuthen)

Wolfgang Friebel,15.10.2001HEPiX Meeting Berkeley

Installing and Running SGE at DESY (Zeuthen)

Oct 15, 2001 SGEEE Zeuthen 2

Introduction Motivations for using a batch system

more effective usage of available computers (e. g. more uniform load) usage of resources 24h/day assignment of resources according to policies (who gets how much CPU when) quicker execution of tasks (system knows most powerful least loaded nodes)

Our goal: You tell the batch system a script name and what you need in terms of disk space, memory, CPU timeThe batch system guarantees fastest possible turnaround

Could even be used to get xterm windows on least loaded machines for interactive use


Batch Systems Overview Condor targeted at using idle workstations (not used at DESY) NQS public domain and commercial versions, basic functionality.

Used for APE100 projects Loadleveler mostly found on IBM machines, used at DESY LSF popular, rich set of features, licensed software, used at DESY PBS public domain and commercial versions, origin: NASA

rich set of features, became popular recently, used in H1 Codine/GRD batch system similar to LSF in functionality, used in HERA-B

and for all farms at DESY Zeuthen SGE/SGEEE Sun Grid Engine (Enterprise Edition), open source

successors of Codine/GRD. Became the only batch system at Zeuthen (except for the legacy APE 100 batch system)


The old Batch System Concept Each group runs a separate cluster with separate

instances of GRD or Codine Project priorities within a group are maintained by

configuring several queues reflecting the priorities Queue names were named after priority, e.g. long, medium, short, idle, ... Could also be named according to task, e.g. simulation, production, test, ... Individuals had to obey group dependent rules to submit jobs

Priorities between different groups were realized by the cluster size (CPU power)

Urgent tasks were tried to carry out by asking other groups to temporarily use their cluster

Administrative overhead to enable accounts on machines Users had to adapt their batch jobs to the new environment

There were always heavily overloaded clusters next to machines with lots of idle CPU cycles


A new Scheme for Batch Processing Two factors led us design a new batch processing scheme

shortcomings of the old system, especially the non uniform usage pattern licensing situation, our GRD license ended, wanted to go to the open source

successor of GRD

One central batch system for all groups dynamic allocation of resources according to the current needs of groups more uniform configuration of batch nodes

Very few queue types basically only two types: Queue for ordinary batch jobs and idle queue most of the scheduling decisions based on other mechanisms (see below)

Resource requests for jobs determine queuing Resource definition based on the concept of complexes (explained later) User should request resources if the defaults are not well suited for the jobs

Bookkeeping of resources within the batch system


The Sun Grid Engine ComponentsComponents of the system

Queues contain information on number of jobs and job characteristics thatare allowed on a given host. Jobs need to fit into a queue to getexecuted. Queues are bound to specific hosts.

Resources Features of hosts or queues that are known to SGE. Resourceattributes are defined in so called (global, host, queue and user defined) complexes

Projectscontain lists of users (usersets) that are working together. Therelative importance to other projects may be defined using shares.

Policies Algorithms that define, which jobs are scheduled to which queuesand how the priority of running jobs has to be set. SGEEE knowsfunctional, share based, urgency based and override policies

Shares SGEEE can use a pool of tickets to determine the importance ofjobs. The pool of tickets owned by a project/job etc. is called share


Benefits Using the SGEEE Batch System For users:

jobs get executed on the most suitable (least loaded, fastest) machine fair scheduling according to defined sharing policies no one else can overuse the system and provoke system degradation users need no knowledge of host names where their jobs can run quick access to load parameters of all managed hosts

For administrators: one time allocation of resources to users, projects, groups no manual intervention to guarantee policies reconfiguration of the running system (to adapt to changing usage

pattern) easy monitoring of hosts and jobs


Policies for the Job handling within SGEEE Within SGEEE tickets are used to distribute the workload User based functional policy

Tickets are assigned to projects, users and jobs. More tickets mean higher priority and faster execution (if concurrent jobs are running on a CPU)

Share based policy Certain fractions of the system resources (shares) can be assigned to projects

and users. Projects and users receive that shares during a configurable moving time

window (e.g. CPU usage for a month based on usage during the past month)

Deadline policy By redistributing tickets the system can assign jobs an increasing weight to

meet a certain deadline. Can be used by authorized users only

Override policy Sysadmins can give additional tickets to jobs, users or projects to temporarily

adjust their relative importance.


Classes of Hosts and Users Submit Host node that is allowed to submit jobs (qsub) and query its

status Exec Host node that is allowed to run (and submit) jobs

Admin Host node from which admin commands may be issued

Master Host node controlling all SGE activity, collecting statusinformation, keeping access control lists etc.

A certain host can have any mixture of the roles above Administrator user that is allowed to fully control SGE

Operator user with admin privileges, who is not allowed tochange the queue configuration

Owner user that is allowed to suspend jobs in queues he ownsor disable owned queues

User can manipulate only his own jobs


The Zeuthen SGEEE Installation

SGEEE built from the source with AFS support Another system (SGE with AFS) was built for the HERA-B experiment

Two separate clusters (no mix of operating systems) 95 Linux nodes in default SGEEE cell Other Linux machines (public login) used as submit hosts 17 HP-UX nodes in cell hp

A cell is a separate pool of nodes controlled by a master node


The Zeuthen SGEEE Installation

In production since 9/2001 Smooth migration from the old system

Two batch systems were running in parallel for a limited time Coexistence of old queue configuration scheme and the new one

Ongoing tuning of the new system Initial goal was to reestablish functionality of the old system Now step by step changes towards a truly homogeneous system Initially some projects were bound to subgroups of hosts


Our Queue Concept

one queue per CPU with large time limit and low priority users have to specify at least a CPU time limit (usually much smaller) Users can request other resources (memory, disk) differing from default values

optionally a second queue that gets suspended as soon as there are jobs in the first queue (idle queue)

interactive use is possible because of low batch priority relation between jobs, users and projects is respected

because of sharing policies


Complexes within SGE Complexes are containers for resource definitions Resources can be requested by a batch job

You can have hard requests that need to be fulfilled (e.g. host architecture) Soft requests are fulfilled if possible

The actual value for some resource parameters is known Amount of available main memory or disk space can be used for decisions Arbitrary "load sensors" can be written to measure resource parameters

Resources can be reserved for the current job Parameters can be made "consumable". A portion of a requested resource gets

subtracted from the value of the currently available resource parameter

The most important parameters are known to SGEEE Parameters like CPU time, virtual free memory etc. are built in already To be used some of them need to be activated in the configuration


Our Complexes Concept

Users have to specify for a job Time limit (CPU time)

Users can request for a job A certain amount of virtual and real free memory The existence of one or two scratch disks(coming soon): The available free disk space for a given scratch disk To have a guaranteed amount of disk space reserved More hardware oriented features like:

Using only machines from a subcluster (farm) Run on a specific host (not recommended)


Experiences System is easily useable from a users point of view System is highly configurable (needs some time to find the

optimum policies to implement) System is very stable

crashing jobs mostly due to failing token renewal (our plugin procedure based on arc and batchtkauth)

other failures due to missing (on purpose!) path aliases for the automounter

System adapts dynamically process priority to meet share policies or to keep up with changing policies

SGE(EE) maintainers are very active and keep implementing new ideas

quick incorporation of patches, reported bugs get fixed asap.


Advanced Use of SGEEE

Using the perl API every aspect of the batch system is accessible through the perl API the perl API is accessible after use SGE; in perl scripts there is almost no documentation but a few sample scripts in

/afs/ifh.de/user/f/friebel/public and in /afs/ifh.de/products/source/gridengine/source/experimental/perlgui

Using the load information reported by SGEEE each host reports a number of load values to the master host

(qmaster) there is a default set of load parameters that are always reported further parameters can be reported by writing load sensors qhost is a simple interface to display that information a powerful monitoring system could be built around that feature,

which is based on the "Performance Data Collection" (PDC) built in subsystem


Conclusions Ease of installation from source Access to source code Chance of integration into a monitoring system API for C and Perl Excellent load balancing mechanisms Managing the requests of concurrent groups Mechanisms for recovery from machine crashes Fallback solutions for dying daemons Weakest point is AFS integration and Token prolongation

mechanism (basically the same code as for Loadleveler and for older LSF versions)


Conclusions SGEEE has all ingredients to build a company wide batch

infrastructure Allocation of resources according to policies ranging from departmental policies

to individual user policies Dynamic adjustment of priorities for running jobs to meet policies Supports interactive jobs, array jobs, parallel jobs Can be used with Kerberos (4 and 5) and AFS

SGEEE is open source maintained by Sun Getting deeper knowledge by studying the code Can enhance the code (examples: more schedulers, tighter AFS integration,

monitoring only daemons) Code is centrally maintained by a core developer team

Could play a more important role in HEP (component of a grid environment, open industry grade batch system as recommended solution within HEPiX?)


References http://gridengine.sunsource.net/servlets/ProjectSource

Download Page for source code of SGE(EE) http://www.arl.hpc.mil/docs/grd/

lots of docs from raytheon http://supportforum.Sun.COM/gridengine/

Supportforum, Mailinglists http://hoover.hpac.tudelft.nl/cugs98cd/S98PROC/AUTHORS/

FERSTL/INDEX.HTM GRD on a Conference 1998 http://www-zeuthen.desy.de/computing/services/batch/

Zeuthen pages with URL to the reference manual http://www-zeuthen.desy.de/…/batch/sge53.pdf

The SGEEE reference manual, user and installation guide


Technical Details of SGEEE(not presented)

Submitting Jobs The graphical interface qmon Job submission and file systems Sample job script Advanced usage of qsub Abnormal job termination


Submitting Jobs Requirements for submitting jobs

have a valid token (verify with tokens), otherwise obtain a new one (klog) ensure that in your .[t]cshrc or .zshrc no commands are executed that need a

terminal (tty) (users have often a stty command in their startup scripts) you are within batch if the env variable JOB_NAME is set or if the env variable

ENVIRONMENT is set to BATCH

Submitting a job specify what resources you need (-l option) and what script should be executedqsub -l t=1:00:00 job_script in the simplest case the job script contains 1 line, the name of the executable many more options available alternatively use the graphical interface to submit jobsqmon &


The Submit Window of qmon


Job Submission and File Systems Current working directory

the directory from where the qsub command was called. STDOUT and STDERR of a job go into files that are created in $HOME. Because of quota limits and archiving policies that is not recommended.

With the -cwd option to qsub the files get created in the current working directory. For performance reasons that should be on a local file system

If cwd is in NFS space, the batch system must not use the real mount point but be translated according to /usr/SGE/default/common/sge_aliases. As every job stores the full info from sge_aliases, it is of advantage to get rid of that file and discourage the use of NFS as current working directory

If required, create your own $HOME/.sge_aliases file

Local file space (Zeuthen policies) /usr1/tmp is guaranteed to exist on all linux nodes and has typically > 10GB /data exists on some linux nodes and has typically > 15GB capacity. A job can

request the existence of /data by -l datadir $TMP[DIR] is a unique directory below /usr1/tmp, that gets erased at the end of

the job. Normal jobs should make use of that mechanism if possible


A Simple Job Script#!/bin/zsh

#$ -S /bin/zsh

#

#$ -l t=0:30:00

#$ -j y

WORKDIR=/usr1/tmp/$LOGNAME/$JOB_ID

DATADIR=/net/ilos/h1data7

echo using working directory $WORKDIR

mkdir -p $WORKDIR

cp $DATADIR/large_input $WORKDIR

cd $WORKDIR

h1_reco

cp large_out $DATADIR

if [ -s large_out = -s $DATADIR/large_out ]; then

cd; rm -r $WORKDIR

fi

otherwise the default shell would be used

the time limit for this job


Advanced Usage of qsub

Option files instead of giving qsub options on the command line, users may store those

in .sge_projects files in their $HOME or current working directories content of a sample .sge_projects file:cwd -S /usr/local/bin/perl -j y -l t=24:00:00

Array jobs SGE allows to schedule n identical jobs with one qsub call using the –t option:qsub -t 1-10 array_job_script within the script use the variable SGE_TASK_ID to select different inputs and

write to distinct output files (SGE_TASK_ID is 1...10 in the example above)

Conditional job execution jobs can be scheduled to wait for dependent jobs to successfully finish (rc=0) jobs can be submitted in hold state (needs to be released by user or operator) jobs can be told not to start before a given date start dependent jobs on the same host (using qalter -q $QUEUE ... within script)


Abnormal Job Termination Termination because of CPU limit exceeded

jobs get an XCPU signal that can be catched by the job. In that case termination procedures can be executed, before the SIGKILL signal is sent

SIGKILL will be sent a few minutes after XCPU was sent. It cannot be catched.

Restart after execution host crashes if a host crashes when a given job is running, the job will be restarted. In

that case the variable RESTARTED is set to 1 The job will be reexecuted from the beginning on any free host. If the job

can be restarted using some results achieved so far, then the variable RESTARTED can be checked. The job can be forced to be executed on the same host by inserting

qalter -q $QUEUE $JOB_ID

literally in the job script

Signaling the end of the job with the qsub option -notify a SIGUSR1 signal is sent to the job a few

minutes before the job is suspended or terminated

Documents

Wolfgang Friebel, 15.10.2001 HEPiX Meeting Berkeley Installing and Running SGE at DESY (Zeuthen)