Derek Wright Computer Sciences Department, UW-Madison Lawrence Berkeley National Labs (LBNL) [email protected]

Derek WrightComputer Sciences Department, UW-

Madison Lawrence Berkeley National Labs (LBNL)

[email protected]://www.cs.wisc.edu/condor

http://sdm.lbl.gov

Condor COD (Computing On

Demand)Condor Week 5/5/2003

www.cs.wisc.edu/condor

What problem are we trying to solve?

› Some people want to run interactive, yet compute-intensive applications

› Jobs that take lots of compute power over a relatively short period of time

› They want to use batch computing resources, but need them right away

› Ideally, when they’re not in use, resources would go back to the batch system


Some example applications:

› A distributed build/compilation of a large software system

› A very complex spreadsheet that takes a lot of cycles when you press “recalculate”

› High-energy physics (HEP) “analysis” jobs

› Visualization tools for data-mining, rendering graphics, etc.


Batch Jobs

Compute Farm

User’s Workstation

Example application for COD

On-demandworkers

Idle nodes

Data

DisplayController

application


› Condor COD: “Computing on Demand” Use Condor to manage the batch

resources when they’re not in use by the interactive jobs

Allow the interactive jobs to come in with high priority and run instead of the batch job on any given resource

What’s the Condor solution?


Why did we have to change Condor for

that?› Doesn’t Condor already notice when

an interactive job starts on a CPU?

› Doesn’t Condor already provide checkpointing when that happens?

› Can’t I configure Condor to run whatever jobs I want with a higher priority on my own machines?


Well, yes… But that’s not good

enough…› Not all jobs can be checkpointed, and

even those that can take some time…

› We want this to be instantaneous, not waiting for the batch system to schedule tasks…

› You can configure Condor to run higher priority jobs, but the other jobs are kicked off the machine…


What’s new about COD?› “Checkpoint to swap space”

When a high-priority COD job appears, the lower-priority batch job is suspended

The COD job can run right away, while the batch job is suspended

Batch jobs (even those that can’t checkpoint) can resume instantly once there are no more active COD jobs


But wait, there’s more…

› The condor_startd can now manage multiple “claims” on each resource If any COD claim becomes active, the regular

Condor claim is automatically suspended Without an active COD, regular claim resumes

› There is a new command-line tool to request, activate, suspend, resume and release these claims

› There’s even a C++ object to do all of that, if you really want it…


COD claim-management commands

› Request: authorizes the user and returns a unique claim ID for future commands

› Activate: spawns an application on a given COD claim, with various options to define the application, job ID, etc Suspends any regular Condor job You can have multiple COD claims on a single

resource, and they can all be running simultaneously


COD commands (cont’d)

› Suspend: Given COD claim is suspended If there are no more active COD claims, a

regular Condor batch job can now run

› Resume: Given COD claim is resumed, suspending the Condor batch job (if any)

› Deactivate: Kill the application but hold onto the COD claim

› Release: Get rid of the COD claim itself


COD command protocol

› All commands use ClassAds Allows for a flexible protocol Excellent error propagation Can use existing ClassAd technology

› Similar to existing Condor protocol Separation of claiming from activation,

so you can have hot-spares, etc.


How does all of that solve the problem?

› The interactive COD application starts up, and goes out to claim some compute nodes

› Once the helper applications are in place and ready, these COD claims are suspended, allowing batch jobs to run

› When the interactive application has work, it can instantly suspend the batch jobs and resume the COD applications to perform the computations



Compute Farm

Step 1: Initial state

Idle nodes

Batch jobs

Idle nodes%



Compute Farm

Step 2: Application spawned

Idle nodes

Batch jobs

Idle nodes% fractal-gen –n 4

Controllerapplicationspawned



Compute Farm

Step 3: Compute node setup

Idle nodes

Batch jobs

request

activa

te

Claiming and initializing [4] compute

nodes for rendering…

Got reply from:c1.cluster.orgc6.cluster.orgc14.cluster.orgc17.cluster.org

SUCCESS!

On-demandworkers

On-demandworkers


% condor_cod_request –name c1.cluster.org \ –classad c1.outSuccessfully sent CA_REQUEST_CLAIM to startd at <128.105.143.14:55642>

Result ClassAd written to c1.outID of new claim is: “<128.105.143.14:55642>#1051656208#2”

% condor_cod_activate –keyword fractgen \ –id “<128.105.143.14:55642>#1051656208#2”Successfully sent CA_ACTIVATE_CLAIM to startd at <128.105.143.14:55642>

% …

Step 3: Commands used



Compute Farm

Step 4: “Checkpoint” to swap

Batch jobs

suspend

Idle nodesIdle nodesSuspended

worker

SELECT FRACTAL TYPE

<Mandelbrot>

(more user input…)



› Rendering application on each COD node is suspended while interactive tool waits for input

› The resources are now available for batch Condor jobs

% condor_cod_suspend \

–id “<128.105.143.14:55642>#1051656208#2”

Successfully sent CA_SUSPEND_CLAIM to startd at <128.105.143.14:55642>

% …



Compute Farm

Step 5: Batch jobs can run

Batchqueue

Batch jobs

Idle nodesIdle nodes

SPECIFY PARAMETERS

max_iterations: 400000

TL: -0.65865, -0.56254

BR: -0.45865, -0.71254

(more user input…)


Compute Farm

Step 6: Computation burst

Idle nodes

Batch jobs


resume

Suspendedbatch job

Interactiveworkers

On-demandworkers

CLICK <RENDER> TO VIEW YOUR FRACTAL…

RENDER



› Batch Condor jobs on COD nodes are suspended

› All COD rendering applications are resumed on each node

% condor_cod_resume \

–id “<128.105.143.14:55642>#1051656208#2”

Successfully sent CA_RESUME_CLAIM to startd at <128.105.143.14:55642>

% …


Compute Farm

Step 7: Results produced

Idle nodes

Batch jobs


Suspendedbatch job

Interactiveworkers

On-demandworkers

Data

Display


Compute Farm

Step 8: User input while batch work resumes

Idle nodesIdle nodes

Batch jobs


ZOOM BOX COORDINATES:

TL = -0.60301, -0.61087

BR = -0.58037, -0.62785

Suspendedworker

suspend


Compute Farm

Step 9: Computation burst #2

Idle nodes

Batch jobs

User’s Workstation Interactive

workers

Suspendedbatch job

On-demandworkers

resume

Data

Display

RENDER


Compute Farm

Step 10: Clean-up

Idle nodes

Batch jobs


release

Idle nodes

REALLY QUIT? Y/N

Releasing compute nodes…

4 nodes terminated successfully!



› The jobs are cleaned up, claims released, and resources returned to batch system

% condor_cod_release \

–id “<128.105.143.14:55642>#1051656208#2”

Successfully sent CA_RELEASE_CLAIM to startd at <128.105.143.14:55642>

State of claim when it was released: "Running"

% …


Other changes for COD:

› The condor_starter has been modified so that it can run jobs without communicating with a condor_shadow All the great job control features of

the starter without a shadow Starter can write its own UserLog Other useful features for COD


condor_status –cod› New “–cod” option to condor_status to view

COD claims in a Condor pool:

Name ID ClaimState TimeInState RemoteUser JobId Keyword

astro.cs.wi COD1 Idle 0+00:00:04 wright

chopin.cs.w COD1 Running 0+00:02:05 wright 3.0 fractgen

chopin.cs.w COD2 Suspended 0+00:10:21 wright 4.0 fractgen

Total Idle Running Suspended Vacating Killing

INTEL/LINUX 3 1 1 1 0 0

Total 3 1 1 1 0 0


What else could I use all these new features

for?› Short-running system administration tasks

that need quick access but don’t want to disturb the jobs in your batch system

› A “Grid Shell” A condor_starter that doesn’t need a

condor_shadow is a powerful job management environment that can monitor a job running under a “hostile” batch system on the grid


Future work› More ways to tell COD about your

application For now, you define important

attributes in your condor_config file and pre-stage the executables

› Ability to transfer files to and from a COD job at a remote machine We’ve already got the functionality in

Condor, so why rely on a shared filesystem or pre-staging?


More future work › Accounting for COD jobs

› Working with some real-world applications and integrating these new COD features Would the real users please stand up?

› Better “Grid Shell” support This is really a separate-yet-related

area of work…


How do you use COD?

› Upgrade to Condor version 6.5.3 or greater… COD is already included

› There will be a new section in the Condor manual (coming soon)

› If you need more help, ask the ever helpful [email protected]

› Find me at the BoF on Wednesday, 9am to Noon (room TBA)

Documents

Derek Wright Computer Sciences Department, UW-Madison Lawrence Berkeley National Labs (LBNL) [email protected]