What’s New in Work What’s New in Work QueueQueue

Michael Albrecht, University of Notre Dame

CCL Workshop, June 2012

2

Overview

New “Batch Job” SystemsMoab / ClusterMPI Queue

Work Queue EnhancementsHierarchical Work Queue

3

Batch Job Systems

Abstraction Layer Library for generic task submission Used by Makeflow

4

Moab & other Clusters

Similar to “SGE” Explicitly support Moab scheduler Support other, similar schedulers

Set name, submit and remove commands

moab

5

MPI Queue

Workflow

W WW WW W W W W

W W W W W

6

MPI Queue

Workflow

W W W W W

W W W W F

7

MPI Queue

Enables arbitrary computation on “MPI-only” clusters

Ranks 1-N talk to Rank 0, which acts as a “foreman”

Assumes shared parallel filesystem for cluster

API very similar to Work QueueFully supported by Batch Job/Makeflow

Work Queue

9

Work Queue is Wonderful

Easily harness 100’s-1000’s of cores

Combine multiple resources for one project

Dynamically scale computational resources

PrivateCluster

CampusCondor

Pool

PublicCloud

Provider

SharedSGE

Cluster

Makefile

Makeflow

Local Files and Programs

sge_submit_workers

W

W

W

ssh

WW

WW

W

Wv

W

condor_submit_workers

W

W

W

10

Work Queue has Limits

Bandwidth, file size, and computation length constrain potential number of workers

11

Work Queue has Limits

Natural parallelism of a workflow leaves extra workers idle

T1 T2 T3 T4

T0

T5

???W5W1 W2 W3 W4

12

Work Queue has Limits

Synchronous transfer leaves network resources idle, increases dispatch time

Add more indirection!

“All problems in computer science can be solved by another layer of indirection”

-David Wheeler

“…except for the problem of too many layers of indirection”

-Kevlin Henney

14

Hierarchical Work Queue

Master(Makeflow)

Worker Worker Worker Worker Worker Worker Worker Worker

15

Hierarchical Work Queue

Master(Makeflow)

Worker Worker Worker Worker Worker Worker Worker Worker

Foreman Foreman

16

Hierarchical Work Queue

Master(Makeflow)

Worker Worker Worker Worker Worker Worker Worker Worker

Foreman Foreman

Shared FS

17

Hierarchical Work Queue

Master(Makeflow)

Worker Worker Worker Worker Worker Worker Worker Worker

Foreman ForemanShared FS Shared FS

18

Shared FS

Hierarchical Work Queue

Master(Makeflow)

Worker Worker Worker Worker Worker Worker Worker Worker

Foreman ForemanShared FS Shared FS

Foreman Foreman

19

Hierarchical Work Queue

19

http://www.nd.edu/~cclhttp://www.nd.edu/~ccl

Coming

Soon!