Transcript
Page 1: Matchmaking: An extensible framework for distributed resource management

Cluster Computing 2 (1999) 129–138 129

Matchmaking: An extensible framework for distributed resourcemanagement

Rajesh Raman, Miron Livny and Marv Solomon

Department of Computer Science, University of Wisconsin, 1210 West Dayton Street, Madison, WI 53703, USA

Federated distributed systems present new challenges to resource management. Conventional resource managers are based on arelatively static resource model and a centralized allocator that assigns resources to customers. Distributed environments, particularlythose built to support high-throughput computing (HTC), are often characterized by distributed management and distributed ownership.Distributed management introduces resource heterogeneity: Not only the set of available resources, but even the set of resource typesis constantly changing. Distributed ownership introduces policy heterogeneity: Each resource may have its own idiosyncratic allocationpolicy. To address these problems, we designed and implemented the Matchmaking resource management framework. Customers andresources are all described by classified advertisements (classads) written in a simple but powerful formal language that describes theirattributes and allocation policies. A Matchmaker server uses a policy-independent matching operation to discover pairings. It notifiesthe parties to the match, which use a separate, bilateral claiming protocol to confirm the allocation. The resulting framework is robust,scalable and flexible, and can evolve with changing resources. Matchmaking is the core of the Condor High Throughput ComputingSystem developed at the University of Wisconsin – Madison. Condor is a production-quality system used by scientists and engineers atsites around the world. Condor derives much of its flexibility, robustness and efficiency from the matchmaking architecture. We describethe use of matchmaking in Condor, presenting several examples that illustrate its flexibility and expressiveness.

1. Introduction

The computing infrastructure of modern organizationsroutinely consists of large numbers of powerful and inex-pensive commodity computing resources. These resourcesare usually distributively owned, i.e., assigned to smallgroups or single users who use them for their daily activi-ties, and are therefore effectively considered as “owners” ofthe resource in question. Studies indicate that resources insuch environments exhibit an average utilization of only30–40%, and are therefore severely underutilized [14].These environments therefore present an opportunity to in-crease the effective computing capacity of organizations,but scavenging excess computational capacity from non-dedicated environments presents several challenges.

In distributively owned environments, resource ownersare willing to “donate” their resources to other computa-tions only if they have complete control over the usagepolicy of the resource. For example, a workstation ownermay stipulate that the workstation may be used for othercomputations only if it is not otherwise being used (as de-termined by load average and console activity) and appli-cations submitted by certain users should never be run onthe workstation. In our experience, the sociological natureof ownership results in policies that vary from indifferentto idiosyncratic. The presence of such policies (defined ona per resource basis) results in policy heterogeneity. Theramification for the resource management framework is thatresources can behave unpredictably and transit from avail-able to unavailable states without advance notice.

Policy heterogeneity is present not only among resourcesof the pool, but customers as well. Like resource owners,

customers of the system define policies which describe con-straints and preferences on the the kinds of resources theyrequire to run applications. In our model, we allow re-source owners and customers to change their policies atrun-time.

A different (and more obvious) dimension of heterogene-ity in the pool exists due to the physical characteristics ofresources that constitute the pool. A resource pool may si-multaneously contain resources which differ in their types(such as workstation, tape drive, and software license) andattributes (such as physical memory, operating system andarchitecture). Furthermore, resources may be changed asobsolete or defective hardware and software componentsare repaired, replaced, or permanently removed while sys-tems with new types and attributes are added to the pool.Thus, the resource pool exhibits both resource heterogeneityand evolution, which places additional requirements on theresource management framework. For example, the seman-tics of what it means to allocate a resource to a customermay vary across resources.

Policy and resource heterogeneity imply that conven-tional resource management systems cannot, in the generalcase, be used to manage distributively owned resources.Conventional resource management systems use a systemmodel to describe the behavior of the resources being man-aged, and a centralized scheduler to control their allocation.The system model provides information to the allocator re-garding the availability and properties of resources at anypoint in time. This information is used by the allocator toallocate resources to customers to optimize a performancemetric. This paradigm works well for high-performancecomputing environments, which use dedicated resources

Baltzer Science Publishers BV

Page 2: Matchmaking: An extensible framework for distributed resource management

130 R. Raman et al. / Matchmaking

that need to be scheduled efficiently to meet response timeconstraints placed by customers.

Policy and resource heterogeneity imply that it is im-possible to formulate a monolithic system model. There istherefore a need for a resource management paradigm thatdoes not require such a model, and that can operate in anenvironment where resource owners and customers can dy-namically define their own system and allocation models.Faced with these problems, we developed and implementedthe classified advertisement (classad) matchmaking frame-work, a flexible and general approach to resource manage-ment in distributed environment with decentralized owner-ship of resources.

The two distinguishing aspects of matchmaking are:

1. Separation of allocation into distinct matching andclaiming phases, which facilitates the definition of al-ternate system and allocation models.

2. Employment of a simple, yet powerful, specification lan-guage to represent the constraints and attributes of re-sources and customers in the pool. The language allowsthe expression of many policies in a straightforward andintuitive manner, and is robust in the face of failures anderrors.

We found the matchmaking paradigm to work extremelywell in an environment where a large number of dis-similar resources transit between available and unavail-able states without advance notice and where resourcesmay be available to some entities, and unavailable to oth-ers. Such an environment has to employ opportunisticscheduling: Resources are used as soon as they becomeavailable and applications are migrated when resourcesneed to be preempted. The applications that most ben-efit from opportunistic scheduling are those that requirehigh throughput rather than high performance. Traditionalhigh-performance applications measure their performancein instantaneous metrics like floating point operations persecond, while high throughput applications usually usesuch application-specific metrics as weather simulations perweek or crystal configurations per year. In other words,rather than MIPS (millions of instructions per second), theperformance of high-throughput applications might be mea-sured in TIPYs (trillions of instructions per year).

The framework was designed to solve real problems en-countered in the deployment of Condor, a high throughputcomputing system developed at the University of Wisconsin– Madison. Condor is heavily used by scientists at numer-ous sites around the world. It derives much of its robustnessand efficiency from the matchmaking architecture.

The components and architecture of the framework arepresented in section 2. The classad language, which in-tegrates the framework, is presented in section 3. Detailsof the matchmaking process and the operation of princi-pals in a matchmaking environment are furnished in sec-tion 4. We then discuss advantages of the matchmakingparadigm in section 5, and provide a real-world example

of the matchmaking architecture by presenting the Condorsystem’s architecture in section 6. Examples which demon-strate the flexibility and expressiveness of matchmaking arepresented in section 7. Related work on heterogeneous re-source management is presented in section 8. The paperconcludes with section 9, which summarizes and discussesfuture research directions.

2. The matchmaking framework

The basic idea of matchmaking is simple: Entities whichprovide and require services advertise their characteris-tics, constraints and preferences in classified advertisements(classads) to a matchmaking service, or matchmaker. Thematchmaker matches classads of servers and customers sothat the constraints specified in the respective advertise-ments are satisfied, and notifies the relevant parties of thematch. The responsibility of the matchmaker then ceaseswith respect to the match. The matched parties establishcontact, possibly negotiate further terms, and cooperate toperform the desired service.

Each entity in the pool sends a classad containing a de-scription of its attributes and constraints to a matchmaker,which therefore describes the entity’s policy. The match-maker determines the compatibility of advertising entitiesby checking if the constraints and preferences expressed inthe respective classads are satisfied. If a match is found,the matched entities are deemed to be compatible, and thepolicies of both entities must have been satisfied. Thus, thematchmaker implements the policies of advertising entitiesas part of the matchmaking process.

A matchmaking framework consists of five distinct com-ponents.

1. The classad specification defines a language for express-ing characteristics and constraints, and the semantics ofevaluating these attributes.

2. The advertising protocol defines basic conventions re-garding what a matchmaker expects to find in a classadif the ad is to be included in the matchmaking process,and how the matchmaker expects to receive the ad fromthe advertiser.

3. The matchmaking algorithm defines how the contentsof ads and the state of system relate to the outcomeof the matchmaking process. In addition to the entitypolicies specified in classads supplied by entities, thematchmaker may impose a policy to arbitrate amongthe entities of the pool, which defines the system policyfor that specific environment.

4. The matchmaking protocol (or match notification proto-col) defines how matched entities are notified and whatinformation they are given in case of a match.

5. The claiming protocol defines what actions the matchedentities take to enable discharge of service.

Page 3: Matchmaking: An extensible framework for distributed resource management

R. Raman et al. / Matchmaking 131

There are several noteworthy aspects of this approachthat distinguish it from conventional resource allocationmodels.

• Conventional resource management systems only allowcustomers to impose constraints on the type of ser-vices they require. Our mechanism also allows serviceproviders to express constraints on the customers theyare willing to serve.

• The semantics of a matchmaker identifying a match be-tween entities A and B is not “allocating A to B”. Rather,a match results in a mutual introduction of the two enti-ties, who may activate a separate claiming protocol, notinvolving the matchmaker, to complete the allocation.Either entity may choose to not proceed further and re-ject the introduction altogether. Thus, a match is to beconstrued as a “hint”.

• The constraints and preferences expressed by the var-ious principals of the environment collectively definethe policy of the environment. Thus, a classad-basedmatchmaking framework is a distributed policy frame-work where the “policy fragments” of several differentprincipals are collated and “implemented” by the match-maker.

The details of the framework are now discussed in fur-ther detail.

3. The classad language

The classad language is a flexible and extensible datamodel that can be used to represent arbitrary services andconstraints. There are four principal language featureswhich allow the use of classads as a distributed policy lan-guage.

1. The language employs a semi-structured data model[15]. Because there is no concept of a “schema” inherentto classads, heterogeneous entities may enter, representthemselves, participate, and leave the system with noneed for schema evolution.

2. The classad language is dynamically typed to facilitatean open system of heterogeneous resources. Each valuehas a type, but the type of an expression is only deter-mined when it is evaluated.

3. The inclusion of “exceptional” undefined and error val-ues and operator semantics for handling them makes theevaluation of expressions robust in the face of hetero-geneity and uncertainty inherent in large open systems.

4. The scoping features of the language allow expressionsto refer to attributes in classads of candidate matchesthat do not yet exist. This feature allows “late binding”of attribute references, and is heavily exploited duringmatchmaking as will be explained in section 4.

[Type = "Machine";Activity = "Idle";KeybrdIdle = ’00:23:12’; // h:m:sDisk = 323.4M; // mbytesMemory = 64M; // mbytesState = "Unclaimed";LoadAvg = 0.042969;Mips = 104;Arch = "INTEL";OpSys = "SOLARIS251";KFlops = 21893;Name = "foo.cs.wisc.edu";Rank = other.Owner=="raman"

? 1 : 0;Constraint = other.Type == "Job"

&& self.LoadAvg < 0.3&& KeybrdIdle>’00:15’&& Memory -other.ImageSize>=15M

]

Figure 1. A classad describing a workstation.

[Type = "Job";QDate = ’Mon Jan 11 10:53:31

1999 (CST) -06:00’;CompletionDate = undefined;Owner = "raman";Cmd = "run_sim";WantRemoteSyscalls = true;WantCheckpoint = true;Iwd = "/usr/raman/sim2";Args = "-Q 17 3200 10";Memory = 31m;Rank = KFlops/1E3 +

other.Memory/32;Constraint = other.Type == "Machine"

&& Arch=="INTEL"&& OpSys=="SOLARIS251"

]

Figure 2. A classad describing a submitted job.

A classad1 is a set of distinctly named expressions,which are called attributes. Each expression may consistof literals, calls to built-in functions, and attribute refer-ences composed with unary, binary and ternary operatorssimilar to those found in programming languages such asC [11], C++ [18] and Java [5]. Expressions can also re-fer to other attributes, as in “Rank >= 10”. The classadlanguage also has list and classad construction operatorswhich allows arbitrary nesting of these aggregate struc-tures.

For example, figure 1 shows a classad that describesa workstation in a Condor [3,13] pool at the University

1 Similar structures have been called records, dictionaries, and frames inother contexts.

Page 4: Matchmaking: An extensible framework for distributed resource management

132 R. Raman et al. / Matchmaking

of Wisconsin.2 The Constraint attribute indicates thatthe workstation only runs jobs when it is idle (as deter-mined by load average and keyboard idle time) and if thejob’s image size is 15 megabytes less than the physicalmemory size of the machine, with a preference for jobssubmitted by user “raman”. Similarly, figure 2 describesa job which has the policy of running only on INTELmachines running the Solaris 2.5.1 operating system. TheRank expression in the job classad expresses a preferencefor machines with larger memories and higher KFlops rat-ings.

4. The matchmaking process

We now describe specific actions taken by the principalsof a matchmaking framework.

4.1. Operation of entities in the matchmaking framework

Providers and customers of services construct classadsdescribing their attributes and requirements and send themto the Matchmaker (step 1 in figure 3). These classadsmust be constructed to conform to the advertising protocolspecified by the matchmaker, which attaches a meaning tosome attributes. For example, the advertising protocol mayspecify that the attribute Constraint indicates compati-bility and the attribute Rank measures the desirability of amatch (see figures 1 and 2). The advertising protocol alsospecifies how the entities send the classads to the match-maker.

The matchmaker then invokes a matchmaking algorithmby which matches are identified (step 2), which involvesfinding resource-customer classad pairs which satisfy theconstraints and preferences of resource owners and cus-tomers. This step is explained further in section 4.

After the matching phase, the matchmaker invokes amatchmaking protocol (or match notification protocol) tonotify relevant parties (step 3) and sends them the matchingads. The customer and server contact each other directly,using a claiming protocol to establish a working relation-ship with the provider (step 4). It is important to note thatidentifying a match and invoking the matchmaking protocoldoes not immediately grant service to a customer. Rather,the match is a mutual introduction between the advertisingentities.

4.2. Operation of the matchmaker

In this section, we describe a high-level view of thematchmaker’s operation. The process of matchmaking in-volves two procedures:

• constructing pairs of candidate matches, and

2 Examples in this paper are adapted from actual ads in use in a workingCondor installation. They have been edited slightly for clarity and toillustrate features of the classad mechanism.

Figure 3. Actions involved in the matchmaking process.

• determining if a candidate match is valid as defined bythe relevant policies.

The classad language serves as a key mechanism that isused in both the above steps. Expressions in the languageare used to rank entity classads in a particular order, so thatcertain kinds of policies (such as those which incorporatea notion of priority) can be enforced. The language is alsoused to determine if the entities involved in a candidatematch are indeed compatible. We therefore briefly discussclassad expression evaluation, which plays an importantrole in matchmaking.

4.2.1. Expression evaluation in the classad languageExpression evaluation is a fundamental operation of the

matchmaker. For the purposes of matchmaking, classad ex-pression evaluation always takes place in an “evaluation en-vironment” which represents a candidate match. The func-tion of this environment is to let the expressions of the re-source and customer classads access each other’s attributes.The design of the classad language allows this environmentitself to be represented as a classad which contains the clas-sads of the entities participating in the candidate match.Thus, when testing for a match, the matchmaker creates a“top-level template” classad into which classads of entitieswill be “plugged in” by inserting them as attributes. Fig-ure 4 illustrates such an environment where the classadsof a workstation and a job have been inserted as attributesnamed “ad” in the “customer” and “resource” scopes of thetop-level created by the matchmaker.

In the environment referred to above, the matchmakerhas created attributes named customer and resource,each of which contains an attribute named other. Thescoping features of the language result in the semanticsthat an attribute reference made from either customer or re-source classad of the form “other.attribute-name” refersto an attribute named attribute-name of the other ad. Inaddition, every classad has a built-in attribute self whichevaluates to the innermost classad containing the refer-ence, so the reference “self.attribute-name” refers to anattribute of the ad containing the reference. If neitherself nor other is mentioned explicitly, the evaluationmechanism assumes the self prefix. For example, in theConstraint of the job ad in figure 1, the sub-expression“self.LoadAvg < 0.3” expresses the requirement thatthe server’s load average be sufficiently small. The expres-sion could also have been written “LoadAvg < 0.3”).

Page 5: Matchmaking: An extensible framework for distributed resource management

R. Raman et al. / Matchmaking 133

[customer =[other = .resource.ad;ad =[Type = "Job";Owner = "raman";Executable = "a.out";Imagesize = 10M;Rank = other.Memory * 10 +

other.KFlops;Constraint = Arch=="INTEL" &&

OpSys=="SOLARIS26"]

];resource =[other = .customer.ad;ad =[Type = "Machine";Arch = "INTEL";OpSys = "SOLARIS26";Name = "foo.cs.wisc.edu";Memory = 128M;KFlops = 21241;LoadAvg = 0.1;KbdIdle = ’00:20:12’;Rank = 1 / other.ImageSize;Constraint = KbdIdle > ’00:15:00’ &&

other.Owner == "raman"]

]]

Figure 4. A classad environment for checking matches.

A reference to a non-existent attribute evaluates to theconstant undefined. Most operators are “strict” with re-spect to this value – if either operand is undefined, theresult is undefined. In particular, comparison operators arestrict, so that

other.Memory > 32,other.Memory == 32,other.Memory != 32,

and

!(other.Memory == 32)

all evaluate to undefined if the target classad has noMemory attribute. The Boolean operators || and && arenon-strict on both arguments, so that

Mips >= 10 || Kflops >= 1000

evaluates to true whenever either of the attributes Mipsor Kflops exists and satisfies the indicated bound. Thereare also non-strict operators is and isnt, which alwaysreturn Boolean results (not undefined), allowing explicitcomparisons to the constant undefined as in

other.Memory is undefined|| other.Memory < 32.

The matchmaking algorithm effectively treats undefined asfalse, so the match fails if the Constraint evaluates toundefined. The three-valued logic used in Boolean expres-sions supports natural expression of constraints on objectswhose types are only partially known a priori.

4.2.2. Matchmakers in generalThe classad in figure 4 constructed for Matchmaking

provides a mechanism for testing matches. The actualmatches that are created also depend on the policy of theprincipals involved in the system and the matchmaking al-gorithm encapsulated in the matchmaker.

A simple matchmaker needs only to create candidatematch pairs and check if the constraints of the involved en-tities are satisfied by evaluating the appropriate expressions.Matchmakers for special environments may be constructedto cater to specific situations, or implement sophisticatedpolicies. For example, a matchmaker may use the prefer-ences expressed by entities to establish stable-marriage [7]matching solutions, or implement fair-share algorithms forcustomers.

5. Advantages of matchmaking

The advantages of matchmaking stem from the use of theextensible classad language, and the separation of matchingand claiming as distinct operations of an allocation. Wediscuss these two issues in further detail.

5.1. Flexibility of using classads as a specificationlanguage

An important advantage of the classad-based matchmak-ing framework is the generality that the language affords.The semi-structured nature of the language’s data modelcontains the classad’s schema information in the classad it-self. The model therefore allows the representation of manydifferent kinds of entities and policies without complicatingthe matchmaker, which only has to evaluate certain well-known attributes to determine compatibility and preference.The constraints expressed in classads by entities are, in asense, queries over the classads of other entities. Thus, theclassad model unifies schema, data and query into a sin-gle specification. The formalism can therefore accomodatedifferent kinds of matchmakers and matchmaking seman-tics without requiring any modification to the underlyinglanguage.

The classad expression language itself includes abstrac-tions which ease the construction of common policies thatdepend on numeric, string and temporal attributes of enti-ties. In addition to the exceptional undefined and error val-ues discussed previously, the language has built-in types forbooleans, numbers (integer and floating-point), strings, time(time stamps and time intervals) and lists. The operators

Page 6: Matchmaking: An extensible framework for distributed resource management

134 R. Raman et al. / Matchmaking

of the language are meaningfully defined over these valueswhenever possible facilitating useful operations, such ascomparison of time stamps or addition of a time stamp anda time interval. Other useful operations, such as obtainingthe current time, pattern matching over strings and mem-bership tests in lists are provided through calls to built-infunctions. We have found the language to be very expres-sive in defining complex policies that, for example, dependon the time of day or which affect only particular users.

5.2. Separation of matching and claiming

The separation of matching and claiming has severalbeneficial properties.

Weak consistency requirements. Since the state of ser-vice providers and requesters may be continuouslychanging, there is a possibility that the matchmakermade a match with a stale advertisement. Claimingallows the provider and customer to verify their con-straints with respect to their current state. This tolerationof weak consistency makes the remainder of the systemsignificantly simpler, more robust, and more efficient.

Authentication. The claiming protocol may use crypto-graphic techniques for the provider and customer to con-vince each other of their identities. A challenge-responsehandshake can be added to the claiming protocol at verylittle cost.

Bilateral specialization. In dynamic heterogeneous envi-ronments, it is not possible to write a matchmaker thatis aware of the specifics of allocating all the differentkinds of resources that may be added to the environ-ment. Indeed, the myriad kinds of resources alreadypresent in the environment may itself present the prob-lem of packing all the resource specific allocation codein the matchmaker.

By pushing the establishment of allocation to theclaiming stage, the details of allocation are containedin the entities which really need to interact with specifickinds of providers and customers. The matchmaker maybe written as a general service which does not dependon the kinds of services, customers and resources thatare being matched.

Bilateral specification implies that since the systemdoes not assume a single monolithic or static alloca-tion model, the allocation models are supplied by theentities involved in providing and using services. Thematchmaking framework thus allows several dissimilar“allocation models” to coexist in the same resource man-agement environment.

End-to-end verification [17]. The principals involved in amatch are themselves responsible for establishing, main-taining and servicing a match. With the exception ofsome accounting information, the matchmaker does notneed to retain any state about the match, a fact that sim-plifies recovery in case of failure and makes the systemmore scalable.

Figure 5. Architecture of the Condor system.

6. Matchmaking at work: The Condor system

The matchmaker we now describe was created for usein the Condor Resource Management Environment to testthe efficacy of the matchmaking paradigm. The architectureof the Condor system follows the matchmaking architectureillustrated in figure 5, and has been designed to operate in anenvironment composed of heterogenous and distributivelyowned resources.

Condor is a high throughput computing (HTC) envi-ronment that can manage large heterogeneous collectionsof distributively owned resources. The architecture of thesystem is structured to provide sophisticated resource man-agement services at the resource, customer and applicationlevels to both sequential and parallel applications [16]. Thissection briefly describes aspects of the Condor system thatare relevant to the problem of matchmaking.3

Resources in the Condor system (computers capable ofrunning Condor jobs) are represented by Worker Agents(WAs),4 which are responsible for enforcing the policiesstipulated by resource owners. A WA periodically probesthe resource to determine its current state, and encapsulatesthis information in a classad along with the owner’s usagepolicy.

Customers of Condor are represented by CustomerAgents (CAs), which maintain per-customer queues of sub-mitted jobs (represented as classads). Matching is coordi-nated by a central manager, which consists of three entities,a collector, an accountant, and a negotiator.

3 More information about the project and the system may be found athttp://www.cs.wisc.edu/condor/.

4 The current Condor system uses slightly different terminology, for histor-ical reasons. What we call a worker agent here is called a start daemon(startd) in Condor, and what we call a customer agent is called a sched-uler daemon (schedd).

Page 7: Matchmaking: An extensible framework for distributed resource management

R. Raman et al. / Matchmaking 135

Each CA periodically sends the collector submitter clas-sads describing users who have submitted jobs. The WAsalso send the collector worker ads describing their state.The collector only stores the most recent ad for each workerand each submitter. These ads conform to an advertisingprotocol that states that every classad should include ex-pressions named Constraint and Rank, as discussedpreviously. The protocol also requires the advertising par-ties to include “contact addresses” with their ads, and allowsa WA to include an “authorization ticket” in each workerad. This advertisement process is illustrated as step 1 infigure 5.

Periodically, the negotiator enters a negotiation cycle. Itretrieves from the collector the current ad for each workerand submitter (step 2). It then sorts customers by priori-ties obtained from the accountant (step 3). The accountantassigns priorities to customers based on past usage of re-sources. It then cycles through the submitters in priorityorder and contacts CAs, requesting job classads from them(step 4). The negotiator matches each job ad with a com-patible worker ad. Since the notion of “compatible” is com-pletely determined by Constraint expressions, classadsmay be matched in a general manner. The Rank expres-sions are used as goodness metrics to identify the moredesirable among the compatible matches – the matchmakeralways matches a customer with the highest ranking avail-able resource, breaking ties with the resource’s rank value.

When the negotiator determines that two classads match,it removes the worker ad from its set of available workersand invokes the matchmaking protocol to contact the ac-countant and the matched principals at the contact addressesspecified in their classads and send them each other’s clas-sads (step 5). The manager also gives the CA the autho-rization ticket supplied by the WA.

The CA then follows the claiming protocol by contactingthe WA and sending the authorization ticket (step 6). TheWA accepts the resource request only if the ticket matchesthe one that it gave the collector, and the request matchesthe WA’s constraints with respect to the updated state ofthe request and resource, which may have changed sincethe last advertisement. If the request is accepted, the work-station runs the customer’s job and informs the accountantof the resources used by advertising itself as an “occu-pied” resource. When the CA finishes using the resource,it relinquishes the claim, and the WA advertises itself asunclaimed by sending a new ad to the collector. The WAmay also send an ad when it starts running the job, indi-cating that although the workstation is currently busy, it isstill interested in hearing from higher priority customers.The specification of what constitutes “higher priority” iscompletely under the control of the WA.

The central manager is highly fault tolerant. A very sim-ple Condor master process ensures that a collector processand a negotiator process are always running. If the col-lector should die, a new one started by the master willquickly learn of the states of all the CAs and WAs fromtheir periodic update messages. The negotiator regenerates

its lists of ads from the collector at the start of each ne-gotiation cycle. Only the accountant maintains persistentstate.5 Established associations between CAs and WAs arenot affected by a malfunction of the central manager.

Classads are used for other purposes in Condor as well.All entities are represented with classads, as are queries sub-mitted by various administrative and user tools. “One-waymatching” protocols are used to find all objects matching agiven pattern. For example, there are tools to check on thestatus of job queues and browse existing resources.

7. Examples

7.1. Workstation access control

Figure 6 shows a classad that describes a workstationand demonstrates the flexibility of the mechanism in ex-pressing fairly sophisticated policies. The Constraintattribute indicates that the workstation is never willing torun applications submitted by users “rival” and “riffraff”, itis always willing to run the jobs of members of the researchgroup, friends may use the resource only if the workstationis idle (as determined by keyboard activity and load aver-age), and others may only use the workstation at night. TheRank expression states that research jobs have higher pri-ority than friends’ jobs, which in turn have higher prioritythan other jobs.

7.2. Time-dependent resource preference

Customers may incorporate environment specific infor-mation to improve the quality of service delivered to theirapplications. For example, many of the university’s work-stations which are used for instructional purposes existin computer laboratories that are locked during the night.Thus, it is beneficial for applications to run on these ma-chines after hours, as they will not be preempted by ma-chine owners during this time.

Figure 7 describes a job which has the policy of run-ning only on INTEL machines with sufficient memory anddisk space running the Solaris 2.5.1 operating system. Inaddition, the Rank expression in the job classad expressesa preference for running on instructional machines duringthe night over running on a machine that has been idle fora long time (which is therefore likely to remain unused),which is in turn preferred over running on any other ma-chine.

7.3. Time-dependent resource constraints

We now present an example in which a customer variesthe resource constraint over time. In the example illustratedin figure 8, the customer waits for up to two hours for aresource with at least one gigabyte of memory. If a match

5 The accountant is currently a library of functions invoked by the nego-tiator and collector, which store usage information in a file on disk.

Page 8: Matchmaking: An extensible framework for distributed resource management

136 R. Raman et al. / Matchmaking

[Type = "Machine";Activity = "Idle";KeybrdIdle = ’00:23:12’; // h:m:sDisk = 323.4m; // mbytesMemory = 64m; // mbytesState = "Unclaimed";LoadAvg = 0.042969;Mips = 104;Arch = "INTEL";OpSys = "SOLARIS251";KFlops = 21893;Name = "foo.cs.wisc.edu";ResearchGp = { "raman", "miron",

"solomon" };Friends = { "calvin", "hobbes" };Untrusted = { "rival", "riffraff" };Rank = member(other.Owner,

ResearchGp) ? 10 :member(other.Owner,Friends) ? 1 : 0;

Constraint = !member(other.Owner,Untrusted) &&

Rank>=10 ? true :Rank>0 ?LoadAvg < 0.3 &&

KeybrdIdle>’00:15’: DayTime()<’8:00’ ||

DayTime()>’18:00’]

Figure 6. Workstation access control.

[Type = "Job";CompletionDate = undefined;RemoteSyscalls = true;Checkpoint = true;QDate = ’Mon Jan 11 10:53:31

1999 (CST) -06:00’;Owner = "raman";Cmd = "run_sim";Iwd = "/usr/raman/sim2";Args = "-Q 17 3200 10";ImageSize = 31M;Rank = DayTime()>’20:00’ &&

DayTime < ’8:00’ &&other.IsInstructional

? 10: other.KeybrdIdle>’3:00’

? 5: 0;

Constraint= other.Type=="Machine" &&Arch=="INTEL" &&OpSys=="SOLARIS251" &&Disk >= 10M &&other.Memory >=

self.ImageSize]

Figure 7. Time-dependent resource preference.

[Type = "Job";CompletionDate = undefined;RemoteSyscalls = true;Checkpoint = true;QDate = ’Mon Jan 11 10:53:31

1999 (CST) -06:00’;Owner = "raman";Cmd = "run_sim";Iwd = "/usr/raman/sim2";Args = "-Q 17 3200 10";ImageSize = 31M;ElapsedTime = CurrentTime( ) - QDate;Rank = other.Memory;Constraint = other.Type=="Machine" &&

Arch=="INTEL" &&OpSys=="SOLARIS251" &&other.Memory >=( ElapsedTime > ’2:00’ ?

0.5G:1.0G

)]

Figure 8. Time-dependent resource constraints.

has not been found in two hours, the customer downgradesthe resource requirement to a resource which has at leasthalf a gigabyte of memory. The Rank expression statesthat machines with larger memories are preferred. In thisexample, the customer will reject machines with less thanone gigabyte of memory for the first two hours, hoping fora better match. This policy is therefore fundamentally dif-ferent from one which merely prefers machines with largerphysical memories.

8. Related work

Although details of current distributed resource manage-ment systems vary, there are aspects that they share. In-stead of providing a survey of a large number of systems,we briefly discuss the basic matching mechanisms of someresource management environments to highlight the differ-ences between conventional resource allocation and match-making.

Systems such as NQE [1], PBS [8], LSF [20] and Load-Leveler [9] process user submitted jobs by finding resourcesthat have been identified either explicitly through a job con-trol language, or implicitly by submitting the job to a par-ticular queue that is associated with a set of resources. Cus-tomers of the system have to identify a specific queue tosubmit to a priori, which then fixes the set of resourcesthat may be used, and hinders dynamic qualitative resourcediscovery. Furthermore, system administrators have to an-ticipate the services that will be requested by customersand set up queues to provide these services. Over time, thesystem may accumulate a large number of queues whose

Page 9: Matchmaking: An extensible framework for distributed resource management

R. Raman et al. / Matchmaking 137

service semantics differ to various extents, complicating theprocess of finding the appropriate queue for a job.

Globus [2,4] defines an architecture for resource man-agement of autonomous distributed systems with provisionsfor policy extensibility and co-allocation. Customers de-scribe required resources through a resource specificationlanguage (RSL) that is based on a pre-defined schema ofthe resources database. The task of mapping specificationsto actual resources is performed by a resource co-allocator,which is responsible for coordinating the allocation andmanagement of resources at multiple sites. The RSL allowscustomers to provide very sophisticated resource require-ments, but no analogous mechanism exists for resources toselect customers.

Legion [6] takes an object-oriented approach to resourcemanagement, formulating the matching problem as an ob-ject placement problem [10]. The identification of a can-didate resource is performed by an object mapper, whoserecommendation is then implemented by a different object.The Legion system defines a notation [10] that is similarto classads, although it uses an object-oriented type sys-tem with inheritance to define resources [12], as contrastedwith the simple attribute-oriented Boolean logic of clas-sads. Legion supports autonomy with a jurisdiction magis-trate (JM), which may reject requests if the offered requestsdo not match the policy of the site being managed by theJM. While the JM gives a resource veto power, there is noway for a resource to describe requests that have not beenoffered that it would rather serve.

The JINI system [19] being developed by Sun Microsys-tems has similar notions as the classad based Condor sys-tem: resources advertise their presence, customers discovertheir presence through a lookup service and claim them forcomputation. Matchmaking differs from JINI in that inclassad matchmaking, resources may also place constraintson customers. Furthermore, the matching policy is explic-itly defined in the classad and can be changed.

9. Conclusions and future research

The classad matchmaking framework is a flexible andgeneral method of resource management in pools of re-sources which exhibit physical and ownership distribution.Novel aspects of the framework include a semi-structureddata model to represent entities, folding the query languageinto the data model, and a clean separation of the match-ing and claiming phases of resource allocation. The rep-resentation and protocols facilitate both static and dynamicheterogeneity of resources, which results in a robust, scal-able and flexible framework that can evolve with changingresources.

The framework has been developed in response to realproblems that have been encountered in the design, de-velopment and deployment of Condor, a high throughputcomputing system developed at the University of Wiscon-sin – Madison, which finds constant use by scientists at

the university and around the world. The success of theframework in a real system demonstrates the validity ofour approach.

Although the classad mechanism has no intrinsic notionof a schema, lists of classads representing resources andcustomers exhibit a high degree of regularity, which is man-ifest in two ways: structural regularity and value regularity.Structural regularity occurs when entities publish attributeswith the same names. For example, all workstations ad-vertise classads with “OpSys”, “Arch”, and “Memory” at-tributes. Value regularity occurs when entities advertiseclassads with similar values. For example, all workstationsfrom a laboratory may be similar, and therefore advertisethe same architecture, operating system and memory val-ues. We are currently investigating techniques for exploit-ing this regularity, and automatically aggregating classadsso that matches may be performed in groups. Group match-ing may be used to both boost matchmaking throughput andservice co-allocation requests.

The complexity of constraints imposed by resources andcustomers may hinder the diagnostic capability of adminis-trators and customers who may wonder why certain requestsare unable to find resources with particular characteristics.To alleviate this problem, we are researching methods foridentifying constraints which can never be satisfied by thepool. In addition to diagnostic utilities, this tool may helpdiscovering hidden characteristics of a pool.

References

[1] Cray Research, Document number in-2153 2/97, Technical Report,Cray Research (1997).

[2] K. Czajkowski, I. Foster, C. Kesselman, S. Martin, W. Smith andS. Tuecke, A resource management architecture for metacomput-ing systems, available from ftp://ftp.globus.org/pub/globus/papers/gram.ps.Z.

[3] D.H.J. Epema, M. Livny, R. van Dantzig, X. Evers and J. Pruyne,A worldwide flock of Condors: load sharing among workstationclusters, Journal on Future Generations of Computer Systems 12(1996).

[4] I. Foster and C. Kesselman, Globus: A metacomputing infrastructuretoolkit, to appear in International Journal of Supercomputer Appli-cations.

[5] J. Gosling, B. Joy and G. Steele, The Java language specification(1996). Available from http://www.java.sun.com/docs/books/jls/index.html.

[6] A.S. Grimsaw and W.A. Wulf, Legion – A view from 50,000 feet,in: Proceedings of the 5th IEEE International Symposium on HighPerformance Distributed Computing (August 1996).

[7] D. Gusfield and R.W. Irving, The Stable Marriage Problem: Struc-ture and Algorithms, (MIT Press, Cambridge, MA, 1989).

[8] R. Henderson and D. Tweten, Portable batch system: External refer-ence specification, Technical Report, NASA, Ames Research Center(1996).

[9] International Business Machines Corporation, IBM Load Leveler:User’s Guide (September 1993).

[10] J.F. Karpovich, Support for object placement in hetergenous dis-tributed systems, Technical Report CS-96-03, University of Virginia(January 1996).

[11] B.W. Kerninghan and D.M. Ritchie, The C Programming Language(Prentice-Hall, Englewood Cliffs, NJ, 2nd ed., 1988).

Page 10: Matchmaking: An extensible framework for distributed resource management

138 R. Raman et al. / Matchmaking

[12] M.J. Lewis and A. Grimshaw, The core Legion object model, in:Proc. of the 5th IEEE International Symposium on High PerformanceDistributed Computing (August 1996).

[13] M.J. Litzkow, M. Livny and M.W. Mutka, Condor – A hunter of idleworkstations, in: Proc. of the 8th International Conf. on DistributedComputing Systems (1988) pp. 104–111.

[14] M. Muttka and M. Livny, The available capacity of a privately ownedworkstation, Performance Evaluation (1991).

[15] S. Nestorov, S. Abiteboul and R. Motwani, Inferring structure insemistructured data, in: Proceedings of the Workshop on Manage-ment of Semistructured Data, Tucson, AZ (May 1997).

[16] J. Pruyne and M. Livny, Interfacing Condor and PVM to harnessthe cycles of workstation clusters, Journal on Future Generations ofComputer Systems 12 (1996).

[17] J.H. Saltzer, D.P. Reed and D.D. Clark, End-to-end arguments in sys-tem design, ACM Transactions on Computer Systems 2(4) (August1984) 277–288.

[18] B. Stroustrup, The C++ Programming Language (Addison-Wesley,Reading, MA, 3rd ed., 1997).

[19] J. Waldo, Jini architecture review: White paper, Su Microsystems Inc.(1998). Available from http://java.sun.com/products/jini/whitepapers/architectureoverview.pdf.

[20] S. Zhou, LSF: Load sharing in large-scale heterogenous distributedsystems, in: Proc. Workshop on Cluster Computing (1992).

Rajesh Raman received a B.A. in computer sci-ence and mathematics from Ohio Wesleyan Uni-versity, and an M.S. degree in computer sciencefrom the University of Wisconsin – Madison. Heis currently working towards a Ph.D. degree indistributed resource management frameworks. Ra-jesh’s interests include distributed systems, operat-ing systems, programming languages, music, sail-ing, food, drink and sleep.E-mail: [email protected]

Miron Livny received a B.S. degree in physicsand mathematics in 1975 from the Hebrew Uni-versity and M.Sc. and Ph.D. degrees in computerscience from the Weizmann Institute of Sciencein 1978 and 1984, respectively. He has been afaculty member at the Computer Sciences Depart-ment of the University of Wisconsin – Madisonsince 1983, and is currently a Professor. His in-terests include high throughput computing, visualexploration of information, scientific databases and

scheduling policies.E-mail: [email protected]

Marvin Solomon received a B.S. degree in math-ematics from the University of Chicago and M.S.and Ph.D. degrees in computer science from Cor-nell University. In 1976, he joined the Departmentof Computer Sciences at the University of Wis-consin – Madison, where he is currently a Pro-fessor. Dr. Solomon was a Visiting Lecturer atAarhus (Denmark) University during 1975–1976and a Visiting Scientist at IBM Research in SanJose, California, during 1984–1985. His research

interests include programming languages, program development environ-ments, operating systems, object-oriented databases and computer net-works.E-mail: [email protected]


Recommended