Mechanisms for Matchmaking and Parallel High Throughput Computing in the Condor Distributed System
Rajesh Raman, [email protected] Tannenbaum, [email protected]
http://www.cs.wisc.edu/condorOct 27, 1997
Condor ProjectOverviewWhat is Condor ?Projects and CollaborationsHigh Throughput ComputingClassAds and MatchMakingParallel Computing with Condor
What is Condor ?High Throughput ComputingDistributed ResourcesPhysically distributed Distributed ownership Resource ManagementIncrease utilization of resources Simple interface to execution environmentUser level interface Application level interface
Important MechanismsMatchmakingCheckpointing (and migration)Owner policies require resource reclamation Need to save (resumable) state of application Remote System CallsPreserves submission environment in execution environment. SandboxingSecurity concerns
The Condor TeamProf. Miron Livny, PIResearch StaffTodd TannenbaumDerek WrightAdding 2 more...
Condor Team, cont.Graduate Students Rajesh Raman (MatchMaking)Jim Basney (Split Execution)Shrinivas Ashwin (Mr. Parallel)Adiel Yoaz (Accounting)Undergraduate StudentsTom Stanis
Condor AlmuniMike LitzkowDavid DewittMarvin SolomonMany others (Produced XXX Masters and XXX PhDs]
Current Collaborators and ProjectsNCSAPACINational GridUW-FlockIntel Sponsorship: $4.2 Million Graduate School, EngineeringmetaNEOS: metacomputing environments for optimizationwith Prof. Michael Ferris
Condor Pool InstallationsUniversitesU of Wisconsin, U of Illinois, U of Michigan, Dartmouth, Duke, U of Washington, U of Virginia, U of California-BerkeleyGovernmentNCSA, Nasa, US Navy, NSA, NIKHEF (Amsterdam), INFN (Italy)CommercialHewlett-Packard Labs, J.P. Morgan, Mercedez-Benz, Dragon Systems
Power of Computing EnvironmentsPower = Work / TimeHigh Performance ComputingFixed amount of work; how much time? Response time/latency orientedTraditional Performance metrics: FLOPS, MIPS High Throughput ComputingFixed amount of time; how much work? Throughput oriented Application specific performance metrics
Distributed Ownership of ResourcesCommodity resourcesUnderutilized: 70% of a pool's cycles are not utilizedFragmented: owned by different people Can provide HTC with these cycles, BUTMust not impact QOS to owner Owners specify access policyExpressed with control expressionsThe current state of the resource (e.g., load average) Characteristics of the request (e.g., who wants to use it?) Time of day, random numbers, etc
Condor ArchitectureStartds ( Represent owners of resources) Implement owner's access control policySchedds( Represent customers of the system) Maintain persistent queues of resource requestsManager Collector: Database of resources Negotiator: Matchmaker Accountant: Priority maintenance
Condor Architecture, cont.
MatchmakingCustomers Require resources with certain characteristicsDiscriminating customersRequests place constraints on resources Distributed ownershipResources service requests which match owner's policy Discriminating resources Resource offers place constraints on customers Matchmaking is symmetric
Matchmaking with Classified AdvertisementsParties requiring matchmaking advertiseCharacteristics and requirements (i.e., constraints)Advertisements matched by a MatchmakerMatched parties contact each other to "claimCommunication, authentication, constraint verification, negotiation of terms, etc. Claiming does not involve the MatchmakerMethod is symmetricNo client/server relation imposed
Classified Advertisement Matchmaking FrameworkExpression and evaluation of characteristicsClassAd, Closure, EvaluationContext Advertising ProtocolContents of advertisements Publication protocol Matchmaking AlgorithmRelates ad contents to matching process Priority schemes, Ranking schemes, etc.
Classified Advertisement Matchmaking Framework (contd.) Matchmaking ProtocolHow are relevant parties informed of a successful match?What information are they given? Claiming ProtocolHow do matched parties claim each other to cooperate?
ClassAd: Mechanism for expressing characteristicsA ClassAd is a set of names, each of which is bound to an expression. e.g., [ Name => "Joe Hacker" ; Height => 182 ; Sex => "Male" ; Disposition => (TimeOfDay() < 600) ? "Sour" : UNDEFINED ; Requirements => (other.Height < Height) && (other.Sex == "Female") ] ExpressionsConstants, attribute references, function calls
ClassAd (contd.) Attribute references may refer to attributes in other adsAttribute references "trigger" expression evaluationScope resolutionEvaluates to UNDEFINED if no such expression exists Values String, integer, real, UNDEFINED and ERROR typesOperators are total (i.e., defined over all values)
Closure: Evaluation Environment for a ClassAdDetermines which ClassAd's attributes to lookup Closure is ClassAd and an ordered mapping of (scope-name, closure) pairsNo name may be repeated
EvaluationContext: Evaluation Environment for several ClassAds A set of closures which is self-containedNo closure reference leaves the context Condor's "Standard Context" is a bit more complexIncludes closures for a matchmaker "advertisement
Matchmaking in CondorOpportunistic Resource ExploitationResource availability is unpredictableExploit resources as soon as they are available Return resources as soon as they are unavailable Matchmaking performed continuously Attractive for malleable parallel applicationsRequest more resources after execution commencesGranted immediately if resources are available, orAs soon as resources become available
Matchmaking in Condor (contd.)Advertising protocol Startd's, Schedd's send classads to CollectorMust contain a "Requirements expressionOptionally contain a"Rank and CurrentRank expressionsStartds send a "private ad" containing a capabilityMatchmaking protocolGive the matched Startd and Schedd the capability from the startd's private ad
Matchmaking in Condor (contd.)Matchmaking AlgorithmRequest ad A matched with offer ad B iffA's "Requirements" expression evaluates to TRUE, andB's "Requirements" expression evaluates to TRUE, andBs"Rank" expression value is greater than "CurrentRank", andAs "Rank" expression value is its greatest when evaluated against BClaiming protocolNegotiate "heartbeat" frequency, checkpoint transfer, etc.
Condor ParallelismJob LevelCondor clusters of processesDagManTask LevelInterfacing Condor and PVMPVM: Message PassingCondor: Resource ManagementPVM Resource Manager Interfacepvm_reg_rm()
Interfacing Condor and PVM, cont.
Interfacing Condor and PVM, cont.CARMI -vs- PVM Resource RequestsPVM: SynchronousCARMI: AsynchronousResource Request MechanismPVM: Hostname and Type StringCARMI: ClassAdCARMI Resource ClassTask ManagementCARMI: Additional NotificationsCARMI: Additional Operations
Master-Worker ModelA good fit for an opportunistic environmentMasterRuns on Submit MachineManages pool of tasksWorkerRuns on remote machinesReceives pieces of work from the Master, returns answerShadowMasterPVMdStarterPVMdWorkerStarterPVMdWorker
Additional Condor/PVM FrameworksCoCheckCheckpoint a Worker or set of WorkersRequirements for a consistent checkpointSynchronize all processesFlush PVM messages in transitPerform Checkpoint (save image)Remap TIDsWoDiA framework for Master-Worker applicationsPerforms optimizations
Future Work Part IIMatchmakingAggregate Resources/RequestsAccountingAuthenticationFlockingJava UniverseSplit Execution
SummaryCondor is an implementation of a High Throughput Computing system in an opportunistic environment.Major Mechanisms to achieve HTC:MatchmakingCheckpointingRemote system callsSandboxingQuestions ?