Distributed Constraint Handlingand Optimization
Alessandro Farinelli1 Alex Rogers2 Meritxell Vinyals1
1Computer Science DepartmentUniversity of Verona, Italy
2Agents, Interaction and Complexity GroupSchool of Electronics and Computer Science
University of Southampton,UK
Tutorial EASSS 2012 Valenciahttps://sites.google.com/site/
easss2012optimization/
Outline
Introduction
Distributed Constraint Reasoning
Applications and Exemplar Problems
Complete algorithms for DCOPs
Approximated Algorithms for DCOPs
Conclusions
Outline
Introduction
Distributed Constraint Reasoning
Applications and Exemplar Problems
Complete algorithms for DCOPs
Approximated Algorithms for DCOPs
Conclusions
Constraints
• Pervade our everyday lives
• Are usually perceived as elements that limit solutions to theproblems we face
Constraints
From a computational point of view, they:
• Reduce the space of possible solutions
• Encode knowledge about the problem at hand
• Are key components for efficiently solving hard problems
Constraint Processing
Many different disciplines deal with hard computational problems thatcan be made tractable by carefully considering the constraints that
define the structure of the problem.
Planning Operational Automated Reasoning ComputerScheduling Research Decision Theory Vision
Constraint Processing in Multi-Agent Systems
Focus on how constraint processing can be used to addressoptimization problems in Multi-Agent Systems (MAS) where:
A set of agents must come to some agreement, typically via someform of negotiation, about which action each agent should take in
order to jointly obtain the best solution for the whole system.
A1 A2 A3
M1 M2
A2
Distributed Constraint Optimization Problems (DCOPs)
We will consider Distributed Constraint Optimization Problems (DCOP)where:
Each agent negotiates locally with just a subset of other agents(usually called neighbors) that are those that can directly influence
his/her behavior.
M1M2
A2
A1 A3
Distributed Constraint Optimization Problems (DCOPs)
After reading this chapter, you will understand:
• The mathematical formulation of a DCOP• The main exact solution techniques for DCOPs
• Key differences, benefits and limitations
• The main approximate solution techniques for DCOPs• Key differences, benefits and limitations
• The quality guarantees these approach provide:• Types of quality guarantees• Frameworks and techniques
Outline
Introduction
Distributed Constraint Reasoning
Applications and Exemplar Problems
Complete algorithms for DCOPs
Approximated Algorithms for DCOPs
Conclusions
Constraint Networks
A constraint network N is formally defined as a tuple 〈X ,D,C〉 where:
• X = {x1, . . . ,xn} is a set of discrete variables;
• D = {D1, . . . ,Dn} is a set of variable domains, which enumerateall possible values of the corresponding variables; and
• C = {C1, . . . ,Cm} is a set of constraints; where a constraint Ci isdefined on a subset of variables Si ⊆ X which comprise thescope of the constraint
• r = |Si| is the arity of the constraint• Two types: hard or soft
Hard constraints
• A hard constraint Chi is a relation Ri that enumerates all the valid
joint assignments of all variables in the scope of the constraint.
Ri ⊆ Di1× . . .×Dir
Ri x j xk
0 11 0
Soft constraints
• A soft constraint Csi is a function Fi that maps every possible joint
assignment of all variables in the scope to a real value.
Fi : Di1× . . .×Dir →ℜ
Fi x j xk
2 0 00 0 10 1 01 1 1
Binary Constraint Networks
• Binary constraint networks are those where:
• Each constraint (soft or hard) is definedover two variables.
• Every constraint network can be mapped toa binary constraint network
• requires the addition of variables andconstraints
• may add complexity to the model
• They can be represented by a constraintgraph
x1
x2 x3
x4
F 1,2 F
1,3
F 2,4
F 1,4
Different objectives, different problems
• Constraint Satisfaction Problem (CSP)
• Objective: find an assignment for all the variables in the networkthat satisfies all constraints.
• Constraint Optimization Problem (COP)
• Objective: find an assignment for all the variables in the networkthat satisfies all constraints and optimizes a global function.
• Global function = aggregation (typically sum) of local functions.F(x) = ∑i Fi(xi)
Distributed Constraint Reasoning
When operating in adecentralized context:
• a set of agents controlvariables
• agents interact to find asolution to the constraintnetwork
A1
A2
A3
A4
Distributed Constraint Reasoning
Two types of decentralized problems:
• distributed CSP (DCSP)
• distributed COP (DCOP)
Here, we focus on DCOPs.
Distributed Constraint Optimization Problem (DCOP)
A DCOP consists of a constraint network N = 〈X ,D,C〉 and a set ofagents A = {A1, . . . ,Ak} where each agent:
• controls a subset of the variables Xi ⊆ X• is only aware of constraints that involve variable it controls
• communicates only with its neighbours
Distributed Constraint Optimization Problem (DCOP)
• Agents are assumed to be fully cooperative• Goal: find the assignment that optimizes the global function, not
their local local utilities.
• Solving a COP is NP-Hard and DCOP is as hard as COP.
Motivation
Why distribute?
• Privacy
• Robustness
• Scalability
Outline
Introduction
Distributed Constraint Reasoning
Applications and Exemplar Problems
Complete algorithms for DCOPs
Approximated Algorithms for DCOPs
Conclusions
Real World Applications
Many standard benchmark problems in computer science can bemodeled using the DCOP framework:
• graph coloring
As can many real world applications:
• human-agent organizations (e.g. meeting scheduling)
• sensor networks and robotics (e.g. target tracking)
Outline
Introduction
Distributed Constraint Reasoning
Applications and Exemplar ProblemsGraph coloringMeeting SchedulingTarget Tracking
Complete algorithms for DCOPs
Approximated Algorithms for DCOPs
Conclusions
Graph coloring
• Popular benchmark
• Simple formulation• Complexity controlled with few parameters:
• Number of available colors• Number of nodes• Density (#nodes/#constraints)
• Many versions of the problem:• CSP, MaxCSP, COP
Graph coloring - CSP
• Nodes can take k colors• Any two adjacent nodes should have different colors
• If it happens this is a conflict
Yes! No!
Graph coloring - Max-CSP
• Minimize the number of conflicts
0 -1
Graph coloring - COP
• Different weights to violated constraints
• Preferences for different colors
0 -2-1
-3
-1
-2
Graph coloring - DCOP
• Each node:• controlled by one agent
• Each agent:• Preferences for different colors• Communicates with its direct neighbours in the graph
-1
-3
-1
-2A1
A2
A3
A4
• A1 and A2 exchangepreferences and conflicts
• A3 and A4 do notcommunicate
Outline
Introduction
Distributed Constraint Reasoning
Applications and Exemplar ProblemsGraph coloringMeeting SchedulingTarget Tracking
Complete algorithms for DCOPs
Approximated Algorithms for DCOPs
Conclusions
Meeting Scheduling
Motivation:
• Privacy
• Robustness
• Scalability
Meeting Scheduling
In large organizations many people, possibly working in differentdepartments, are involved in a number of work meetings.
Meeting Scheduling
People might have various private preferences on meeting start times
Better after 12:00am
Meeting Scheduling
Two meetings that share a participant cannot overlap
Window: 15:00-18:00Duration: 2h
Window: 15:00-17:00Duration: 1h
DCOP formalization for the meeting scheduling problem
• A set of agents representing participants
• A set of variables representing meeting starting times accordingto a participant.
• Hard Constraints:• Starting meeting times across different agents are equal• Meetings for the same agent are non-overlapping.
• Soft Constraints:• Represent agent preferences on meeting starting times.
Objective: find a valid schedule for the meeting while maximizing thesum of individuals’ preferences.
Outline
Introduction
Distributed Constraint Reasoning
Applications and Exemplar ProblemsGraph coloringMeeting SchedulingTarget Tracking
Complete algorithms for DCOPs
Approximated Algorithms for DCOPs
Conclusions
Target Tracking
Motivation:
• Privacy
• Robustness
• Scalability
Target Tracking
A set of sensors tracking a set of targets in order to provide anaccurate estimate of their positions.
T1
T2
T4
T3
Crucial for surveillance and monitoring applications.
Target Tracking
Sensors can have different sensing modalities that impact on theaccuracy of the estimation of the targets’ positions.
T1
T4
T3
T2
MODES
MODES
MODES
MODES
Target Tracking
Collaboration among sensors is crucial to improve systemperformance
T1
T2
T4
T3
DCOP formalization for the target tracking problem
• Agents represent sensors
• Variables encode the different sensing modalities of each sensor• Constraints
• relate to a specific target• represent how sensor modalities impacts on the tracking
performance
• Objective:• Maximize coverage of the environment• Provide accurate estimations of potentially dangerous targets
Outline
Introduction
Distributed Constraint Reasoning
Applications and Exemplar Problems
Complete algorithms for DCOPs
Approximated Algorithms for DCOPs
Conclusions
Complete Algorithms
U Always find an optimal solution
D Exhibit an exponentially increasing coordination overhead
D Very limited scalability on general problems.
Complete Algorithms
• Completely decentralised• Search-based.
• Synchronous: SyncBB, AND/OR search• Asynchronous: ADOPT, NCBB and AFB.
• Dynamic programming.
• Partially decentralised• OptAPO
Next, we focus on completely decentralised algorithms
Decentralised Complete Algorithms
Search-based
• Uses distributed search
• Exchange individual values
• Small messages but. . . exponentially many
Representative: ADOPT [Modiet al., 2005]
Dynamic programming
• Uses distributed inference
• Exchange constraints
• Few messages but. . . exponentially large
Representative: DPOP [Petcuand Faltings, 2005]
Outline
Introduction
Distributed Constraint Reasoning
Applications and Exemplar Problems
Complete algorithms for DCOPsSearch Based: ADOPTDynamic Programming DPOP
Approximated Algorithms for DCOPs
Conclusions
ADOPT
ADOPT (Asynchronous Distributed OPTimization) [Modi et al., 2005]:
• Distributed backtrack search using a best-first strategy
• Best value based on local information:
• Lower/upper bound estimates of each possible value of itsvariable
• Backtrack thresholds used to speed up the search of previouslyexplored solutions.
• Termination conditions that check if the bound interval is less thana given valid error bound (0 if optimal)
ADOPT by example
4 variables (4 agents): x1,x2,x3,x4 with D = {0,1}
x1
x2 x3
x4
F 1,2 F
1,3
F 2,4 F 1,
44 identical cost functions
Fi, j xi x j
2 0 00 0 10 1 01 1 1
Goal: find a variable assignment with minimal cost
Solution: x1 = 1, x2 = 0, x3 = 0 and x4 = 1giving total cost 1.
DFS arrangement
• Before executing ADOPT, agents must be arranged in a depthfirst search (DFS) tree.
• DFS trees have been frequently used in optimization becausethey have two interesting properties:
• Agents in different branches of the tree do not share anyconstraints;
• Every constraint network admits a DFS tree.
ADOPT by example
x1
x2 x3
x4
F 1,2 F
1,3
F 2,4 F 1,
4Fi, j xi x j
2 0 00 0 10 1 01 1 1
→
DFS arrangement
A1 (root)
A2 A3
A4
←ch
ild, p
aren
t→
←parent,child
→
←ch
ild, p
aren
t→
Cost functions
The local cost function for an agent Ai (δ (xi)) is the sum of the valuesof constraints involving only higher neighbors in the DFS.
ADOPT by example
A1 δ (x1) = 0
A2δ (x1,x2) = F1,2(x1,x2) A3 δ (x1,x3) = F1,3(x1,x3)
A4
δ (x1,x2,x4) = F1,4(x1,x4)+F2,4(x2,x4)
Initialization
Each agent initially chooses a random value for their variables andinitialize the lower and upper bounds to zero and infinity respectively.
A1x1 = 0,LB = 0,UB = ∞
A2x2 = 0,LB = 0,UB = ∞ A3 x3 = 0,LB = 0,UB = ∞
A4x4 = 0,LB = 0,UB = ∞
ADOPT by example
Value messages are sent by an agent to all its neighbors that arelower in the DFS tree
A1x1 = 0
A2x2 = 0 A3 x3 = 0
A4x4 = 0
x 1=
0←−−−
x1=
0
−−−→
x 2=
0←−−− x 1=
0←−−−
A1 sends three valuemessage to A2, A3 andA4 informing them that itscurrent value is 0.
ADOPT by example
Current Context: a partial variable assignment maintained by eachagent that records the assignment of all higher neighbours in the DFS.
A1
A2c2 : {x1 = 0} A3
c3 : {x1 = 0}
A4
c4 : {x1 = 0,x2 = 0}
• Updated by each VALUEmessage
• If current context is notcompatible with some childcontext, the latter is re-initialized(also the child bound)
ADOPT by example
Each agent Ai sends a cost message to its parent Ap
A1
A2 A3
A4
[0,∞
,c2]
[0,0,c3 ]
[0,0,c
4]
Each cost message reports:
• The minimum lower bound (LB)
• The maximum upper bound (UB)
• The context (ci)
[LB,UP,ci]
Lower bound computation
Each agent evaluates for each possible value of its variable:
• its local cost function with respect to the current context
• adding all the compatible lower bound messages received fromchildren.
Analogous computation for upper bounds
ADOPT by example
Consider the lower bound in the cost message sent by A4:
A1
A2 A3
A4
[0,0,c
4]
• Recall that A4 local cost function is:δ (x1,x2,x4) = F1,4(x1,x4)+F2,4(x2,x4)
• Restricted to the current contextc4 = {(x1 = 0,x2 = 0)}:λ (0,0,x4) = F1,4(0,x4)+F2,4(0,x4).
• For x4 = 0:λ (0,0,0) = F1,4(0,0)+F2,4(0,0) = 2+2 = 4.
• For x4 = 1:λ (0,0,1) = F1,4(0,1)+F2,4(0,1) = 0+0 = 0.
Then the minimum lower bound across variablevalues is LB = 0.
ADOPT by example
Each agent asynchronously chooses the value of its variable thatminimizes its lower bound.
A1
A2x2 = 0→ 1 A3
A4
[0,2,c
2]
x 2=
1
A2 computes for each possible value of itsvariable its local function restricted to thecurrent context c2 = {(x1 = 0)}(λ (0,x2) = F1,2(0,x2)) and adding lowerbound message from A4 (lb).
• For x2 = 0: LB(x2 = 0) = λ (0,x2 =0)+ lb(x2 = 0) = 2+0 = 2.
• For x2 = 1: LB(x2 = 1) = λ (0,x2 =1)+0 = 0+0 = 0.
A2 changes its value to x2 = 1 with LB = 0.
Backtrack thresholds
The search strategy is based on lower bounds
Problem
• Values abandoned before proven to besuboptimal
• Lower/upper bounds only stored for thecurrent context
Solution
• Backtrack thresholds: used to speed upthe search of previously exploredsolutions.
ADOPT by example
A1
x1 = 0→ 1→ 0
A2 A3
A4
A1 changes its value and the context withx1 = 0 is visited again.
• Reconstructing from scratch is inefficient
• Remembering solutions is expensive
Backtrack thresholds
Solution: Backtrack thresholds
• Lower bound previously determined by children
• Polynomial space
• Control backtracking to efficiently search
• Key point: do not change value until LB(currentvalue)> threshold
A child agent will not change its variable value so long as cost is lessthan the backtrack threshold given to it by its parent.
A1LB(x1 = 0) = 1
A2LB(x2 = 0)> 12 ? A3 LB(x3 = 0)> 1
2 ?
A4
t(x 1
=0)
=1 2
t(x1=
0)=
12
Rebalance incorrect threshold
How to correctly subdivide threshold among children?
• Parent distributes the accumulated bound among children• Arbitrarily/Using some heuristics
• Correct subdivision as feedback is received from children• LB < t(CONT EXT )• t(CONT EXT ) = ∑Ci t(CONT EXT )+δ
Backtrack Threshold Computation
A1
A2 A3
A4
(2)
t(x 1
=0)
=1
(1) L
B=
1
(2)12 t(x
1=
0)=
0
• When A1 receives a new lower boundfrom A2 rebalances thresholds
• A1 resends threshold messages to A2and A3
ADOPT extensions
• BnB-ADOPT [Yeoh et al., 2008] reduces computation time byusing depth-first search with branch and bound strategy
• [Ali et al., 2005] suggest the use of preprocessing techniques forguiding ADOPT search and show that this can result in aconsistent increase in performance.
Outline
Introduction
Distributed Constraint Reasoning
Applications and Exemplar Problems
Complete algorithms for DCOPsSearch Based: ADOPTDynamic Programming DPOP
Approximated Algorithms for DCOPs
Conclusions
DPOP
DPOP (Dynamic Programming Optimization Protocol) [Petcu andFaltings, 2005]:
• Based on the dynamic programming paradigm.
• Special case of Bucket Tree Elimination Algorithm (BTE)[Dechter, 2003].
DPOP by example
x1
x2 x3
x4
Objective: find assignmentwith maximal value
F 1,2 F
1,3
F 2,4
F2,3F 1,4
Fi, j xi x j2 0 00 0 10 1 01 1 1
=>
DFS arrangement
x1
x2
x3x4
←P
2 ,child→
←ch
ild,P
4→
←P
3 ,child→
←ps
eudo
child
,PP 4→
←P
P3 , pseudochild
→
DPOP phases
Given a DFS tree structure, DPOP runs in two phases:• Util propagation: agents exchange util messages up the tree.
• Aim: aggregate all info so that root agent can choose optimalvalue
• Value propagation: agents exchange value messages down thetree.
• Aim: propagate info so that all agents can make their choice givenchoices of ancestors
Sepi: set of agents preceding Ai in the pseudo-tree order that areconnected with Ai or with a descendant of Ai.
x1 Sep1 = /0
x2 Sep2 = {x1}
x3 Sep3 = {x1,x2}x4Sep4 = {x1,x2}
←P
2 ,c hild→
←ch
ild,P
4→
←P
3 ,child→
←ps
eudo
child
,PP 4→
←P
P3 , pseudochild
→
Util message
The Util message Ui→ j that agent Ai sends to its parent A j can becomputed as:
Ui→ j(Sepi) = maxxi
( ⊗Ak∈Ci
Uk→i⊗⊗
Ap∈Pi∪PPi
Fi,p
)
All incoming messages Shared constraints withSize exponentialfrom children parents/pseudoparentsin Sepi
The ⊗ operator is a join operator that sums up functions with differentbut overlapping scores consistently.
Join operator
F2,4 x2 x42 0 00 0 10 1 01 1 1
AddProject
out x4F1,4 x1 x42 0 00 0 10 1 01 1 1
F1,4⊗F2,4x1 x2 x4
4 0 0 00 0 0 12 0 1 01 0 1 12 1 0 02 1 0 10 1 1 02 1 1 1
max{x4}(F1,4⊗F2,4)
x1 x2 x4
max(4,0)0 0 00 0 1
max(2,1)0 1 00 1 1
max(2,2)1 0 01 0 1
max(0,2)1 1 01 1 1
Complexity exponential to the largest Sepi.Largest Sepi = induced width of the DFS tree ordering used.
A1 (root)
A2max
x2(U3→2⊗U4→2⊗F1,2)
A3
maxx3
(F1,3⊗F2,3)
A4
12 max
x4(F1,4⊗F2,4)
U1→
2←−−−
U 4→2
−−−→
U3→
2
←−−−
U2→1 x10 101 5
U4→2 x1 x24 0 02 0 12 1 02 1 1
U3→2 x1 x24 0 02 0 12 1 02 1 1
Sep2
Sep3
Sep4
Value message
Keeping fixed the value of parent/pseudoparents, finds the value thatmaximizes the computed cost function in the util phase:
x∗i = argmaxxi
(∑
A j∈Ci
U j→i(xi,x∗p)+ ∑A j∈Pi∪PPi
Fi, j(xi,x∗j)
)
where x∗p =⋃
A j∈Pi∪PPi{x∗j} is the set of optimal values for Ai’s parent
and pseudoparents received from Ai’s parent.Propagates this value through children down the tree:
Vi→ j = {xi = x∗i }∪⋃
xs∈Sepi∩Sep j
{xs = x∗s}
A1x∗1 = max
x1U1→2(x1)
A2x∗2 = max
x2(U3→2(x∗1,x2)⊗U4→2(x∗1,x2)⊗F1,2(x∗1,x2))
A3
x∗3 = maxx3
(F1,3(x∗1,x3)⊗F2,3(x∗2,x3))
A4
x∗4 = maxx4
(F1,4(x∗1,x4)⊗F2,4(x∗2,x4))V
1→2
− −→
V 2→4
←−−
V2→
3
−−→
DPOP extensions
• MB-DPOP [Petcu and Faltings, 2007] trades-off message sizeagainst the number of messages.
• A-DPOP trades-off message size against solution quality [Petcuand Faltings, 2005(2)].
Conclusions
• Constraint processing• exploit problem structure to solve hard problems efficiently
• DCOP framework• applies constraint processing to solve decision making problems
in Multi-Agent Systems• increasingly being applied within real world problems.
References I
• [Modi et al., 2005] P. J. Modi, W. Shen, M. Tambe, and M.Yokoo. ADOPT: Asynchronousdistributed constraint optimization with quality guarantees. Artificial Intelligence Jour- nal,(161):149-180, 2005.
• [Yeoh et al., 2008] W. Yeoh, A. Felner, and S. Koenig. BnB-ADOPT: An asynchronousbranch-and-bound DCOP algorithm. In Proceedings of the Seventh International JointConference on Autonomous Agents and Multiagent Systems, pages 591Ð598, 2008.
• [Ali et al., 2005] S. M. Ali, S. Koenig, and M. Tambe. Preprocessing techniques foraccelerating the DCOP algorithm ADOPT. In Proceedings of the Fourth International JointConference on Autonomous Agents and Multiagent Systems, pages 1041Ð1048, 2005.
• [Petcu and Faltings, 2005] A. Petcu and B. Faltings. DPOP: A scalable method formultiagent constraint opti- mization. In Proceedings of the Nineteenth International JointConference on Arti- ficial Intelligence, pages 266-271, 2005.
• [Dechter, 2003] R. Dechter. Constraint Processing. Morgan Kaufmann, 2003.
References II
• [Petcu and Faltings, 2005(2)] A. Petcu and B. Faltings. A-DPOP: Approximations indistributed optimization. In Principles and Practice of Constraint Programming, pages802-806, 2005.
• [Petcu and Faltings, 2007] A. Petcu and B. Faltings. MB-DPOP: A new memory-boundedalgorithm for distributed optimization. In Proceedings of the Twentieth International JointConfer- ence on Artificial Intelligence, pages 1452-1457, 2007.
• [S. Fitzpatrick and L. Meetrens, 2003] S. Fitzpatrick and L. Meetrens. Distributed SensorNetworks: A multiagent perspective, chapter Distributed coordination through anarchicoptimization, pages 257- 293. Kluwer Academic, 2003.
• [R. T. Maheswaran et al., 2004] R. T. Maheswaran, J. P. Pearce, and M. Tambe.Distributed algorithms for DCOP: A graphical game-based approach. In Proceedings ofthe Seventeenth International Conference on Parallel and Distributed ComputingSystems, pages 432-439, 2004.
Outline
Introduction
Distributed Constraint Reasoning
Applications and Exemplar Problems
Complete algorithms for DCOPs
Approximated Algorithms for DCOPs
Conclusions
Approximate Algorithms: outline
• No guarantees
– DSA-1, MGM-1 (exchange individual assignments)
– Max-Sum (exchange functions)– Max-Sum (exchange functions)
• Off-Line guarantees
– K-optimality and extensions
• On-Line Guarantees
– Bounded max-sum
Why Approximate Algorithms
• Motivations
– Often optimality in practical applications is not achievable
– Fast good enough solutions are all we can have
• Example – Graph coloring• Example – Graph coloring
– Medium size problem (about 20 nodes, three colors per
node)
– Number of states to visit for optimal solution in the worst
case 3^20 = 3 billions of states
• Key problem
– Provides guarantees on solution quality
Exemplar Application: Surveillance
• Event Detection
– Vehicles passing on a road
• Energy Constraints
– Sense/Sleep modes
– Recharge when sleeping– Recharge when sleeping
• Coordination
– Activity can be detected by single sensor
– Roads have different traffic loads
• Aim [Rogers et al. 10]
– Focus on road with more traffic load
time
duty cycle
Heavy traffic road small road
Good Schedule
Bad Schedule
Surveillance demo
Guarantees on solution quality
• Key Concept: bound the optimal solution
– Assume a maximization problem
– optimal solution, a solution
–
– percentage of optimality
• [0,1]
• The higher the better
– approximation ratio
• >= 1
• The lower the better
– is the bound
Types of Guarantees
Accuracy
Accuracy: high alpha
Generality: less use of
instance specific knowledgeBounded Max-
Sum
DaCSA
Instance-specific
Generality
MGM-1,
DSA-1,
Max-Sum
K-optimality
T-optimality
Region Opt.
Instance-generic
No guarantees
Centralized Local Greedy approaches• Greedy local search
– Start from random solution
– Do local changes if global solution improves
– Local: change the value of a subset of variables, usually one
-1-1 -1
-4-1
-1 -1-2
-4
0
0 -20
Centralized Local Greedy approaches
• Problems
– Local minima
– Standard solutions: RandomWalk, Simulated Annealing
-1 -1-1 -1
-2
-1 -1
-1 -1 -1 -1 -1 -1
Distributed Local Greedy approaches
• Local knowledge
• Parallel execution:
– A greedy local move might be harmful/useless
– Need coordination
-1-1-1 -1
-4
-1
-4
-1
-1
0 -2 0 -2 0 -20 -2
Distributed Stochastic Algorithm
• Greedy local search with activation probability to
mitigate issues with parallel executions
• DSA-1: change value of one variable at time
• Initialize agents with a random assignment and
communicate values to neighborscommunicate values to neighbors
• Each agent:
– Generates a random number and execute only if rnd less
than activation probability
– When executing changes value maximizing local gain
– Communicate possible variable change to neighbors
DSA-1: Execution Example
-1
P = 1/4
-1
-1
-1
rnd > ¼ ? rnd > ¼ ? rnd > ¼ ? rnd > ¼ ?
0 -2
DSA-1: discussion
• Extremely “cheap” (computation/communication)
• Good performance in various domains
– e.g. target tracking [Fitzpatrick Meertens 03, Zhang et al. 03],
– Shows an anytime property (not guaranteed)– Shows an anytime property (not guaranteed)
– Benchmarking technique for coordination
• Problems
– Activation probablity must be tuned [Zhang et al. 03]
– No general rule, hard to characterise results across domains
Maximum Gain Message (MGM-1)
• Coordinate to decide who is going to move
– Compute and exchange possible gains
– Agent with maximum (positive) gain executes
• Analysis [Maheswaran et al. 04]• Analysis [Maheswaran et al. 04]
– Empirically, similar to DSA
– More communication (but still linear)
– No Threshold to set
– Guaranteed to be monotonic (Anytime behavior)
MGM-1: Example
-1-1
0 -2-1 -1 0 -2
-1 -1
G = -2
G = 0 G = 2
G = 0
Local greedy approaches
• Exchange local values for variables
– Similar to search based methods (e.g. ADOPT)
• Consider only local information when maximizing
– Values of neighbors– Values of neighbors
• Anytime behaviors
• Could result in very bad solutions
Max-sum
Agents iteratively computes local functions that depend
only on the variable they control
X1 X2
Choose arg max
X4 X3 Shared constraint
All incoming
messages except x2
All incoming
messages
Choose arg max
Factor Graph and GDL• Factor Graph
– [Kschischang, Frey, Loeliger 01]
– Computational framework to represent factored computation
– Bipartite graph, Variable - Factor
)|()|()(),,( XXHXXHXHXXXH ++=
1x 2x
3x
1x 2x
3x
)|( 12 XXH
)( 1XH
)|( 13 XXH
)|()|()(),,( 13121321 XXHXXHXHXXXH ++=
Max-Sum on acyclic graphs• Max-sum Optimal on acyclic
graphs – Different branches are
independent
– Each agent can build a correct estimation of its contribution to the global problem (z functions)
1x 2x
)|( 12 XXH
)( 1XH
global problem (z functions)
• Message equations very similar to Util messages in DPOP– GDL generalizes DPOP [Vinyals
et al. 2010a]
3x
)|( 13 XXH
sum up info from other nodes
local maximization step
(Loopy) Max-sum Performance
• Good performance on loopy networks [Farinelli et al. 08]
– When it converges very good results
• Interesting results when only one cycle [Weiss 00]
– We could remove cycle but pay an exponential price (see DPOP)
– Java Library for max-sum http://code.google.com/p/jmaxsum/– Java Library for max-sum http://code.google.com/p/jmaxsum/
Max-Sum for low power devices
• Low overhead
– Msgs number/size
• Asynchronous computation
– Agents take decisions whenever new messages arrive
• Robust to message loss
Max-sum on hardware
Max-Sum for UAVs
Task Assignment for UAVs [Delle Fave et al 12]
Video Streaming
Interest points
Max-Sum
Task utility
Priority
Task completion
First assigned UAVs reaches task
24
Last assigned UAVs leaves task (consider battery life)
Urgency
Factor Graph Representation
2PDA
1PDA2UAV
U T
1T2X
1UAV
2U 2T
3PDA
1U
3U
3T
1X
UAVs Demo
Quality guarantees for approx.
techniques
• Key area of research
• Address trade-off between guarantees and
computational effort
• Particularly important for many real world applications• Particularly important for many real world applications
– Critical (e.g. Search and rescue)
– Constrained resource (e.g. Embedded devices)
– Dynamic settings
Instance-generic guarantees
Accuracy
Characterise solution quality without
running the algorithm
Bounded Max-
Sum
DaCSA
Instance-specific
Generality
Accuracy
MGM-1,
DSA-1,
Max-Sum
K-optimality
T-optimality
Region Opt.
Instance-generic
No guarantees
K-Optimality framework
• Given a characterization of solution gives bound on
solution quality [Pearce and Tambe 07]
• Characterization of solution: k-optimal
• K-optimal solution:• K-optimal solution:
– Corresponding value of the objective function can not be
improved by changing the assignment of k or less
variables.
K-Optimal solutions
1 1 1 1
1
1
1
1
2-optimal ? No3-optimal ? Yes
0 0 1 0
2
2
0
2
Bounds for K-Optimality
For any DCOP with non-negative rewards [Pearce and Tambe 07]
Number of agents Maximum arity of constraints
K-optimal solution
Binary Network (m=2):
K-Optimality Discussion
• Need algorithms for computing k-optimal solutions
– DSA-1, MGM-1 k=1; DSA-2, MGM-2 k=2 [Maheswaran et al. 04]
– DALO for generic k (and t-optimality) [Kiekintveld et al. 10]
• The higher k the more complex the computation
(exponential)(exponential)
Percentage of Optimal:
• The higher k the better
• The higher the number of
agents the worst
Trade-off between generality and solution
quality
• K-optimality based on worst case analysis
• assuming more knowledge gives much better bounds
• Knowledge on structure [Pearce and Tambe 07]
Trade-off between generality and
solution quality• Knowledge on reward [Bowring et al. 08]
• Beta: ratio of least minimum reward to the maximum
Off-Line Guarantees: Region
Optimality• k-optimality: use size as a criterion for optimality
• t-optimality: use distance to a central agent in the constraint graph
• Region Optimality: define regions based on general criteria (e.g. S-size bounded distance) [Vinyals et al 11]criteria (e.g. S-size bounded distance) [Vinyals et al 11]
• Ack: Meritxell Vinyals
3-size regions
x0 x1 x2 x3
x0
x1 x2
x3
x0
x1 x2
x3
x0
x1 x2
x3
x0
x1 x2
x3
1-distance regions
x0 x1 x2 x3
C regions
Size-Bounded Distance
• Region optimality can explore new regions: s-size bounded distance
• One region per agent, largest t-distance group whose size is less than s
3-size bounded distance
x0 x0than s
• S-Size-bounded distance – C-DALO extension of DALO for general
regions
– Can provides better bounds and keep under control size and number of regions
x1 x2
x3
x0
x1 x2
x3
x1 x2
x3
x0
x1 x2
x3
t=1
t=0 t=1
t=0
Max-Sum and Region Optimality
• Can use region optimality to provide bounds for Max-sum [Vinyals et al 10b]
• Upon convergence Max-Sum is optimal on SLT regions of the graph [Weiss 00]
• Single Loops and Trees (SLT): all groups of agents whose • Single Loops and Trees (SLT): all groups of agents whose vertex induced subgraph contains at most one cycle
x1
x0
x2
x3
x0
x1 x2
x3
x0
x1 x2
x3
x0
x1 x2
x3
x0
x1 x2
x3
Bounds for Max-Sum
• Complete: same as
3-size optimality
• bipartite
• 2D grids
Variable Disjoint Cycles
Very high quality guarantees if smallest cycle is large
Instance-specific guarantees
Characterise solution quality after/while
running the algorithm
Bounded Max-
Sum
DaCSA
Instance-specific
Generality
Accuracy
MGM-1,
DSA-1,
Max-Sum
DaCSA
K-optimality
T-optimality
Region Opt.
Instance-generic
No guarantees
Build Spanning tree
Bounded Max-SumAim: Remove cycles from Factor Graph avoiding
exponential computation/communication (e.g. no junction tree)
Key Idea: solve a relaxed problem instance [Rogers et al.11]
X1 F2 X3 X1 F2 X3
Run Max-SumCompute Bound
X1 X2
X2F1 F3 X2F1 F3
X3
Optimal solution on tree
Factor Graph Annotation
• Compute a weight for
each edge
– maximum possible impact of the variable on the
X1 F2 X3
w21 w23of the variable on the function
X2F1 F3
w21
w11
w12
w22
w23
w33
w32
Factor Graph Modification
X1 F2 X3
w21 w23
• Build a Maximum
Spanning Tree
– Keep higher weights
• Cut remaining
dependencies
– Compute
X2F1 F3
w11
w12
w22 w33
w32
W = w22 + w23
– Compute
• Modify functions
• Compute bound
Results: Random Binary Network
Optimal
Approx.
Lower Bound
Upper Bound
Bound is significant
– Approx. ratio is typically 1.23 (81 %)
Comparison with k-optimal
with knowledge on
reward structure
Much more accurate less general
Discussion
• Discussion with other data-dependent techniques
– BnB-ADOPT [Yeoh et al 09]
• Fix an error bound and execute until the error bound is met
• Worst case computation remains exponential
– ADPOP [Petcu and Faltings 05b]– ADPOP [Petcu and Faltings 05b]
• Can fix message size (and thus computation) or error bound and
leave the other parameter free
• Divide and coordinate [Vinyals et al 10]
– Divide problems among agents and negotiate agreement
by exchanging utility
– Provides anytime quality guarantees
Summary
• Approximation techniques crucial for practical applications:
surveillance, rescue, etc.
• DSA, MGM, Max-Sum heuristic approaches
– Low coordination overhead, acceptable performance
– No guarantees (convergence, solution quality)– No guarantees (convergence, solution quality)
• Instance generic guarantees:
– K-optimality framework
– Loose bounds for large scale systems
• Instance specific guarantees
– Bounded max-sum, ADPOP, BnB-ADOPT
– Performance depend on specific instance
References IDOCPs for MRS• [Delle Fave et al 12] A methodology for deploying the max-sum algorithm and a case study on
unmanned aerial vehicles. In, IAAI 2012
• [Taylor et al. 11] Distributed On-line Multi-Agent Optimization Under Uncertainty: Balancing Exploration and Exploitation, Advances in Complex Systems
MGM • [Maheswaran et al. 04] Distributed Algorithms for DCOP: A Graphical Game-Based Approach,
PDCS-2004
DSADSA• [Fitzpatrick and Meertens 03] Distributed Coordination through Anarchic Optimization,
Distributed Sensor Networks: a multiagent perspective.
• [Zhang et al. 03] A Comparative Study of Distributed Constraint algorithms, Distributed Sensor Networks: a multiagent perspective.
Max-Sum• [Stranders at al 09] Decentralised Coordination of Mobile Sensors Using the Max-Sum
Algorithm, AAAI 09
• [Rogers et al. 10] Self-organising Sensors for Wide Area Surveillance Using the Max-sum Algorithm, LNCS 6090 Self-Organizing Architectures
• [Farinelli et al. 08] Decentralised coordination of low-power embedded devices using the max-sum algorithm, AAMAS 08
References IIInstance-based Approximation• [Yeoh et al. 09] Trading off solution quality for faster computation in DCOP search algorithms,
IJCAI 09
• [Petcu and Faltings 05b] A-DPOP: Approximations in Distributed Optimization, CP 2005
• [Rogers et al. 11] Bounded approximate decentralised coordination via the max-sum algorithm, Artificial Intelligence 2011.
Instance-generic Approximation• [Vinyals et al 10b] Worst-case bounds on the quality of max-product fixed-points, NIPS 10
• [Vinyals et al 11] Quality guarantees for region optimal algorithms, AAMAS 11• [Vinyals et al 11] Quality guarantees for region optimal algorithms, AAMAS 11
• [Pearce and Tambe 07] Quality Guarantees on k-Optimal Solutions for Distributed Constraint Optimization Problems, IJCAI 07
• [Bowring et al. 08] On K-Optimal Distributed Constraint Optimization Algorithms: New Bounds and Algorithms, AAMAS 08
• [Weiss 00] Correctness of local probability propagation in graphical models with loops, Neural Computation
• [Kiekintveld et al. 10] Asynchronous Algorithms for Approximate Distributed Constraint Optimization with Quality Bounds, AAMAS 10