In practice, we run into three common issues faced by concurrent optimization algorithms. We alter our model-shaping to mitigate these by reasoning about the types of interactions we have:– Slow convergence Prioritization– Oscillation Probabilistic shaping– Local optima Optimistic policy initialization
Distributed POMDPs with Coordination Locales (DPCLs)This work uses the DPCL problem model[2]. DPCLs are similar to Dec-POMDPs in representing problems as sets of states, actions and observations with joint transition, reward, and observation functions. However, DPCLs differ in that they factor state space into global and per-agent local components, and interactions among agents are limited to coordination locales.
Evaluation in a Heterogeneous Rescue Robot DomainConsider the problem of a team of robots planning to search a disaster area. Some robots can assist victims, while others can clear otherwise intraversable debris. Robot observations and movements are subject to uncertainty. We evaluate D-TREMOR’s performance on a number of these planning problems, in teams of up to 100 agents.
AcknowledgementsThis research has been funded in part by the AFOSR MURI grant FA9550-08-1-0356. This material is based upon work supported under a National Science Foundation Graduate Research Fellowship.
Tractable Planning in Large TeamsEmerging team applications require the cooperation of 100s of members (humans, robots, agents). Team members must complete complex, collaborative tasks in dynamic and uncertain environments. How can we effectively and tractably plan in these domains?
Distributed Model Shaping for Scaling to Decentralized POMDPs with Hundreds of Agents
Prasanna Velagapudi, Pradeep Varakantham, Paul Scerri, Katia Sycara
Scaling up from TREMOR[2] to D-TREMOR
Conclusions and Future WorkWe introduce D-TREMOR, an approach to scale distributed planning under uncertainty into the hundreds of agents using information exchange and model-shaping. Results suggest competitive performance while improving scalability and reducing computational cost. We are working to further improve performance through better modeling of interaction dynamics and intelligent information dissemination between agents.
References[1] M. Kearns, Y. Mansour, and A. Y. Ng. A sparse sampling algorithm for near-optimal planning in large Markov decision
processes. Machine Learning. 2002.[2] P. Varakantham, J. Kwak, M. Taylor, J. Marecki, P. Scerri, and M. Tambe. Exploiting Coordination Locales in Distributed
POMDPs via Social Model Shaping. Proc. of ICAPS, 2009. [3] P. Varakantham, R.T. Maheswaran, T. Gupta, and M. Tambe. Towards Efficient Computation of Error Bounded Solutions in
POMDPs: Expected Value Approximation and Dynamic Disjunctive Beliefs. Proc. of IJCAI, 2007.
Role Allocation Policy Solution Interaction Detection CoordinationTREMOR
Branch & Bound MDP
Independent EVA[3] solvers
Joint policy evaluation
Reward shapingof local models
D-TREMOR
DecentralizedAuction
Sampling & message passing
Reward shaping of local models with
convergence heuristics
Search and Rescue
Disaster Response
Command & Control
Rescue Agent
Cleaner Agent
Narrow Corridor
Victim
Unsafe Cell
Clearable Debris
Example Map: Rescue Domain
Objective function: Get rescue agents to as many victims as possible within a fixed time horizon while minimizing collisions and unsafe cells.
Agents can collide in narrow corridors (only one agent can fit at a time) and with clearable debris (blocks rescue agents, but can be cleared by cleaner agents).
• D-TREMOR policies– Max-joint-value– Last iteration
• Comparison policies– Independent– Optimistic– Do-nothing– Random
• Scaling dataset:– 10 to 100 agents– Random maps
• Density dataset– 100 agents– Concentric ring maps
• 3 problems/condition• 20 planning iterations• 7 time step horizon• 1 CPU per agent
D-TREMOR:Distributed Team REshaping of Models for Rapid-executionWe extend the TREMOR[2] algorithm for solving DPCLs to produce D-TREMOR, a fully-distributed solver that scales to problems with hundreds of agents. It approximates DPCLs as a set of single-agent POMDPs which are solved in parallel, then iteratively reshaped using messages that describe CL interactions between agent policies.
D-TREMOR rescues many more victims.
D-TREMOR resolves many, but not all collisions.
Task Allocation
Local Planning
Interaction Exchange
Model Shaping
Finding the probability of a CL[1]:• Evaluate local policy
• Compute frequency of associated si, ai
Entered corridor in 95 of 100
runs: PrCL= 0.95
Finding the value of a CL[1]:• Sample local policy value
with/without interactions– Test interactions independently
• Compute change in value if interaction occurred
No collision
Collision
+1
-6
ValCL= -7
• Send CL messages to teammates:
• Sparsity Relatively small # of messages
• Shape local model rewards/transitions based on remote interactions
Probability of interaction
Interactionmodel functions
Independentmodel functions
• Re-solve shaped local models to get new policies
• Result: new locally-optimal policies new interactions
Distributed Interaction Detection using Sampling and Message Exchange
Improved model shaping of local agent models with convergence heuristics
Increases in time are related to # of CLs, not # of agents.
Results of Scaling Dataset
Results of Density Dataset
Naïve Policies (optimistic, do-nothing, random)
D-TREMOR Policies
Number of agents and map size are varied as density of debris, corridors, and unsafe cells is held constant.
Concentric rings of narrow corridors are added from outside in on a map where victims are at the center.
High risk of collision penalties makes do-nothing seem competitive.
Ignoring interactions = poor performanceIndependent
& Optimistic
Independent & Optimistic
Independent & Optimistic
?
Agents not interacting, use independent functions:
Agents are interacting, use joint CL functions:
CL =Nature of time
constraint (e.g. affects only same-time, affects
any future-time)
Coordination Locales define regions of state-action space where joint transition/reward functions are needed
Relevant region of joint state-action space
: Joint Transition
: Joint Reward
: Joint Observation
: Set of States:
: Set of Actions:
: Set of Observations:
: Initial Belief StateTime
constraint