Sample-based Planning for Continuous Action

Markov Decision Processes[on robots]

Ari Weinstein

1

Reinforcement Learning (RL)

• Agent takes an action in the world, gets information including numerical reward; how does it learn to maximize that reward?

• Fundamental concept is exploration vs. exploitation. Must take actions in the world in order to learn about it, but eventually use what was learned to get high reward

• Bandits (stateless), Markov Decision Processes (state)

2

The Goal• I want to be here:

• Most RL algorithms are here [Knox Stone 09]:

• Some RL done with robots, but its rare, partly because its hard:

3

Overview

• RL Basics (bandits/Markov decision processes)• Planning – Bandits– MDPs (novel)

• Model Building• Exploring• Acting (novel)

Composing pieces in this manner is novel

4

k-armed Bandits• Agent selects from k-arms, each with a

distribution over rewards• If we call the arm pulled at step t at , and the

reward at t rt~R(at)

• The regret is the difference in reward between the arm pulled and optimal arm; want cumulative regret to increase sub-linearly in t

)(regret

)(maxarg

*

0t

*

aRr

aREa

tt

a

5

Hierarchical Optimistic Optimization (HOO)[Bubeck et al. 08]

• Partition action space by a tree– Keep track of rewards for each subtree

• Blue is the bandit, red is the decomposition of HOO tree– Thickness represents

estimated reward

• Tree grows deeper and builds estimates at high resolution where reward is highest

6

HOO continued

• Exploration bonuses for number of samples and size of each subregion– Regions with large volume and few samples are unknown, vice

versa• Pull arm in region according to maximal

• Has optimal regret: sqrt(t), independent of action dimension

)(),(max),(min)(

)(

)ln(2)(ˆ)(

2,112,1,,

1,

,,

tBtBtUtB

vtN

tttU

ihihihih

h

ihihih

7

Markov Decision Processes• Composed of:– States S (s, s’ from S)– Actions A (a from A)– Transition distribution T(s’|s,a)– Reward function R(s,a)– Discount factor 0<γ<1

• Goal in is to find a policy π, a mapping from states to actions, maximizing expected long term discounted reward: where rt is reward at time t.

• Maximize long-term reward but favor immediate reward more heavily; decayed by γ. How much long term reward is possible is measured by value function

][0

tt

trE

8

Value Function

• Value of a state s under policy π:

• Q-value of an action a under the same definition:

• Optimally,

)'())(,|'())(,()('

sVsssTssRsVs

)'(),|'(),(),('

sVassTasRasQs

)'(),|'(),(max)( *

'

* sVassTasRsVs

a

),(maxarg)( ** asQsa

9

Sample-based Planning [Kearns Mansour 99]

• In simplest case, agent can query domain for any:<s,a>, get <r,s’>

• Flow:– Domain informs agent of current state, s– Agent queries domain for any number of <s,a,r,s’>– Agent informs domain of true action to take, a– Domain informs agent of new state, s

10

Planning with HOO (HOLOP)

• Call this approach HOLOP – Hierarchical Open Loop Optimistic Planning

• Can treat the n-step planning problem as a large optimization problem

• Probability of splitting for a particular value of n proportional to γn

• Use HOO to optimize n-step planning, and then use action recommended in the first step.

11

1-Step Lookahead in HOLOP

• Just maximizing immediate reward, r1

• 1 dimensional; horizontal axis is splitting immediate action

12

2-Step Lookahead in HOLOP

• Maximizing r1+ γ r2

• 2 dimensional; horizontal axis is splitting immediate action, vertical is splitting next action

13

3-Step Lookahead in HOLOP

• Maximizing r1+ γ r2 + γ2 r3

• 3 dimensional; horizontal axis is splitting immediate action, vertical is splitting next action, depth is third action

14

Properties of HOLOP• Planning of HOO/HOLOP (regret) improves at rate of

sqrt(t), and independent of n • Cost independent of |S|

– Open loop control means agnostic to state• Anytime planner• Open loop control means guarantees don’t exist in noisy

domains

15

Learning System Update• If generative model is available, can use HOLOP

directly

16

HOLOP in Practice:Double Integrator Domain

[Santamaría et al. 98]

• Object with position(p) and velocity(v). Control acceleration (a). R((p,v), a) = -(p2+a2)

– Stochasticity introduced with noise added to action command• Planning done to 50 steps• As an anytime planner, can stop and ask for an action anytime (left)• Performance degrades similarly to optimal as noise varies (right)

– Action corrupted by +/- amount on x-axis, uniformly distributed. Action range is [-1.5, 1.5]

17

Building a Model: KD Trees [Moore 91]

• HOLOP needs a model – where does it come from?• KD Tree is a simple method of partitioning a space• At each node, a split is made in some dimension in the

region that node represents– Various rules for deciding when, where to split

• To make an estimation, find the leaf that the query point fits in, use some method to make an estimation– Commonly use the mean, I used linear regression

• This is used to build models of reward, transitions

18

KD Trees Approximating Gaussian• Samples drawn iid from Gaussian, labeled with pdf of Gaussian at

point• Piecewise linear fit of function

19

Learning System Update• Model and Environment now 2 pieces

– Generative model not required• Model learns from environment when true <s,a,r,s’> samples available• HOLOP uses learned model to plan

20

Efficient Exploration• Multi-resolution Exploration (MRE) [Nouri Littman 08] is a method

that allows any RL algorithm to have efficient exploration• Partitions space by a tree with each node representing

knownness based on sample density in leaf• When using samples, treat a sample as having a transition to

Smax with probability calculated by tree. Smax is a state with a self transition and maximum possible reward

• While doing rollouts in HOLOP, MRE perturbs results in order to drive the agent to explore 21

Learning System Update• When HOLOP queries model for <s,a,r,s’>, MRE can step in and lie,

say transition to Smax occurs instead– Happens with high probability if <s,a> is not well known

22

Double Integrator+/- 0.05 units uniformly distributed noise on actions• Explosion from discretization causes slow learning• Near-optimal behavior with 10 trajectories, 2000 total samples• Discrete algorithms have fast convergence to poor results, or slow

convergence to good results

23

3 Link Swimmer[Tassa et al.]

• Big domain: 2 action dimensions, 9 state dimensions– Model building needs 11 input dimensions, 10 output dimensions

• Tested a number of algorithms in this domain, HOLOP has best performance

• Rmax still worse than HOLOP after 120 trials

24

Next Step: Doing it all Quickly• In simulations, can replan with HOLOP at every

step• In real time robotics control, can’t stop world

while the algorithm plans. • Need a method of caching planning online. This

is tricky as model is updated online – when is policy updated?

• As with the other algorithms discussed (HOO, KD Trees, MRE) trees are used here. – Adaptively partitioning a space based on sample

concentration is both very powerful and very efficient

25

[Preliminary]

TCP: Tree-based Cached PlanningAlgorithm:• Start with root that covers entire state space, initialize

with exploratory action• As samples are experienced, add them to the tree,

partition nodes based on some splitting rule• Child nodes inherit action from parent on creationRunning it:• Can ask tree for an action to take from a state; leaf

returns its cached action. – If planner is not busy request it to plan from the center of

the area leaf represents in different thread

26

Cached PoliciesDouble Integrator with Generative Model

• Shade indicates action cached (black -1.5, white 1.5)• Rewards:[-44.3, -5.5, -2.2, -4.6, -21.1, -2.6] (-1.3 optimal)

27

Close UpOrder is red, green, blue, yellow, magenta, cyan• In policy error, black indicates 0 error, white indicates

maximum possible error in policy• Minimal error along optimal trajectory – errors off optimal

trajectory are acceptable

28

Learning System Update:Our Happy Forest

• Agent acts based on policy cached in TCP

• TCP sends request for updated policy, for state s

29

Conclusions• There are many existing classes of RL algorithms, but

almost all fail at least one requirement of real time robotics control. My approach addresses all requirements

• Hierarchical Open Loop Optimistic Planning is introduced:– Operates directly in continuous action spaces, and is

agnostic to state. No need for tuning– No function approximation of value function

• Tree-based Cached Planning is introduced:– Develops policies for regions when enough data is

available to accurately determine policy– Opportunistic updating of policy allows for real-time

polling, with policy updated frequently30

Citations• Bubeck et al 08:, S. Bubeck, R. Munos, G. Stoltz, C. Szepesvari. Online

Optimization in X-Armed Bandits, NIPS 08• Kearns Mansour 99: Kearns M., Mansour S., Ng A., A Sparse Sampling

Algorithm for Near-Optimal Planning in Large MDPs, IJCAI 99• Knox Stone 09: Knox B. W., Stone, P. Interactively Shaping Agents via

Human Reinforcement: The TAMER Framework. K-CAP, 2009.• Moore 91: Efficient Memory-based Learning for Robot Control. PhD.

Thesis; University of Cambridge, 1991.• Nouri Littman 08:Nouri, A. and Littman, M. L. Mutli-resolution Exploration

in Continuous Spaces. NIPS 2008• Santamaría el al 98: Santamaría, Juan C., Sutton, R., and Ram, Ashwin.

Experiments with reinforcement learning in problems with continuous state and action spaces. In Adaptive Behavior 6, 1998.

• Tassa et al.: Tassa, Yuval, Erez, Tom, and Smart, William D. Receding horizon differential dynamic programming. In Advances in Neural Information Processing Systems 21. 2007.

31