Efficient Automated Planning with New Formulations

Ruoyun HuangAdvisors: Dr. Yixin Chen and Dr. Weixiong Zhang

Washington University in St. Louis

Efficient Automated Planning with New Formulations

Ruoyun HuangAdvisors: Dr. Yixin Chen and Dr. Weixiong Zhang

Washington University in St. Louis

Dr. Yixin Chen (co-advisor)

Dr. Christopher D. Gill

Dr. Norman I. Katz

Dr. Yinjie Tang

Dr. Kilian Weinburger

Dr. Weixiong Zhang(co-advisor)

Committee Members

Classical Planning (STRIPS)loc1 loc2

Init State: (AT pkg loc1); (AT truck loc2);

loc1 loc2

Goal: (AT pkg loc2)

Applicable actions: MOVE; LOAD;UNLOAD;

?

One solution with time step N=4: Time Step 1: MOVE truck loc2 loc1 Time Step 2: LOAD pkg truck loc1Time Step 3: MOVE truck loc1 loc2 Time Step 4: UNLOAD pkg truck loc2

3

Each action is consisted of a set of preconditions and a set of effects

NASA Mars rover High speed printer in PARC Hubble space telescope[Zweben94]

Mobile robots[Talamadupula10]

Anti-air defense system[Benaskeur10]

Biological network[Bryce10]

Natural language processing[Geib07]

…

Research in “Automated” planning provides a basis for any domain specific planning applications

Applications

4

High complexity

Key Challenge

… Branching factor: O(2A)Number of states: O(2AN)…

state

……

action

…… … …

… … …

… … …

5

A: Number of actionsN: Length of plan

Expressiveness Complicated real world scenarios

Summay of DissertationClassical Planning Temporally Expressive

PlanningSTRIPS SAS+

Search

Lots of existing works.

A abstraction search method. Most existing temporal planners fall into this category.

SAT

All the existing SAT-based planners are STRIPS based.

1. Utilize SAS+ for stronger mutex constraints;

2. A new encoding scheme based on SAS+.

1. Two new encoding schemes (STRIPS and SAS+, respectively);

2. Extended to handle action costs.

Motivation: Formulation/Representation is vital to problem solving

The SAS+ formulation SAS+ on planning algorithms

Abstraction search SAS+ formulation as Satisfiability(SASE)Why SASE works?

Temporally expressive planning A planning as SAT approachApply SASE to temporal planningFurther handle action costs

Agenda

SAS+ Representation[Backstrom:96]

loc1 loc2

in truck

at loc1 at loc2

V(pkg)

at loc1 at loc2

V(truck)

AT pkg loc1 AT pkg loc2 IN pkg truck AT truck loc1 AT truck loc2

Strips SAS+

Transition: Change between values in a multi-valued variable

Transitionspkg:loc1truckpkg:truckloc1pkg:truckloc2pkg:loc2truck

9

DTG(Domain Transition Graph)

SAS+ is more compactfact…fact

… fact…fact

…

fact…

fact…

fact…

fact…fact

…

fact…

fact…

…… 2F…

… …

…… …

… ……

10F/10

F: number of STRIPS factsAssume avg. facts/value per DTG is 10

fact… fact

…

10

SAS+ also captures more hidden problem structures

11

Causal Dependency and Causal Graphloc1 loc2

at loc1 at loc2

in truck

at loc1 at loc2

pkg’s DTG

truck’s DTG

pkg’s DTG

truck’s DTG

LOAD pkg truck loc2:Pre: (AT truck loc2)

Causal Graph

12

Example of Hierarchy

To build a house

1. Drive to the city hall2. Fill out an application3. Drive to the bank4. Ask for money5. Drive somewhere6. Look for a builder7. Sign a Contract8. Build foundation9. Build frame10. Build roof11. Build wall13. Buy furniture … …

1. Get a permit2. Get enough money3. Construction4. Interior … …

1.a Drive to the city call1.b Fill out an application

2.a Drive to bank2.b Fill out an application

3.a Find a builder3.b Sign a contract3.c Build foundation3.d Build frame3.e Build roof3.f build wall

… …

13

Abstraction State Space

truck’s DTG

package’s DTG

full state space

Abstraction

loc1 loc2

1. LOAD pkg truck loc1 2. UNLOAD pkg truck loc2

1.a MOVE truck loc2 loc11.b LOAD pkg truck loc1

2.a MOVE truck loc1 loc22.b UNLOAD pkg truck loc2in truck

at loc1 at loc2

14

Abstraction State Space Search (Iteration #1)

… … … …

… … … …

Goal1Goal

2

Abstraction State SpaceDTG DTG

DTG DTG DTG DTG

DTG DTG DTG DTG

Finds a plan quickly

15

Abstraction State Space Search (Iteration #2)

… … … …

… … … …

G1 G2

Abstraction State SpaceDTG DTG

DTG DTG DTG DTG

DTG DTG DTG DTG

Keep running to improve the plan quality

SAS+ on planning algorithms Abstraction search STRIPS based planning as SAT An encoding completely based on SAS+ (SASE)

Recent work Why SASE works?

Temporally expressive planning A planning as SAT approach

Recent work: Apply SASE to temporal planning Further handle action costs

Agenda

Long distance mutual exclusion Limitations

SAS+ Planning as Satisfiability, a new encoding Motivation How it works Experimental Results Why it works

Agenda of SAT & SAS+

Two facts/actions cannot be true at the same time Fact mutual exclusion Action mutual exclusion

Mutual Exclusion (Mutex)

Move-truck-loc1-loc2

Move-truck-loc2-loc1

(in pkg truck)(at truck loc1) (at truck loc2)

(at pkg loc1) (at pkg loc2)

(in pkg truck)(at truck loc1) (at truck loc2)

(at pkg loc1) (at pkg loc2)

A key reason for the efficiency of SAT-based planning18

19

Mutual Exclusion

Time #2 Time #3 #4 #5 #6 #7 #8 #9 #10Time #1

Invalid

(at truck loc1) (at truck loc2)

20

Long Distance Mutual Exclusion(Londex)

Time #2 Time #3 #4 #5 #6 #7 #8 #9 #10Time#1

Invalid

(at truck loc1) (at truck loc4)

DTG of truck(at truck loc1) (at truck loc2) (at truck loc3) (at truck loc4)

Several limitations

Limitations of long distance mutual exclusion SAS+ not fully exploited Not memory efficient

We want to make better use of SAS+

SAS+ Based Encoding (SASE)

Actions versus TransitionsSTRIPS SAS+

Definition

a set of preconditions, a set of add effects, a set of delete effects

A set of transitions

Example

(LOAD pkg truck loc1)Pre:

(at truck loc1),(at pkg loc1) pkg:(loc1truck)

truck: (loc1loc1)Del: (at pkg loc1)Add: (in pkg truck)

Hierarchical relationships between actions and transitions

Representing Actions:

22

Motivation: Clumsy Actions AIRPORT benchmark domain[Hoffmann:04]

10 actions (in first order logic)

:action move:parameters (?a - airplane ?t - planetype ?d1 ?d2- direction ?s1 ?s2 - segment) :precondition (has-type ?a ?t) (is-moving ?a) (not (= ?s1 ?s2)) (facing ?a ?d1) (can-move ?s1 ?s2 ?d1) (move-dir ?s1 ?s2 ?d2) (at-segment ?a ?s1) (not (exists (?a1 - airplane) (and (not (= ?a1 ?a)) (blocked ?s2 ?a1)))) (forall (?s - segment) (imply (and (is-blocked ?s ?t ?s2 ?d2) (not (= ?s ?s1))) (not (occupied ?s)) ))

:effects (occupied ?s2) (blocked ?s2 ?a) (not (occupied ?s1)) (when (not (is-blocked ?s1 ?t ?s2 ?d2)) (not (blocked ?s1 ?a))) (when (not (= ?d1 ?d2)) (not (facing ?a ?d1))) (not (at-segment ?a ?s1)) (forall (?s - segment) (when (is-blocked ?s ?t ?s2 ?d2) (blocked ?s ?a))) (at-segment ?a ?s2) (when (not (= ?d1 ?d2)) (facing ?a ?d2)) ) … … More .. ….

This action can be break into 7 transitions

Agenda

A Typical Strips Encoding

(at pkg loc1)

(at truck loc2)

move

(at pkg loc1)

(at truck loc2)

(at truck loc1)

Time step 1 Time step 2

(at pkg loc1)

move

move

(at truck loc2)

(at truck loc1)

load (in pkg truck) ……………

Time step 3,4,5,…

Compile from planning graph

25

SAT Instance:VariablesClauses

Planning GraphNodes (facts, actions)Edges

Results in a whole chunk of encoding

New Encoding, An Overview

…………

SAT Instance (Part 1):transitions

SAT Instance (Part 2):matching actions and transitions (multiple independent ones)… … …

Transitions

Actions

t = 1 t =2 t =3

SAT Instance:Facts and actionsActions&

Facts

Planning graph

… … …

t = 1 t =2 t =3, 4, …

Strips Based Encoding

Transition Based Encoding26

27

Constraintstruck:loc2

Time step 1 Time step 2

……………

Time step 3,4,5,…

pkg: loc1

truck:loc1 truck:loc1

truck:loc2

truck:loc1

truck:loc2

pkg: loc1pkg: truckpkg: loc2

pkg: loc1pkg: truckpkg: loc2

Find matchings

set of actions set of actions … …

pkg: truckpkg: loc2

1. Progression of transitions over time steps (blue one implies green ones)2. Initial state and goal (Bold ones)3. Matching actions and transitions4. Action mutual exclusions and transition mutual exclusions

Results of SASE

Results of SASE

Worst Case State Space Size

…………

State Space (Part 1):O((2T)N)State Space (Part 2):O(NK)

… … …

Transitions

Actions

t = 1 t =2 t =3

State Space:O((2A)N)

ActionsPlanning graph

… … …

t = 1 t =2 t =3, 4, …

Strips Based Encoding

Transition Based Encoding

30

Total: O((2T)NNK)

K K K

Note: T < A

A Number of actionsN Number of Time Step

T Number of transitions

Worst case analysis has limitations Real SAT solvers do not enumerate Real SAT solvers may not favor transition variables

We want better understanding

Reason: significance of transition variables Why more significant? How much? Correlates to the speedup?

Why SASE works?

32

The DPLL procedure Depth First Search and Inference Used by almost all SAT solvers

SAT Algorithms

V1

-

V2

… …… …

No Solution

V5

V10

… …No

solution

Solution Found

at certain point

V7

Which variable to pick matters

33

VSIDS Heuristic Widely used, very effective h(), number of times a variable appears in all clauses EMA of all variables’ h(), clause database changes over

time

Example: (a∨b∨c) ; (a∨d); (a∨b∨d) h(a) = 3; h(b) = 2; h(c)=1

What kind of instances make VSIDS effective and what kind of instances not? Consider an extreme case: random SAT

VSIDS Heuristic

we probably want to augment the deviation of the h values of all variables34

h() on SASE Instances

…………

… … …

Transitions

Actions t = 1 t =2 t =3

36

K K K

h() on IPC DomainsAverage h of Transition Variables Average h of Action Variables

1% 2% 5% 10% 1% 2% 5% 10%Airport 98.5 75.5 59.2 23.4 39.6 35.9 34.7 32.4Depot 32.5 28.6 23.7 20.9 - - - -Driverlog 43.9 34.6 26.5 23.4 - - - -Elevator 27.0 18.9 18.9 18.9 - - - -Freecell 115.6 86.0 49.0 34.7 - - - -Openstacks 17.2 17.2 16.2 15.2 - - - -Parcprinter 44.3 42.8 26.7 17.8 30.0 30.0 30.0 30.0Pathways 33.6 26.9 15.9 14.5 - - - -Pegsol 30.0 29.8 17.2 15.5 - - - -Pipe-notank 77.1 57.5 36.4 25.3 - - - -Pipe-tankage 266.4 174.0 86.8 56.0 - - - -Rovers 175.1 86.0 35.5 35.5 - - - -Satellite 35.0 35.0 35.0 35.0 - - - -Scanalyzer 242.8 175.8 129.7 129.7 - - - -Sokoban 16.6 14.1 12.8 11.2 - - - 10.0Storage 10.4 10.4 9.0 8.1 - - - 9.0TPP 84.2 57.8 34.4 24.6 - - - -Transport 99.6 58.1 52.0 41.6 - - - -Trucks 56.7 38.2 20.8 16.3 - - - -Wood 23.1 22.2 18.9 17.3 - - - 13.1Zenotravel 51.3 51.3 36.2 28.5 - - - -

Action variables even do not exist in top percentage of variables

37

If we ranking all variables by h Transition variables dominates the head of the list More likely to be picked as decision variables

earlier (statistically)

But VSIDS heuristic dynamically updates … We need evidence that reflects the dynamic

updating

Significance of Transition Variables

38

A measurement of how often transition variables (or action variables) are chosen as decision variable

Branching Frequency

V1

-

V2

… …

V10

… …

V7

V5

… …

For every 1000 consecutive decision variables

39

Branching Frequency

Zenotravel #15 Speedup = 4.21

Openstack #5Speedup = 7.88

Transition variables have higher branching frequency in most instances

40

Branching Frequency

Storage #12Speedup = 0.83

Sokoban #8Speedup = 0.62

SASE could be less efficient in a few domains

41

Transitions variables are more likely to be decision variables

The more branching on transition variables, the higher speedup

Transition Index : An indicator of the significance

Transition Index

p

pt

VVThe ratio of transition

variables in the top ranked p percent variables

VVt Normalized by the ratio

of transition variables among all variables

42

Transition Index versus Speedup

Transition index of 1%

Transition index of 10%

43

Transition Index versus Speedup

Transition index of 50% Transition index of 90%

44

Transition variables occurs more frequently

Transition index correlates with the speedup that SASE can achieve

Conclusions

45

OutlineAutomated Planning

Classical Planning Temporally Expressive Planning

STRIPS SAS+ STRIPS

Search

SAT Search

SAT Search

SAT

4

SAS+

SAT

Actions have durations

Temporal Planning

Light a Match

find a manual

e.g. got a blackout:

Repair a fuse

Time

47

Temporally Expressive (Required Concurrency) Two actions need to be executed in parallel, in any

valid plan.

Temporally Expressive[W. Cushing et al. 2007]

adds fact (has light)

Light a match

deletes fact (has light)

Time

Read Manualcondition conditionrequires condition

(has light)

48

A planning algorithm is temporally expressive if it can handle required concurrency, otherwise it is temporally simple

Required Concurrency is important in the real world PDDL2.1 supports temporally expressiveness e.g. Firework, Missile launch

Most state of the art temporal planners are not able to handle temporally expressive

Action cost is in addition an important feature No planner can handle both temporal

expressiveness and action costs

Facts about State-of-the-art

49

Overview of Our WorkTemporally Expressive planning

STRIPS

A STRIPS based encoding [Huang09]

SAS+

SASE for temporal planning

PDDL2.1 Spec

Cost Sensitive Temporally Expressive Planning

Extend to support action costs

MaxSAT solvers A Branch & Bound Algorithm

50

A MinCost SAT Formulation

Durative Actions as SAS+

adds fact (has light)

Light a match

deletes fact (has light)

Time

Light a match

Time

51

SASE with Durative Actions

Experimental Results

PET and SET are our SAT-based approach, with different encoding schemes;Crikey2, Crikey3, TFD, LPG-c are several heuristic search based planners53

Cost Sensitive Temporally Expressive Planning

MaxSAT solvers A Branch & Bound Algorithm

MinCost

54

Extend the encoding into MinCost SAT problem MinCost: variables have costs, to be minimized

1. Convert MinCost instances into MaxSAT MaxSAT is widely accept

2. A planning specialized branch and bound algorithm

Use relaxed planning graph to estimate a heuristic function

Cost Sensitive Temporally Expressive

55

Results

56

More efficient algorithmSearchPlanning as SAT

More expressive featuresTemporal expressivenessAction costs

Summary

57

Other planning algorithmscausal link search

Other planning modelsprobabilistic, uncertainty, etc

Beyond planning researchBounded model checking[Clarke01]

Answer set programming[Lin02]

Significance of Research

58

Integrate existing techniques into SASE

More understanding on what makes an SAT instance easy or hard Backdoor set / Backbone set Goal: improve general SAT solving

More advanced planning features Preferences, numerics, etc.

SAS+ for other planning techniques

Future Directions

59

R. Huang, Y. Chen, and W. Zhang, “SAS+ planning as Satisfiability”. Journal of Artificial Intelligence Research, invited and submitted

R. Huang, Y. Chen, and W. Zhang, “A Novel Transition Based Encoding Scheme for Planning as Satisfiability”. AAAI’10

R. Huang, Y. Chen, and W. Zhang, “An Optimal Temporally Expressive Planner: Initial Results and Application to P2P Network Optimization”. ICAPS’09

Y. Chen, R. Huang, Z. Xing and W. Zhang, “Long-distance Mutual Exclusion for Planning”. Artificial Intelligence Journal, 2009

Y. Chen, R. Huang, and W. Zhang, “Fast Planning by Search in Domain Transition Graphs”. AAAI’08

C. Hsu, B. Wah, R. Huang, and Y. Chen, “ Constraint Partitioning for Solving Planning Problems with Trajectory Constraints and Goal Preferences”. IJCAI’07

Publications

60

Thank You