Embed Size (px)
Citation preview
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Following Natural Language Route Instructions
Matt [email protected]
Computer Sciences Intelligent Robotics LaboratoryPsychology Shape and Space Laboratory
Dept. of Electrical and Computer EngineeringUniversity of Texas at Austin
Ph.D. Defense
June 19, 2007
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 1
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Task Overview and MotivationRoute Instruction DefinitionsThesis and ContributionsTalk Roadmap
Introduction to Route InstructionsTask Overview and MotivationRoute Instruction DefinitionsThesis and ContributionsTalk Roadmap
MARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task Corpus
MARCO Performance and Ablation Experiments
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 2
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Task Overview and MotivationRoute Instruction DefinitionsThesis and ContributionsTalk Roadmap
Route Instruction Following
Natural language understanding for spatial navigation
Motivation: Guide robotnavigation with natural language
Route instructions connectlanguage, knowledge, action
Integrate intelligent capabilities
Investigate human behavior
Follow natural instructionserrors, assumptions, omissionsWheelchair, USAR, EVA Asst.
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 3
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Task Overview and MotivationRoute Instruction DefinitionsThesis and ContributionsTalk Roadmap
Cinematic examples
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 4
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Task Overview and MotivationRoute Instruction DefinitionsThesis and ContributionsTalk Roadmap
Situated instructions give linguistic and spatial context
Route Instruction Text 1
1. face the chair,
2. move to it,
3. turn left,
4. walk along the bluepast a sofa to thehat rack,
5. turn left,
6. move into the corner,
7. this is 5
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 5
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Task Overview and MotivationRoute Instruction DefinitionsThesis and ContributionsTalk Roadmap
Infer procedures from linguistic and spatial contextRoute Instruction Text 2
1. take the wood pathtowards the red brickintersection.
2. turn left onto thered brick.
3. at the chair, goright.
4. take a left onto theblue path
5. and then another leftat the coat rack.
6. at the corner, youare at position 5.
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 6
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Task Overview and MotivationRoute Instruction DefinitionsThesis and ContributionsTalk Roadmap
Route Instruction Definitions
Route instructions explain how to move from one place to anothera.k.a. “route directions,” “directions,” and “route guidance”
Director
I Knows environmentI Plans routeI Describes route
Follower
I Does not know environmentI Interprets route instructionsI Navigates to destination
Outcome of follower’s spatialsequential decision-making
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 7
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Task Overview and MotivationRoute Instruction DefinitionsThesis and ContributionsTalk Roadmap
Route Instruction Definitions
Route instructions explain how to move from one place to anothera.k.a. “route directions,” “directions,” and “route guidance”
Director
I Knows environmentI Plans routeI Describes route
Follower
I Does not know environmentI Interprets route instructionsI Navigates to destination
Outcome of follower’s spatialsequential decision-making
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 7
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Task Overview and MotivationRoute Instruction DefinitionsThesis and ContributionsTalk Roadmap
Route Instruction Definitions
Route instructions explain how to move from one place to anothera.k.a. “route directions,” “directions,” and “route guidance”
Director
I Knows environmentI Plans routeI Describes route
Follower
I Does not know environmentI Interprets route instructionsI Navigates to destination
Outcome of follower’s spatialsequential decision-making
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 7
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Task Overview and MotivationRoute Instruction DefinitionsThesis and ContributionsTalk Roadmap
Thesis and Contributions
I MARCO architecture can follow natural route instructions
I Measure impact of instruction-following behaviors
I What is necessary to follow natural instructions, and how often?
I Evaluate against language and task corpus
I How often are implicit procedures required?
I How often are fundamental navigation skills required?
I How often are different types of landmarks required?
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 8
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Task Overview and MotivationRoute Instruction DefinitionsThesis and ContributionsTalk Roadmap
Thesis and ContributionsI MARCO architecture can follow natural route instructions
I Measure impact of instruction-following behaviors
I What is necessary to follow natural instructions, and how often?
I Evaluate against language and task corpusI People describing routes learned by navigationI People following routes from instructionsI MARCO agent following routes from instructions
I How often are implicit procedures required?
I How often are fundamental navigation skills required?
I How often are different types of landmarks required?
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 8
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Task Overview and MotivationRoute Instruction DefinitionsThesis and ContributionsTalk Roadmap
Thesis and Contributions
I MARCO architecture can follow natural route instructions
I Measure impact of instruction-following behaviors
I What is necessary to follow natural instructions, and how often?
I Evaluate against language and task corpus
I How often are implicit procedures required?
I How often are fundamental navigation skills required?
I How often are different types of landmarks required?
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 8
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Task Overview and MotivationRoute Instruction DefinitionsThesis and ContributionsTalk Roadmap
Cognitive Skills and Route Instructions
I Natural language processing
I Procedure execution
I Spatial reasoning
I World modeling and acting
MARCO Route Follower
1. Parse route instruction text
2. Frame surface semantics
3. Model actions & conditions4. for instruction in model:
a. Interpret in contextb. Infer implicit
proceduresc. Act in the world
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 9
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Task Overview and MotivationRoute Instruction DefinitionsThesis and ContributionsTalk Roadmap
Evaluation: Language and Task Corpus
1. Directorexplores freely
Human
MacMahon &Stankiewicz,CogSci ’06
2. Director typesinstructions
Human
3. Followersread instructions
Human
MARCO
Go three segments
down the grassy hall
to the blue-tiled
hall, passing the
hatrack. Turn left
into the blue-tiled
hall. Follow it ...
4. Followersnavigate route
Human
MARCO variantsfiEnd, Wall, End,
Wall, Empty, Yellow
flfi
Fish, Grass, Fish,Wall, Empty, Cement
flfi
Fish, Grass, Fish,Wall, Hatrack, Brick
fl
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 10
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Task Overview and MotivationRoute Instruction DefinitionsThesis and ContributionsTalk Roadmap
Talk Outline
Introduction to Route Instructions
MARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task Corpus
MARCO Performance and Ablation Experiments
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 11
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
MARCO Route Instruction Understanding OverviewRoute Instruction Following OverviewRelated WorkFollowing One Command, in Different Contexts
Introduction to Route Instructions
MARCO Architecture for Modeling and Following InstructionsMARCO Route Instruction Understanding OverviewRoute Instruction Following OverviewRelated WorkFollowing One Command, in Different Contexts
Evaluation through Language and Task Corpus
MARCO Performance and Ablation Experiments
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 12
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
MARCO Route Instruction Understanding OverviewRoute Instruction Following OverviewRelated WorkFollowing One Command, in Different Contexts
MARCO Instruction Processing Pipeline
Route Instruction Text
↓
Imperative Procedural Model
↓↑
Movement Actions,Match View Description to
Observations
MARCO Route Follower
1. Parse route instruction text
2. Frame surface semantics
3. Model actions & conditions4. for instruction in model:
a. Interpret in contextb. Infer implicit
proceduresc. Act in the world
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 13
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
MARCO Route Instruction Understanding OverviewRoute Instruction Following OverviewRelated WorkFollowing One Command, in Different Contexts
Do what the director explicitly commandedRoute Text
“turn to facethe green hallway,”
SYNTAXPARSER
⇓Parse Tree
⇓
TURN
TURNV
turn
PURPOSE
PURPOSEP
to
S
FACEV
face
PATH
DET
the
APPEAR
green
PATHN
hallway
CONTENTFRAMER
⇓Content Frame
⇓
2
6
6
6
6
6
6
6
6
6
6
6
6
4
TURN2
6
6
6
6
6
6
6
6
6
4
V/TURN Turn_V_1
PURPOSE2
6
6
6
6
4
V/FACE Face_V_1
PATH
2
6
4
DEF +APPEAR Green_ADJ_1PATH Hall_N_1
3
7
5
3
7
7
7
7
5
3
7
7
7
7
7
7
7
7
7
5
3
7
7
7
7
7
7
7
7
7
7
7
7
5
INSTRUCTIONMODELER
⇓Procedural
Specification⇓
Turnp(until=( object=Path, appear=Green,side=Front, dist=0:
))
EXECUTOR⇓
InstantiatedReactive
Procedure⇓
until Verifyp( object=Path, appear=Green,side=Front, dist=0:
):
do turna(left)
MARCO Route Follower
1. Parse route instruction textSemantic PCFG
2. Frame surface semanticsMatrix of phrases &WordNet senses
3. Model actions & conditionsReactive procedures forstated conditions
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 13
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
MARCO Route Instruction Understanding OverviewRoute Instruction Following OverviewRelated WorkFollowing One Command, in Different Contexts
Do the procedures the director said or intended
Route Text “walk forward three times,”
SYNTAXPARSER
⇓Parse Tree
⇓
TRAVEL
TRAVELV
walk
DIR
forward
DIST
COUNT
three
UNIT
times
CONTENTFRAMER
⇓Content Frame
⇓
2
6
6
6
6
6
6
6
4
TRAVEL2
6
6
6
6
4
V/TRAVEL Walk_V_1DIR Forward_ADV_1
DIST
"
COUNT 3UNIT Time_N_2
#
3
7
7
7
7
5
3
7
7
7
7
7
7
7
5
INSTRUCTIONMODELER
⇓Procedural
Specification⇓
Travelp(dir=Forward, dist=3)
EXECUTOR⇓
InstantiatedReactive
Procedure⇓
Facep( object=Path, side=Front, dist=0)for i in 1::3:travela(forward)
MARCO Route Follower
1. Parse route instruction textSemantic PCFG
2. Frame surface semanticsMatrix of phrases &WordNet senses
3. Model actions & conditionsReactive procedures forstated & implied conditions
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 13
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
MARCO Route Instruction Understanding OverviewRoute Instruction Following OverviewRelated WorkFollowing One Command, in Different Contexts
Do what the director explicitly commandedRoute Text
“turn to facethe green hallway,”
SYNTAXPARSER
⇓Parse Tree
⇓
TURN
TURNV
turn
PURPOSE
PURPOSEP
to
S
FACEV
face
PATH
DET
the
APPEAR
green
PATHN
hallway
CONTENTFRAMER
⇓Content Frame
⇓
2
6
6
6
6
6
6
6
6
6
6
6
6
4
TURN2
6
6
6
6
6
6
6
6
6
4
V/TURN Turn_V_1
PURPOSE2
6
6
6
6
4
V/FACE Face_V_1
PATH
2
6
4
DEF +APPEAR Green_ADJ_1PATH Hall_N_1
3
7
5
3
7
7
7
7
5
3
7
7
7
7
7
7
7
7
7
5
3
7
7
7
7
7
7
7
7
7
7
7
7
5
INSTRUCTIONMODELER
⇓Procedural
Specification⇓
Turnp(until=( object=Path, appear=Green,side=Front, dist=0:
))
EXECUTOR⇓
InstantiatedReactive
Procedure⇓
until Verifyp( object=Path, appear=Green,side=Front, dist=0:
):
do turna(left)
MARCO Route Follower
1. Parse route instruction textSemantic PCFG
2. Frame surface semanticsMatrix of phrases &WordNet senses
3. Model actions & conditionsReactive procedures forstated & implied conditions
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 13
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
MARCO Route Instruction Understanding OverviewRoute Instruction Following OverviewRelated WorkFollowing One Command, in Different Contexts
Do what the director explicitly commandedRoute Text
“turn to facethe green hallway,”
SYNTAXPARSER
⇓Parse Tree
⇓
TURN
TURNV
turn
PURPOSE
PURPOSEP
to
S
FACEV
face
PATH
DET
the
APPEAR
green
PATHN
hallway
CONTENTFRAMER
⇓Content Frame
⇓
2
6
6
6
6
6
6
6
6
6
6
6
6
4
TURN2
6
6
6
6
6
6
6
6
6
4
V/TURN Turn_V_1
PURPOSE2
6
6
6
6
4
V/FACE Face_V_1
PATH
2
6
4
DEF +APPEAR Green_ADJ_1PATH Hall_N_1
3
7
5
3
7
7
7
7
5
3
7
7
7
7
7
7
7
7
7
5
3
7
7
7
7
7
7
7
7
7
7
7
7
5
INSTRUCTIONMODELER
⇓Procedural
Specification⇓
Turnp(until=( object=Path, appear=Green,side=Front, dist=0:
))
EXECUTOR⇓
InstantiatedReactive
Procedure⇓
until Verifyp( object=Path, appear=Green,side=Front, dist=0:
):
do turna(left)
MARCO Route Follower
1. Parse route instruction textSemantic PCFG
2. Frame surface semanticsMatrix of phrases &WordNet senses
3. Model actions & conditionsReactive procedures forstated & implied conditions
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 13
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
MARCO Route Instruction Understanding OverviewRoute Instruction Following OverviewRelated WorkFollowing One Command, in Different Contexts
Focus within Route InstructionsLimit to a static series of text instructions
Current workI natural languageI series of commands, descsI text references to worldI find references, conditionsI act, perceive to disambiguateI executing procedures in simI learning syntaxI unreliable instructionsI complex routes through
unknown, large-scale space
Future workI speech recognitionI discourse structureI maps and gesturesI unifying elaborationsI dialog to disambiguateI physical robotI learning semanticsI unreliable sensors, actionsI small-scale spatial &
non-spatial actions
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 14
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
MARCO Route Instruction Understanding OverviewRoute Instruction Following OverviewRelated WorkFollowing One Command, in Different Contexts
Related Work and Spatial ScalesBy Freundschuh and Egenhofer’s division of spatial models
Manipulable, Object SpaceReachable, Table-topWinograd, Deb Roy,Yu & BallardImmediate Space
Visible, Panoramic SpaceRooms, Open SpacesSkubic, Schultz, TellexSmall-Scale Space
Navigable, Environmental SpaceBeyond the sensory horizonIBL, Simmons, BremenLarge-Scale Space
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 15
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
MARCO Route Instruction Understanding OverviewRoute Instruction Following OverviewRelated WorkFollowing One Command, in Different Contexts
Related Work and Spatial ScalesBy Freundschuh and Egenhofer’s division of spatial models
Manipulable, Object SpaceReachable, Table-topWinograd, Deb Roy,Yu & BallardImmediate Space
Visible, Panoramic SpaceRooms, Open SpacesSkubic, Schultz, TellexSmall-Scale Space
Navigable, Environmental SpaceBeyond the sensory horizonIBL, Simmons, BremenLarge-Scale Space
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 15
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
MARCO Route Instruction Understanding OverviewRoute Instruction Following OverviewRelated WorkFollowing One Command, in Different Contexts
Related Work and Spatial ScalesBy Freundschuh and Egenhofer’s division of spatial models
Manipulable, Object SpaceReachable, Table-topWinograd, Deb Roy,Yu & BallardImmediate Space
Visible, Panoramic SpaceRooms, Open SpacesSkubic, Schultz, TellexSmall-Scale Space
Navigable, Environmental SpaceBeyond the sensory horizonIBL, Simmons, BremenLarge-Scale Space
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 15
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
MARCO Route Instruction Understanding OverviewRoute Instruction Following OverviewRelated WorkFollowing One Command, in Different Contexts
Do what the director explicitly commandedRoute Text
“turn to facethe green hallway,”
SYNTAXPARSER
⇓Parse Tree
⇓
TURN
TURNV
turn
PURPOSE
PURPOSEP
to
S
FACEV
face
PATH
DET
the
APPEAR
green
PATHN
hallway
CONTENTFRAMER
⇓Content Frame
⇓
2
6
6
6
6
6
6
6
6
6
6
6
6
4
TURN2
6
6
6
6
6
6
6
6
6
4
V/TURN Turn_V_1
PURPOSE2
6
6
6
6
4
V/FACE Face_V_1
PATH
2
6
4
DEF +APPEAR Green_ADJ_1PATH Hall_N_1
3
7
5
3
7
7
7
7
5
3
7
7
7
7
7
7
7
7
7
5
3
7
7
7
7
7
7
7
7
7
7
7
7
5
INSTRUCTIONMODELER
⇓Procedural
Specification⇓
Turnp(until=( object=Path, appear=Green,side=Front, dist=0:
))
EXECUTOR⇓
InstantiatedReactive
Procedure⇓
until Verifyp( object=Path, appear=Green,side=Front, dist=0:
):
do turna(left)
MARCO Route Follower
1. Parse route instruction textSemantic PCFG
2. Frame surface semanticsMatrix of phrases &WordNet senses
3. Model actions & conditionsReactive procedures forstated conditions
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 16
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
MARCO Route Instruction Understanding OverviewRoute Instruction Following OverviewRelated WorkFollowing One Command, in Different Contexts
Model instructions as procedures with conditions
Route navigation procedurep Conditions for executing actionsa
Travelp along:Path(appear:Blue,side:Front,dist:0)until:Sofa(side:At,dist:0)
I turna at a placeI travela between
placesI verifya a View Desc.I declarea done
Match view description against viewI Pre/ Post conditionsI While/ Until conditions
Maintain state over procedureI Distance, Past conditionsI View behind
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 17
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
MARCO Route Instruction Understanding OverviewRoute Instruction Following OverviewRelated WorkFollowing One Command, in Different Contexts
Spatial Route Instruction Procedures & Surface Forms
TURNp Switch paths orchange orientation
TRAVELp Traversal alonga path
VERIFYp Describeenvironment
DECLAREp Assert a nameor end.
FINDp Undirected searchfor desc.
FOLLOWp Followconstrained route
Corpus Examples:I turn to face the green
hallway.
I go all the way down thered brick corridor.
I The side alleys are redbrick.
I At the sofa, you are atPosition 2.
I Find the intersection ofthe red and blue hallways.
I go all the way down thewinding hallway.
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 18
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
MARCO Route Instruction Understanding OverviewRoute Instruction Following OverviewRelated WorkFollowing One Command, in Different Contexts
Spatial Route Instruction Procedures & Surface Forms
TURNp Switch paths orchange orientation
TRAVELp Traversal alonga path
VERIFYp Describeenvironment
DECLAREp Assert a nameor end.
FINDp Undirected searchfor desc.
FOLLOWp Followconstrained route
Corpus Examples:I turn to face the green
hallway.
I go all the way down thered brick corridor.
I The side alleys are redbrick.
I At the sofa, you are atPosition 2.
I Find the intersection ofthe red and blue hallways.
I go all the way down thewinding hallway.
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 18
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
MARCO Route Instruction Understanding OverviewRoute Instruction Following OverviewRelated WorkFollowing One Command, in Different Contexts
Spatial Route Instruction Procedures & Surface Forms
TURNp Switch paths orchange orientation
TRAVELp Traversal alonga path
VERIFYp Describeenvironment
DECLAREp Assert a nameor end.
FINDp Undirected searchfor desc.
FOLLOWp Followconstrained route
Corpus Examples:I turn to face the green
hallway.
I go all the way down thered brick corridor.
I The side alleys are redbrick.
I At the sofa, you are atPosition 2.
I Find the intersection ofthe red and blue hallways.
I go all the way down thewinding hallway.
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 18
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
MARCO Route Instruction Understanding OverviewRoute Instruction Following OverviewRelated WorkFollowing One Command, in Different Contexts
Spatial Route Instruction Procedures & Surface Forms
TURNp Switch paths orchange orientation
TRAVELp Traversal alonga path
VERIFYp Describeenvironment
DECLAREp Assert a nameor end.
FINDp Undirected searchfor desc.
FOLLOWp Followconstrained route
Corpus Examples:I turn to face the green
hallway.
I go all the way down thered brick corridor.
I The side alleys are redbrick.
I At the sofa, you are atPosition 2.
I Find the intersection ofthe red and blue hallways.
I go all the way down thewinding hallway.
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 18
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
MARCO Route Instruction Understanding OverviewRoute Instruction Following OverviewRelated WorkFollowing One Command, in Different Contexts
Spatial Route Instruction Procedures & Surface Forms
TURNp Switch paths orchange orientation
TRAVELp Traversal alonga path
VERIFYp Describeenvironment
DECLAREp Assert a nameor end.
FINDp Undirected searchfor desc.
FOLLOWp Followconstrained route
Corpus Examples:I turn to face the green
hallway.
I go all the way down thered brick corridor.
I The side alleys are redbrick.
I At the sofa, you are atPosition 2.
I Find the intersection ofthe red and blue hallways.
I go all the way down thewinding hallway.
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 18
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
MARCO Route Instruction Understanding OverviewRoute Instruction Following OverviewRelated WorkFollowing One Command, in Different Contexts
Spatial Route Instruction Procedures & Surface Forms
TURNp Switch paths orchange orientation
TRAVELp Traversal alonga path
VERIFYp Describeenvironment
DECLAREp Assert a nameor end.
FINDp Undirected searchfor desc.
FOLLOWp Followconstrained route
Corpus Examples:I turn to face the green
hallway.
I go all the way down thered brick corridor.
I The side alleys are redbrick.
I At the sofa, you are atPosition 2.
I Find the intersection ofthe red and blue hallways.
I go all the way down thewinding hallway.
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 18
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
MARCO Route Instruction Understanding OverviewRoute Instruction Following OverviewRelated WorkFollowing One Command, in Different Contexts
Spatial Route Instruction Procedures & Surface Forms
TURNp Switch paths orchange orientation
TRAVELp Traversal alonga path
VERIFYp Describeenvironment
DECLAREp Assert a nameor end.
FINDp Undirected searchfor desc.
FOLLOWp Followconstrained route
Turn Examples Surface Variety:I turn to face the green
hallway.
I turn right.
I take a left onto the bluepath.
I face the chair.
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 18
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
MARCO Route Instruction Understanding OverviewRoute Instruction Following OverviewRelated WorkFollowing One Command, in Different Contexts
Recognizing Context: Views and View Descriptions
View Descriptions: Constraints on the expected view
What – Object & Perceptual QualitiesObject, Structural, AppearanceLandmarks
Where – Spatial RelationshipsRelative direction, distance
Take the yellow hallto the blue hall.
Travelp(along:Path(appear:yellow,dist:0, dir:Front),until:Path(appear:blue,dist:0, dir:At))
At the corner,turn left.
Turnp(dir:Left,Loc:Structure(type:Corner,dist:0, dir:At))
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 19
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
MARCO Route Instruction Understanding OverviewRoute Instruction Following OverviewRelated WorkFollowing One Command, in Different Contexts
Recognizing Context: Views and View Descriptions
View Descriptions: Constraints on the expected view
What – Object & Perceptual QualitiesObject, Structural, AppearanceLandmarks
Where – Spatial RelationshipsRelative direction, distance
Take the yellow hallto the blue hall.
Travelp(along:Path(appear:yellow,dist:0, dir:Front),until:Path(appear:blue,dist:0, dir:At))
At the corner,turn left.
Turnp(dir:Left,Loc:Structure(type:Corner,dist:0, dir:At))
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 19
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
MARCO Route Instruction Understanding OverviewRoute Instruction Following OverviewRelated WorkFollowing One Command, in Different Contexts
Implicit Procedures in Route Instructions
Route instructions mostly explicit and sequential
Implicit procedures necessary in some contexts – how often
Instruction text and explicitprocedure:
“Take the blue hall tothe sofa”
Travelp down blue hall until sofa
Possible implicitly commandedprocedures:
Turnp to face blue hall
Turnp to face sofa
Travelp to blue hall
Findp the blue hall
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 20
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
MARCO Route Instruction Understanding OverviewRoute Instruction Following OverviewRelated WorkFollowing One Command, in Different Contexts
Implicit Procedures in Route Instructions
Route instructions mostly explicit and sequential
Implicit procedures necessary in some contexts – how often
Instruction text and explicitprocedure:
“Take the blue hall tothe sofa”
Travelp down blue hall until sofa
Possible implicitly commandedprocedures:
Turnp to face blue hall
Turnp to face sofa
Travelp to blue hall
Findp the blue hall
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 20
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
MARCO Route Instruction Understanding OverviewRoute Instruction Following OverviewRelated WorkFollowing One Command, in Different Contexts
Perform the explicit procedure
“Take the blue hall to the sofa”
Travelp(along:Path(appear:BlueTile,
side:Front, dist:0),until:Sofa(side:At,dist:0)
)
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 21
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
MARCO Route Instruction Understanding OverviewRoute Instruction Following OverviewRelated WorkFollowing One Command, in Different Contexts
Perform the explicit procedure
“Take the blue hall to the sofa”
Travelp(along:Path(appear:BlueTile,
side:Front, dist:0),until:Sofa(side:At,dist:0)
)
travela forward
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 21
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
MARCO Route Instruction Understanding OverviewRoute Instruction Following OverviewRelated WorkFollowing One Command, in Different Contexts
Perform the explicit procedure
“Take the blue hall to the sofa”
Travelp(along:Path(appear:BlueTile,
side:Front, dist:0),until:Sofa(side:At,dist:0)
)
travela forward
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 21
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
MARCO Route Instruction Understanding OverviewRoute Instruction Following OverviewRelated WorkFollowing One Command, in Different Contexts
Perform the explicit procedure
“Take the blue hall to the sofa”
Travelp(along:Path(appear:BlueTile,
side:Front, dist:0),until:Sofa(side:At,dist:0)
)
Instruction Done
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 21
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
MARCO Route Instruction Understanding OverviewRoute Instruction Following OverviewRelated WorkFollowing One Command, in Different Contexts
Implicit turn procedure necessary to travel
“Take the blue hall to the sofa”
Turnp(until:Path(appear:BlueTile,
side:Front, dist:0),Sofa(side:Front, dist:1-∞)
)Travelp(along:Path(appear:BlueTile,
side:Front, dist:0),until:Sofa(side:At,dist:0)
)
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 21
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
MARCO Route Instruction Understanding OverviewRoute Instruction Following OverviewRelated WorkFollowing One Command, in Different Contexts
Implicit turn procedure necessary to travel
“Take the blue hall to the sofa”
Turnp(until:Path(appear:BlueTile,
side:Front, dist:0),Sofa(side:Front, dist:1-∞)
)Travelp(along:Path(appear:BlueTile,
side:Front, dist:0),until:Sofa(side:At,dist:0)
)
[turna left]
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 21
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
MARCO Route Instruction Understanding OverviewRoute Instruction Following OverviewRelated WorkFollowing One Command, in Different Contexts
Implicit turn procedure necessary to travel
“Take the blue hall to the sofa”
Turnp(until:Path(appear:BlueTile,
side:Front, dist:0),Sofa(side:Front, dist:1-∞)
)Travelp(along:Path(appear:BlueTile,
side:Front, dist:0),until:Sofa(side:At,dist:0)
)
travela forward
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 21
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
MARCO Route Instruction Understanding OverviewRoute Instruction Following OverviewRelated WorkFollowing One Command, in Different Contexts
Implicit turn procedure necessary to travel
“Take the blue hall to the sofa”
Turnp(until:Path(appear:BlueTile,
side:Front, dist:0),Sofa(side:Front, dist:1-∞)
)Travelp(along:Path(appear:BlueTile,
side:Front, dist:0),until:Sofa(side:At,dist:0)
)
travela forward
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 21
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
MARCO Route Instruction Understanding OverviewRoute Instruction Following OverviewRelated WorkFollowing One Command, in Different Contexts
Implicit turn procedure necessary to travel
“Take the blue hall to the sofa”
Turnp(until:Path(appear:BlueTile,
side:Front, dist:0),Sofa(side:Front, dist:1-∞)
)Travelp(along:Path(appear:BlueTile,
side:Front, dist:0),until:Sofa(side:At,dist:0)
)
Instruction Done
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 21
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
MARCO Route Instruction Understanding OverviewRoute Instruction Following OverviewRelated WorkFollowing One Command, in Different Contexts
Implicit travel and turn procedures necessary
“Take the blue hall to the sofa”Travelp(until:Path(appear:BlueTile,
side:Front, dist:0))Turnp(until:Path(appear:BlueTile,
side:Front, dist:0),Sofa(side:Front, dist:1-∞)
)Travelp(along:Path(appear:BlueTile,
side:Front, dist:0),until:Sofa(side:At,dist:0)
)Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 21
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
MARCO Route Instruction Understanding OverviewRoute Instruction Following OverviewRelated WorkFollowing One Command, in Different Contexts
Implicit travel and turn procedures necessary
“Take the blue hall to the sofa”Travelp(until:Path(appear:BlueTile,
side:Front, dist:0))Turnp(until:Path(appear:BlueTile,
side:Front, dist:0),Sofa(side:Front, dist:1-∞)
)Travelp(along:Path(appear:BlueTile,
side:Front, dist:0),until:Sofa(side:At,dist:0)
)
[travela forward]
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 21
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
MARCO Route Instruction Understanding OverviewRoute Instruction Following OverviewRelated WorkFollowing One Command, in Different Contexts
Implicit travel and turn procedures necessary
“Take the blue hall to the sofa”Travelp(until:Path(appear:BlueTile,
side:Front, dist:0))Turnp(until:Path(appear:BlueTile,
side:Front, dist:0),Sofa(side:Front, dist:1-∞)
)Travelp(along:Path(appear:BlueTile,
side:Front, dist:0),until:Sofa(side:At,dist:0)
)
[turna left]
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 21
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
MARCO Route Instruction Understanding OverviewRoute Instruction Following OverviewRelated WorkFollowing One Command, in Different Contexts
Implicit travel and turn procedures necessary
“Take the blue hall to the sofa”Travelp(until:Path(appear:BlueTile,
side:Front, dist:0))Turnp(until:Path(appear:BlueTile,
side:Front, dist:0),Sofa(side:Front, dist:1-∞)
)Travelp(along:Path(appear:BlueTile,
side:Front, dist:0),until:Sofa(side:At,dist:0)
)
travela forward
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 21
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
MARCO Route Instruction Understanding OverviewRoute Instruction Following OverviewRelated WorkFollowing One Command, in Different Contexts
Situated instructions give linguistic and spatial context
Route Instruction Text 1
1. face the chair,
2. move to it,
3. turn left,
4. walk along the bluepast a sofa to thehat rack,
5. turn left,
6. move into the corner,
7. this is 5
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 22
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
MARCO Route Instruction Understanding OverviewRoute Instruction Following OverviewRelated WorkFollowing One Command, in Different Contexts
Situated instructions give linguistic and spatial context
Route Instruction Text 1
1. face the chair,
2. move to it,
3. turn left,
4. walk along the bluepast a sofa to thehat rack,
5. turn left,
6. move into the corner,
7. this is 5
Compound Action Set 1
1. Turnp(until: Chair(side:Front))
2. Travelp(until:Chair(dist:0, side:At))
3. Turnp(dir:Left)
4. Travelp(precond:Path(appear:blue, dist:0, dir:Front),past:Sofa(dist:0),until:Hatrack(dist:0, side:At))
5. Turnp(dir:Left)
6. Travelp(until:Corner(dist:0, side:At))
7. DeclareGoalp(goal:’5’)
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 22
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
MARCO Route Instruction Understanding OverviewRoute Instruction Following OverviewRelated WorkFollowing One Command, in Different Contexts
Infer procedures from linguistic and spatial contextRoute Instruction Text 2
1. take the wood pathtowards the red brickintersection.
2. turn left onto thered brick.
3. at the chair, goright.
4. take a left onto theblue path
5. and then another leftat the coat rack.
6. at the corner, youare at position 5.
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 23
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
MARCO Route Instruction Understanding OverviewRoute Instruction Following OverviewRelated WorkFollowing One Command, in Different Contexts
Infer procedures from linguistic and spatial contextRoute Instruction Text 2
1. take the wood pathtowards the red brickintersection.
2. turn left onto thered brick.
3. at the chair, goright.
4. take a left onto theblue path
5. and then another leftat the coat rack.
6. at the corner, youare at position 5.
Compound Action Set 1
1. Travelp(precond:Path(app:Wood, dist:0, side:F),Intersection(app:Brick, side:F))
2. Turnp(dir:Lefton:Path(app:Brick, dist:0, side:F))
3. Turnp(dir:Right,loc:Chair(dist:0, side:At))
4. Turnp(dir:Left,on:Path(app:Blue, dist:0, side:F))
5. Turnp(dir:Left,loc:Hatrack(dist:0, side:At))
6. DeclareGoalp(goal:’5’,loc:Corner(dist:0, side:At))
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 23
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Route Instruction Following EvaluationCollecting Natural Language Route InstructionsRoute Instruction Corpus StatisticsHuman Subjective Ratings and Success Rates
Introduction to Route Instructions
MARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusRoute Instruction Following EvaluationCollecting Natural Language Route InstructionsRoute Instruction Corpus StatisticsHuman Subjective Ratings and Success Rates
MARCO Performance and Ablation Experiments
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 24
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Route Instruction Following EvaluationCollecting Natural Language Route InstructionsRoute Instruction Corpus StatisticsHuman Subjective Ratings and Success Rates
MARCO Instruction Processing Pipeline
Route Instruction Text
↓
Imperative Procedural Model
↓↑
Movement Actions,Match View Description to
Observations
MARCO Route Follower
1. Parse route instruction text
2. Frame surface semantics
3. Model actions & conditions4. for instruction in model:
a. Interpret in contextb. Infer implicit
proceduresc. Act in the world
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 25
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Route Instruction Following EvaluationCollecting Natural Language Route InstructionsRoute Instruction Corpus StatisticsHuman Subjective Ratings and Success Rates
How to evaluate MARCO performance?
Evaluation : Can the follower reach the destination?
Concerns :Control for route complexity and instruction qualityMeasure when implicit procedures are necessary
Approach : Language and task corpus gives distribution
Baseline : Follow only explicit commands
Standard : People follow same instructions
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 26
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Route Instruction Following EvaluationCollecting Natural Language Route InstructionsRoute Instruction Corpus StatisticsHuman Subjective Ratings and Success Rates
Evaluation: Language and Task Corpus
1. Directorexplores freely
Human
MacMahon &Stankiewicz,CogSci ’06
2. Director typesinstructions
Human
3. Followersread instructions
Human
MARCO
Go three segments
down the grassy hall
to the blue-tiled
hall, passing the
hatrack. Turn left
into the blue-tiled
hall. Follow it ...
4. Followersnavigate route
Human
MARCO variantsfiEnd, Wall, End,
Wall, Empty, Yellow
flfi
Fish, Grass, Fish,Wall, Empty, Cement
flfi
Fish, Grass, Fish,Wall, Hatrack, Brick
fl
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 27
Human View in Virtual Environment
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Route Instruction Following EvaluationCollecting Natural Language Route InstructionsRoute Instruction Corpus StatisticsHuman Subjective Ratings and Success Rates
Visual and Symbolic Views in Environment
Human immersive virtual reality view MARCO symbol view⟨End, Wall, End,
Wall, Empty, Yellow
⟩⟨
Fish, Grass, Fish,Wall, Empty, Cement
⟩
⟨Fish, Grass, Fish,
Wall, Hatrack, Brick
⟩
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 29
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Route Instruction Following EvaluationCollecting Natural Language Route InstructionsRoute Instruction Corpus StatisticsHuman Subjective Ratings and Success Rates
Another View in Virtual Environment
Human immersive virtual reality view MARCO symbol view
⟨End, Wall, End,
Wall, Empty, Wall
⟩
⟨Eiffel, Cement, Eiffel,
Stone, Empty, Wall
⟩
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 30
Routes in Three Environments
I 42 routes between 7 named positions in eachI 11 objects, 3 wall pictures, 8 floor textures
Compact EnvironmentMean path length 2.7, med. 2Intersects / path len: 0.96Unforced choices: 71%
Medium EnvironmentMean path len: 2.9, med. 2.5Intersects / path len: 0.67Unforced choices: 44%
Sparse EnvironmentMean path len: 2, med. 1Intersects / path len: 0.72Unforced choices: 30%
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Route Instruction Following EvaluationCollecting Natural Language Route InstructionsRoute Instruction Corpus StatisticsHuman Subjective Ratings and Success Rates
Human Directors and Followers30 Directors: 15 M, 15 F
Corpus 1 3 M, 3 FCorpus 2 6 M, 6 FCorpus 3 6 M, 6 F
Learned envs by explorationTyped instructions, navigated
101 Followers: 55 M, 46 F
Corpus 1 17 M, 14 FCorpus 2 21 M, 22 FCorpus 3 17 M, 10 F*
Read, followed, rated instructionsCould re-read while navigating
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 32
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Route Instruction Following EvaluationCollecting Natural Language Route InstructionsRoute Instruction Corpus StatisticsHuman Subjective Ratings and Success Rates
Occurrence and success rates by mean rating for people
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 33
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Route Instruction Following EvaluationCollecting Natural Language Route InstructionsRoute Instruction Corpus StatisticsHuman Subjective Ratings and Success Rates
Does human performance match human ratings?
All: Corpora
People : Corpus167.7%
People : Corpus264.0%
People : Corpus358.6%
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 34
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Route Instruction Following EvaluationCollecting Natural Language Route InstructionsRoute Instruction Corpus StatisticsHuman Subjective Ratings and Success Rates
Does human performance match human ratings?
All
People 64.3%
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 35
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Human Performance and Executing Explicit CommandsCues for implicitly commanded proceduresImpact of Spatial Navigation Procedures and LandmarksConclusions
Introduction to Route Instructions
MARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task Corpus
MARCO Performance and Ablation ExperimentsHuman Performance and Executing Explicit CommandsCues for implicitly commanded proceduresImpact of Spatial Navigation Procedures and LandmarksConclusions
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 36
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Human Performance and Executing Explicit CommandsCues for implicitly commanded proceduresImpact of Spatial Navigation Procedures and LandmarksConclusions
How often are explicit instructions enough?
Explicit
People 64.3%
Marco w/oImplicit
Procedures 19.8%
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 37
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Human Performance and Executing Explicit CommandsCues for implicitly commanded proceduresImpact of Spatial Navigation Procedures and LandmarksConclusions
People, MARCO w/o implicit procedures as baselines
Implicit
People 64.3%
Marco 51.5%
Marco w/oImplicit
Procedures 19.8%
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 38
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Human Performance and Executing Explicit CommandsCues for implicitly commanded proceduresImpact of Spatial Navigation Procedures and LandmarksConclusions
Training and Test CorporaHuman Directors Procedure
Corpus 1 6 DirectorsEach map, Discrete motion691 instructionsRatings: 3.3, 4.6, 5.2
Corpus 2 12 DirectorsOne map, Discrete motion432 instructionsRatings: 2.8, 4, 5
Corpus 3 12 DirectorsOne map, Continuous motion399 instructionsRatings: 2.7, 3.5, 4.7
MARCO Follower Development
All Corpora : Provide parse treesw/ hand-correctionsEvaluate Modeler, Executor,View Desc Recognizers
Corpus 1 Development CorpusExamine corpus and buildsystem to follow
Corpus 2 Test + Devel CorpusRun as from Corpus 1Add local fixes and rerun
Corpus 3 Test CorpusRun as from Corpus 1,2
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 39
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Human Performance and Executing Explicit CommandsCues for implicitly commanded proceduresImpact of Spatial Navigation Procedures and LandmarksConclusions
Training and Test CorporaHuman Directors Procedure
Corpus 1 6 DirectorsEach map, Discrete motion691 instructionsRatings: 3.3, 4.6, 5.2
Corpus 2 12 DirectorsOne map, Discrete motion432 instructionsRatings: 2.8, 4, 5
Corpus 3 12 DirectorsOne map, Continuous motion399 instructionsRatings: 2.7, 3.5, 4.7
MARCO Follower Development
All Corpora : Provide parse treesw/ hand-correctionsEvaluate Modeler, Executor,View Desc Recognizers
Corpus 1 Development CorpusExamine corpus and buildsystem to follow
Corpus 2 Test + Devel CorpusRun as from Corpus 1Add local fixes and rerun
Corpus 3 Test CorpusRun as from Corpus 1,2
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 39
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Human Performance and Executing Explicit CommandsCues for implicitly commanded proceduresImpact of Spatial Navigation Procedures and LandmarksConclusions
Training and Test CorporaHuman Directors Procedure
Corpus 1 6 DirectorsEach map, Discrete motion691 instructionsRatings: 3.3, 4.6, 5.2
Corpus 2 12 DirectorsOne map, Discrete motion432 instructionsRatings: 2.8, 4, 5
Corpus 3 12 DirectorsOne map, Continuous motion399 instructionsRatings: 2.7, 3.5, 4.7
MARCO Follower Development
All Corpora : Provide parse treesw/ hand-correctionsEvaluate Modeler, Executor,View Desc Recognizers
Corpus 1 Development CorpusExamine corpus and buildsystem to follow
Corpus 2 Test + Devel CorpusRun as from Corpus 1Add local fixes and rerun
Corpus 3 Test CorpusRun as from Corpus 1,2
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 39
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Human Performance and Executing Explicit CommandsCues for implicitly commanded proceduresImpact of Spatial Navigation Procedures and LandmarksConclusions
Training and Test CorporaHuman Directors Procedure
Corpus 1 6 DirectorsEach map, Discrete motion691 instructionsRatings: 3.3, 4.6, 5.2
Corpus 2 12 DirectorsOne map, Discrete motion432 instructionsRatings: 2.8, 4, 5
Corpus 3 12 DirectorsOne map, Continuous motion399 instructionsRatings: 2.7, 3.5, 4.7
MARCO Follower Development
All Corpora : Provide parse treesw/ hand-correctionsEvaluate Modeler, Executor,View Desc Recognizers
Corpus 1 Development CorpusExamine corpus and buildsystem to follow
Corpus 2 Test + Devel CorpusRun as from Corpus 1Add local fixes and rerun
Corpus 3 Test CorpusRun as from Corpus 1,2
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 39
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Human Performance and Executing Explicit CommandsCues for implicitly commanded proceduresImpact of Spatial Navigation Procedures and LandmarksConclusions
People, MARCO w/o implicit procedures as baselines
Implicit: Corpus1
People 67.7%
Marco 66.0%
Marco w/oImplicit
Procedures 30.2%
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 40
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Human Performance and Executing Explicit CommandsCues for implicitly commanded proceduresImpact of Spatial Navigation Procedures and LandmarksConclusions
Corpus2 enhancements evaluation
Corpus2 Options:Corpus2
People 64.0%
Marco 46.2%
Marco – w/oCorpus2 Options
37.2%
Marco w/oImplicit
Procedures 10.0%
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 41
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Human Performance and Executing Explicit CommandsCues for implicitly commanded proceduresImpact of Spatial Navigation Procedures and LandmarksConclusions
Corpus2 enhancements evaluation
Corpus2 Options:Corpus3
People 58.6%
Marco 32.3%
Marco – w/oCorpus2 Options
29.7%
Marco w/oImplicit
Procedures 12.2%
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 42
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Human Performance and Executing Explicit CommandsCues for implicitly commanded proceduresImpact of Spatial Navigation Procedures and LandmarksConclusions
Development and Evaluation Corpora
Devel Eval
People:Development
67.7%
Marco:Development
66.0%
People: Evaluation61.4%
Marco: Evaluation39.6%
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 43
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Human Performance and Executing Explicit CommandsCues for implicitly commanded proceduresImpact of Spatial Navigation Procedures and LandmarksConclusions
Discrepancy analysis for people and MARCO
Analysis of 3 instructions per director in all 3 Corporaeach correctly followed in at least one MARCO or human run
Corpus 1 Corpus 2 C. 2++ Corpus 3 TotalSame 15 16 19 12 46
People > +25% 2 17 13 16 31MARCO > +25% 1 3 4 8 13
Total 18 36 36 36 90
What needs to be added for MARCO match human performance?Primary reason for failure instruction countmodeling phrase meaning 12
combining descriptions 10modeling word meaning 3spatial / procedural error 4
other 2
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 44
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Human Performance and Executing Explicit CommandsCues for implicitly commanded proceduresImpact of Spatial Navigation Procedures and LandmarksConclusions
Correlations between ratings and performance
Spearman rank correlations between means
Human mean to human ratings :r(19) = 0.977, p ≤ 0.001
MARCO mean to human mean :r(19) = 0.956, p ≤ 0.001
MARCO mean to human ratings :r(19) = 0.968, p ≤ 0.001
MARCO w/o Look-ahead for Travel Termination to human mean :r(19) = 0.979, p ≤ 0.001 (48.2%)
MARCO w/o Fuzzy Meanings to human mean :r(19) = 0.978, p ≤ 0.001 (47.9%)
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 45
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Human Performance and Executing Explicit CommandsCues for implicitly commanded proceduresImpact of Spatial Navigation Procedures and LandmarksConclusions
MARCO Computational Experiments
I How often are different implicit procedure cues required?
I How often are different types of implicit procedures required?
I How often are fundamental navigation skills required?
I How often are different types of landmarks required?
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 46
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Human Performance and Executing Explicit CommandsCues for implicitly commanded proceduresImpact of Spatial Navigation Procedures and LandmarksConclusions
Implicit Procedure Cue Types: Syntactic
Syntactic Closed set ofsyntactic markings,independent of verb
Semantic Open set ofsemantic cues, dependenton verb
Exploratory Action needed tochange knowledge state
Pragmatic Interpretdeclarative as imperative orinsert procedure frominstruction context
Syntactic Cue Definition:Condition explicitly marked by oneof small set of syntactic markers
Syntactic Cue Examples:I At the end of the hall,
turn left.Travelp to end
I With your back to thewall, go forward.Turnp away from wall
I Facing the chair, moveforward one block.Turnp toward chair
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 47
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Human Performance and Executing Explicit CommandsCues for implicitly commanded proceduresImpact of Spatial Navigation Procedures and LandmarksConclusions
Implicit Procedure Cue Types: Semantic
Syntactic Closed set ofsyntactic markings,independent of verb
Semantic Open set ofsemantic cues, dependenton verb
Exploratory Action needed tochange knowledge state
Pragmatic Interpretdeclarative as imperative orinsert procedure frominstruction context
Semantic Cue Definition:Condition modeled in verb frame orimplicit preposition meaning
Semantic Cue Examples:I Go to the coat rack.
Turnp toward coat rackI Walk left past the
chair.Turnp toward chair
I Turn left past thechair.Travelp past chair
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 47
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Human Performance and Executing Explicit CommandsCues for implicitly commanded proceduresImpact of Spatial Navigation Procedures and LandmarksConclusions
Implicit Procedure Cue Types: Exploratory
Syntactic Closed set ofsyntactic markings,independent of verb
Semantic Open set ofsemantic cues, dependenton verb
Exploratory Action needed tochange knowledge state
Pragmatic Interpretdeclarative as imperative orinsert procedure frominstruction context
Exploratory Cue Definition:Exploratory procedures, acting togain information
Exploratory Cue Examples:I Go to the longer end of
the hall.Turnp to view both ends
I Go down the brick path.If not visible, Findp brick path
I Find the intersectionof the red and bluehalls.Findp int w/ red & blue halls
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 47
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Human Performance and Executing Explicit CommandsCues for implicitly commanded proceduresImpact of Spatial Navigation Procedures and LandmarksConclusions
Implicit Procedure Cue Types: Pragmatic
Syntactic Closed set ofsyntactic markings,independent of verb
Semantic Open set ofsemantic cues, dependenton verb
Exploratory Action needed tochange knowledge state
Pragmatic Interpretdeclarative as imperative orinsert procedure frominstruction context
Pragmatic Cue Definition:Interpret based on domain reasoning,not surface semantics
Pragmatic Cue Examples:I It’s at the end of the
hall.Travelp to end
I Turn left & left again.Travelp between turns
I Go forward to thecorner. Then go to thenext corner.Turnp between travels
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 47
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Human Performance and Executing Explicit CommandsCues for implicitly commanded proceduresImpact of Spatial Navigation Procedures and LandmarksConclusions
MARCO w/o cues for navigation procedure inference
Action Cues
People 64.3%
Marco 51.5%
– w/o ImplicitPragmatic 43.4%
– w/o ImplicitExploration 39.9%
– w/o ImplicitSyntactic 32.0%
– w/o ImplicitSemantic 24.9%
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 48
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Human Performance and Executing Explicit CommandsCues for implicitly commanded proceduresImpact of Spatial Navigation Procedures and LandmarksConclusions
How often is each implicit procedure necessary?Each Implicit
People 64.3%
Marco 51.5%
Marco w/oImplicit Travels
32.5%
Marco w/oImplicit Turns
25.3%
Marco w/oImplicit
Procedures 19.8%
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 49
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Human Performance and Executing Explicit CommandsCues for implicitly commanded proceduresImpact of Spatial Navigation Procedures and LandmarksConclusions
Fundamental Large-Scale Space Navigation Skills
Open- and closed-loop procedures, whether explicit or implicit
Turnp Direction turna Left,Right, or Around
Facep Description turna untila View Description is met
Travelp Distance Counttravela for count
Travelp Until travela until aView Description is met
Turnp Direction Example:
I At the end of the hall,turn left.
I Face the easel down thelong, carpeted hall.
I Go to the thirdintersection.
I Take the blue hall tothe red intersection.
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 50
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Human Performance and Executing Explicit CommandsCues for implicitly commanded proceduresImpact of Spatial Navigation Procedures and LandmarksConclusions
Fundamental Large-Scale Space Navigation Skills
Open- and closed-loop procedures, whether explicit or implicit
Turnp Direction turna Left,Right, or Around
Facep Description turna untila View Description is met
Travelp Distance Counttravela for count
Travelp Until travela until aView Description is met
Facep Description Example:
I At the end of the hall,turn left.
I Face the easel down thelong, carpeted hall.
I Go to the thirdintersection.
I Take the blue hall tothe red intersection.
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 50
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Human Performance and Executing Explicit CommandsCues for implicitly commanded proceduresImpact of Spatial Navigation Procedures and LandmarksConclusions
Fundamental Large-Scale Space Navigation Skills
Open- and closed-loop procedures, whether explicit or implicit
Turnp Direction turna Left,Right, or Around
Facep Description turna untila View Description is met
Travelp Distance Counttravela for count
Travelp Until travela until aView Description is met
Travelp Distance Example:
I At the end of the hall,turn left.
I Face the easel down thelong, carpeted hall.
I Go to the thirdintersection.
I Take the blue hall tothe red intersection.
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 50
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Human Performance and Executing Explicit CommandsCues for implicitly commanded proceduresImpact of Spatial Navigation Procedures and LandmarksConclusions
Fundamental Large-Scale Space Navigation Skills
Open- and closed-loop procedures, whether explicit or implicit
Turnp Direction turna Left,Right, or Around
Facep Description turna untila View Description is met
Travelp Distance Counttravela for count
Travelp Until travela until aView Description is met
Travelp Until Example:
I At the end of the hall,turn left.
I Face the easel down thelong, carpeted hall.
I Go to the thirdintersection.
I Take the blue hall tothe red intersection.
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 50
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Human Performance and Executing Explicit CommandsCues for implicitly commanded proceduresImpact of Spatial Navigation Procedures and LandmarksConclusions
MARCO w/o fundamental spatial skills
Skills
People 64.3%
Marco 51.5%
– w/o TurnDirection 43.4%
– w/o DistanceCount 26.7%
– w/o Travel Until21.1%
– w/o FaceDescription 12.4%
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 51
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Human Performance and Executing Explicit CommandsCues for implicitly commanded proceduresImpact of Spatial Navigation Procedures and LandmarksConclusions
MARCO w/o spatial skills or implicitsSkills: Implicits
People 64.3%
Marco 51.5%
– w/o TurnDirection 43.4%
– w/o DistanceCount 26.7%
– w/o Travel Until21.1%
– w/o ImplicitProcedures 19.8%
– w/o FaceDescription 12.4%
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 52
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Human Performance and Executing Explicit CommandsCues for implicitly commanded proceduresImpact of Spatial Navigation Procedures and LandmarksConclusions
Object, Structural, and Appearance Landmarks
Object Landmarks :Discrete point landmarks
I hatrack, coatrack
I Eiffel Towers,butterflies,pictures
I sofa, bench,couch, futon,chair
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 53
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Human Performance and Executing Explicit CommandsCues for implicitly commanded proceduresImpact of Spatial Navigation Procedures and LandmarksConclusions
Object, Structural, and Appearance Landmarks
Structural Landmarks :Extended, affect navigation
Simplewall, path,hallway, way to go
Intersectioncorner,T intersection,dead end
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 53
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Human Performance and Executing Explicit CommandsCues for implicitly commanded proceduresImpact of Spatial Navigation Procedures and LandmarksConclusions
Object, Structural, and Appearance Landmarks
Appearance Landmarks :Color, texture attributes
I rose, pink, carpet
I brick, red, brown
I cement, grey,plain
I wood, brown,orange
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 53
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Human Performance and Executing Explicit CommandsCues for implicitly commanded proceduresImpact of Spatial Navigation Procedures and LandmarksConclusions
Three major types of landmarks
Landmarks: Major
People 64.3%
Marco 51.5%
– w/o ObjectLandmarks 38.7%
– w/o AppearanceLandmarks 33.5%
– w/o StructuralLandmarks 21.5%
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 54
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Human Performance and Executing Explicit CommandsCues for implicitly commanded proceduresImpact of Spatial Navigation Procedures and LandmarksConclusions
Thesis and Contributions
I MARCO architecture can follow natural route instructionsNatural language processingProcedure execution
Linguistic modeling,reasoningSpatial modeling, reasoning
I Measure impact of instruction-following behaviorsI What is necessary to follow natural instructions, and how often?I Evaluate against language and task corpus
I How often are implicit procedures required?
I How often are fundamental navigation skills required?
I How often are different types of landmarks required?
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 55
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Human Performance and Executing Explicit CommandsCues for implicitly commanded proceduresImpact of Spatial Navigation Procedures and LandmarksConclusions
Thesis and Contributions
I MARCO architecture can follow natural route instructions
I Measure impact of instruction-following behaviorsI What is necessary to follow natural instructions, and how often?I Evaluate against language and task corpus
I People describing routes learned by navigationI People following routes from instructionsI MARCO agent following routes from instructions
I How often are implicit procedures required?
I How often are fundamental navigation skills required?
I How often are different types of landmarks required?
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 55
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Human Performance and Executing Explicit CommandsCues for implicitly commanded proceduresImpact of Spatial Navigation Procedures and LandmarksConclusions
Thesis and ContributionsI MARCO architecture can follow natural route instructions
I Measure impact of instruction-following behaviorsI What is necessary to follow natural instructions, and how often?I Evaluate against language and task corpus
I How often are implicit procedures required?I Almost all low-rated instructionsI About half of high-rated instructions
I How often are fundamental navigation skills required?I Inferring implicit procedures is a fundamental following skillI Closed-loop procedures higher impact than open-loop
I How often are different types of landmarks required?I Structural greater impact than Appearance and ObjectI Both Intersection landmarks and Paths and Walls crucial
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 55
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Human Performance and Executing Explicit CommandsCues for implicitly commanded proceduresImpact of Spatial Navigation Procedures and LandmarksConclusions
Thesis and ContributionsI MARCO architecture can follow natural route instructions
I Measure impact of instruction-following behaviorsI What is necessary to follow natural instructions, and how often?I Evaluate against language and task corpus
I How often are implicit procedures required?I Almost all low-rated instructionsI About half of high-rated instructions
I How often are fundamental navigation skills required?I Inferring implicit procedures is a fundamental following skillI Closed-loop procedures higher impact than open-loop
I How often are different types of landmarks required?I Structural greater impact than Appearance and ObjectI Both Intersection landmarks and Paths and Walls crucial
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 55
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Human Performance and Executing Explicit CommandsCues for implicitly commanded proceduresImpact of Spatial Navigation Procedures and LandmarksConclusions
Thesis and ContributionsI MARCO architecture can follow natural route instructions
I Measure impact of instruction-following behaviorsI What is necessary to follow natural instructions, and how often?I Evaluate against language and task corpus
I How often are implicit procedures required?I Almost all low-rated instructionsI About half of high-rated instructions
I How often are fundamental navigation skills required?I Inferring implicit procedures is a fundamental following skillI Closed-loop procedures higher impact than open-loop
I How often are different types of landmarks required?I Structural greater impact than Appearance and ObjectI Both Intersection landmarks and Paths and Walls crucial
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 55
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Human Performance and Executing Explicit CommandsCues for implicitly commanded proceduresImpact of Spatial Navigation Procedures and LandmarksConclusions
Ongoing and Future Work
Work in ProgressI Further analysis of human spatial cognition
Future WorkI Better handling of phrase and discourse structure variationI Learning translation, word-percept mappings, etc.I Integration with hardware robots and wheelchairsI Speech recognition, dialog, interaction, maps and gesturesI Apply findings to instruction generation
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 56
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Human Performance and Executing Explicit CommandsCues for implicitly commanded proceduresImpact of Spatial Navigation Procedures and LandmarksConclusions
Unsolved Issues in Linguistic Variation
Future WorkI Lexical variation: MARCO uses spelling correction, WordNet
Does not handle new descriptions (turtle shell),prepositions (across) or procedural verbs (exit)
I Syntactic variation: Handles seen within-phrase variationNeed better grammar generalization, fallback to shallow parseModeling needs to be more general, e.g. no in verb phrase
I Discourse structure variation: handles linear structure,anaphora, look-ahead for termination conditionsDoes not handle descriptions split into multiple utterancesDoes not handle route summary, then instructions
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 57
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Human Performance and Executing Explicit CommandsCues for implicitly commanded proceduresImpact of Spatial Navigation Procedures and LandmarksConclusions
Unsolved Issues in Linguistic Variation
Future WorkI Lexical variation: MARCO uses spelling correction, WordNet
Does not handle new descriptions (turtle shell),prepositions (across) or procedural verbs (exit)
I Syntactic variation: Handles seen within-phrase variationNeed better grammar generalization, fallback to shallow parseModeling needs to be more general, e.g. no in verb phrase
I Discourse structure variation: handles linear structure,anaphora, look-ahead for termination conditionsDoes not handle descriptions split into multiple utterancesDoes not handle route summary, then instructions
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 57
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Human Performance and Executing Explicit CommandsCues for implicitly commanded proceduresImpact of Spatial Navigation Procedures and LandmarksConclusions
Unsolved Issues in Linguistic Variation
Future WorkI Lexical variation: MARCO uses spelling correction, WordNet
Does not handle new descriptions (turtle shell),prepositions (across) or procedural verbs (exit)
I Syntactic variation: Handles seen within-phrase variationNeed better grammar generalization, fallback to shallow parseModeling needs to be more general, e.g. no in verb phrase
I Discourse structure variation: handles linear structure,anaphora, look-ahead for termination conditionsDoes not handle descriptions split into multiple utterancesDoes not handle route summary, then instructions
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 57
Introduction to Route InstructionsMARCO Architecture for Modeling and Following Instructions
Evaluation through Language and Task CorpusMARCO Performance and Ablation Experiments
Human Performance and Executing Explicit CommandsCues for implicitly commanded proceduresImpact of Spatial Navigation Procedures and LandmarksConclusions
Thank you!
Matt [email protected]
MacMahon, Stankiewicz, & Kuipers, AAAI 2006, Boston, MA.MacMahon & Stankiewicz, CogSci 2006, Vancouver, BC.This work was funded under ONR work order N0001405WX30001 for the NRL Research Option, Coordinated Teams of
Autonomous Systems; and supported by AFOSR grant FA9550- 04-1-0236, FA9550-05-1-0321 and NIH grant EY016089
to B.J. Stankiewicz. Thanks to the Intelligent Robotics Laboratory and the Shape & Space Laboratory at the University of
Texas at Austin and the Naval Research Laboratory’s NCARAI.
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 58
Appendix
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 59
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 60
People and MARCO’s success rates per director: Corpus 1
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 61
People and MARCO’s success rates per director: Corpus 2
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 62
People and MARCO’s success rates per director: Corpus 3
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 63
Corpus2 enhancements evaluation
Corpus2 Options:Corpus2
People 64.0%
Marco 46.2%
Marco – w/oCorpus2 Options
37.2%
Marco w/oImplicit
Procedures 10.0%
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 64
Corpus2 enhancements evaluation
Corpus2 Options:Corpus3
People 58.6%
Marco 32.3%
Marco – w/oCorpus2 Options
29.7%
Marco w/oImplicit
Procedures 12.2%
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 65
Simple and complex stuctural landmarks
Landmarks:Structural
People 64.3%
Marco 51.5%
– w/o IntersectionLandmarks 38.5%
– w/o StructuralLandmarks 21.5%
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 66
MARCO w/o landmark recognition
Landmarks
People 64.3%
Marco 51.5%
– w/o ObjectLandmarks 38.7%
– w/o IntersectionLandmarks 38.5%
– w/o AppearanceLandmarks 33.5%
– w/o StructuralLandmarks 21.5%
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 67
Implicit procedures from linguistic and spatial context
Gaps“Turn right on the black path.
[Travelp to blue path, Facep the sofa on the blue path]
Take the blue hall to the sofa.”Director Mistakes
“Turn right onto the blue path.
(really Left)[Turnp around to face sofa.]
Take the blue hall to the sofa.”Follower Mistakes
“Take a left at the end of the hall.”
wrong hall
Facep blue path, Travelp to blue path, Turnp onto blue path “Takethe blue hall to the sofa.”
Complex actions
[Travelp forward to the 2nd intersection with a path on the left.]
“Take the second left.”Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 68
Implicit procedures from linguistic and spatial context
Gaps“Turn right on the black path.[Travelp to blue path, Facep the sofa on the blue path]Take the blue hall to the sofa.”
Director Mistakes“Turn right onto the blue path.
(really Left)[Turnp around to face sofa.]
Take the blue hall to the sofa.”Follower Mistakes
“Take a left at the end of the hall.”
wrong hall
Facep blue path, Travelp to blue path, Turnp onto blue path “Takethe blue hall to the sofa.”
Complex actions
[Travelp forward to the 2nd intersection with a path on the left.]
“Take the second left.”Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 68
Implicit procedures from linguistic and spatial context
Gaps“Turn right on the black path.[Travelp to blue path, Facep the sofa on the blue path]Take the blue hall to the sofa.”
Director Mistakes“Turn right onto the blue path. (really Left)
[Turnp around to face sofa.]
Take the blue hall to the sofa.”Follower Mistakes
“Take a left at the end of the hall.”
wrong hall
Facep blue path, Travelp to blue path, Turnp onto blue path “Takethe blue hall to the sofa.”
Complex actions
[Travelp forward to the 2nd intersection with a path on the left.]
“Take the second left.”Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 68
Implicit procedures from linguistic and spatial context
Gaps“Turn right on the black path.[Travelp to blue path, Facep the sofa on the blue path]Take the blue hall to the sofa.”
Director Mistakes“Turn right onto the blue path. (really Left)[Turnp around to face sofa.]Take the blue hall to the sofa.”
Follower Mistakes“Take a left at the end of the hall.”
wrong hall
Facep blue path, Travelp to blue path, Turnp onto blue path “Takethe blue hall to the sofa.”
Complex actions
[Travelp forward to the 2nd intersection with a path on the left.]
“Take the second left.”Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 68
Implicit procedures from linguistic and spatial context
Gaps“Turn right on the black path.[Travelp to blue path, Facep the sofa on the blue path]Take the blue hall to the sofa.”
Director Mistakes“Turn right onto the blue path. (really Left)[Turnp around to face sofa.]Take the blue hall to the sofa.”
Follower Mistakes“Take a left at the end of the hall.” wrong hall
Facep blue path, Travelp to blue path, Turnp onto blue path “Takethe blue hall to the sofa.”
Complex actions
[Travelp forward to the 2nd intersection with a path on the left.]
“Take the second left.”Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 68
Implicit procedures from linguistic and spatial context
Gaps“Turn right on the black path.[Travelp to blue path, Facep the sofa on the blue path]Take the blue hall to the sofa.”
Director Mistakes“Turn right onto the blue path. (really Left)[Turnp around to face sofa.]Take the blue hall to the sofa.”
Follower Mistakes“Take a left at the end of the hall.” wrong hall
Facep blue path, Travelp to blue path, Turnp onto blue path “Takethe blue hall to the sofa.”
Complex actions
[Travelp forward to the 2nd intersection with a path on the left.]
“Take the second left.”Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 68
Implicit procedures from linguistic and spatial context
Gaps“Turn right on the black path.[Travelp to blue path, Facep the sofa on the blue path]Take the blue hall to the sofa.”
Director Mistakes“Turn right onto the blue path. (really Left)[Turnp around to face sofa.]Take the blue hall to the sofa.”
Follower Mistakes“Take a left at the end of the hall.” wrong hall
Facep blue path, Travelp to blue path, Turnp onto blue path “Takethe blue hall to the sofa.”
Complex actions
[Travelp forward to the 2nd intersection with a path on the left.]
“Take the second left.”Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 68
Implicit procedures from linguistic and spatial context
Gaps“Turn right on the black path.[Travelp to blue path, Facep the sofa on the blue path]Take the blue hall to the sofa.”
Director Mistakes“Turn right onto the blue path. (really Left)[Turnp around to face sofa.]Take the blue hall to the sofa.”
Follower Mistakes“Take a left at the end of the hall.” wrong hall
Facep blue path, Travelp to blue path, Turnp onto blue path “Takethe blue hall to the sofa.”
Complex actions[Travelp forward to the 2nd intersection with a path on the left.]“Take the second left.”
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 68
Integrating pragmatic information across utterances
Implicit Actions from Instruction ContextImplied Travelp from consecutive Turnsp: Turn left at thechair and then left again.Implied Travelp: Go to the end of the hall and turnleft. There is a chair in front of you. [End ofinstructions]Implied Turnp from consecutive Travelps: Walk down to theblue hall. Go to the [other] end of that hall.
Combining Utterances for Unterminated ActionsGo down the hall. Turn at the firstintersection.Walk past the chair and take the second right.
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 69
MARCO w/o integrating across utterancesImplicit Pragmatic
Cross
People 64.3%
Marco 51.5%
– w/o Nopragmatics withinutterances 47.0%
– w/o Nopragmatics acrossutterances 46.5%
– w/o Nopragmatics 43.4%
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 70
Spatial Semantic Hierarchy for Spatial Reasoning
Layered spatial ontologyI Control level
continuous space, obstacle avoidanceI Causal level
discrete travels, turns, abstract viewsI Topological level
places, paths, regionsI Global Metrical
integrated bird’s eye view map
� -
� -
� -
��������)
@@
@@R
?
? ?
?
ZZ
ZZ
ZZ
ZZ
ZZ~
XXXXXXXXXXXXXXXXXXXXz
XXXXXXXz
Continuous
Sensor values
Quantitative
Local 2-D
Global 2-D
Travel distanceTurn angle
1-D distancesLocal headings
geometry
geometry
Attributes
Qualitative
Control
Causal
Topological
ModelAnalog
Arrows representdependencies
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 71
Spatial Semantic Hierarchy & Multi-Tiered Architectures
SSH Layered control & reasoningI Control level Control Behaviors
continuous space, obstacle avoidanceI Causal level Reactive Sequencer
discrete travels, turns, abstract viewsI Topological level Deliberative Planning
places, paths, regionsI Global Metrical Active SLAM
integrated bird’s eye view map
� -
� -
� -
��������)
@@
@@R
?
? ?
?
ZZ
ZZ
ZZ
ZZ
ZZ~
XXXXXXXXXXXXXXXXXXXXz
XXXXXXXz
Continuous
Sensor values
Quantitative
Local 2-D
Global 2-D
Travel distanceTurn angle
1-D distancesLocal headings
geometry
geometry
Attributes
Qualitative
Control
Causal
Topological
ModelAnalog
Arrows representdependencies
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 71
MARCO across Hybrid Spatial Semantic HierachyHSSH
People 64.3%
Marco 51.5%
– w/o Topological38.3%
– w/o Open LoopCausal 23.7%
– w/o LocalTopological 23.1%
– w/o LocalMetrical 12.1%
– w/o Closed LoopCausal 5.3%
– w/o Causal 0.9%Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 72
Route Instructions: Sequential language describing sequential actions
go to the far end of thebrick hall, then righton the stone hall, lefton the grass hall andit’s in where you meetthe wood hall.
go past green and greyhall, get onto brickhall and go to end takeright and go down bluehallway get onto flowerhallway and go to hatpost
follow the black flooredhall one room to theyellow floored crosshall. this is position2. turn right, move tothe end of the yellowfloored hall.
turn around,go straight 3 times,turn left,go straight
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 73
Explicit Action Route Instructions Text & Model Next
Route Instruction Text 3
1. face the chair,
2. move to it,
3. turn left,
4. walk along the bluepast a sofa to the hatrack,
5. turn left,
6. move into the corner,
7. this is 5
Compound Action Set 1
1. Turnp(until: Chair(side:Front))
2. Travelp(until:Chair(dist:0, side:At))
3. Turnp(dir:Left)
4. Travelp(precond:Path(appear:blue, dist:0,dir:Front),past:Sofa(dist:0),until:Hatrack(dist:0, side:At))
5. Turnp(dir:Left)
6. Travelp(until:Corner(dist:0, side:At))
7. DeclareGoalp(goal:’5’)
Explicit Action Route Instructions Model Next
True map (unknown to Follower)
1. face the chair at 6,Turnp(until: Chair(side:Front))
1. Check for Chair(. . . )
Explicit Action Route Instructions Model & Actions Next
True map (unknown to Follower)
1. face the chair at 6,Turnp(until: Chair(side:Front))
2. Turn Left
Explicit Action Route Instructions Model & Actions Next
True map (unknown to Follower)
1. face the chair at 6,Turnp(until: Chair(side:Front))
3. Utterance 1 Done
Explicit Action Route Instructions Model & Actions Next
True map (unknown to Follower)
2. move to it,Travelp(until: Chair(dist:0, side:At))
4. Travel Forward
Explicit Action Route Instructions Model & Actions Next
True map (unknown to Follower)
2. move to it,Travelp(until: Chair(dist:0, side:At))
5. Travel Forward
Explicit Action Route Instructions Model & Actions Next
True map (unknown to Follower)
2. move to it,Travelp(until: Chair(dist:0, side:At))
6. Utterance 2 Done
Explicit Action Route Instructions Model & Actions Next
True map (unknown to Follower)
3. turn left,Turnp(dir:Left)
7. Turn Left
Explicit Action Route Instructions Model & Actions Next
True map (unknown to Follower)
3. turn left,Turnp(dir:Left)
8. Utterance 3 Done
Explicit Action Route Instructions Model & Actions Next
True map (unknown to Follower)
4. walk along the blue pasta sofa to the hat rack,Travelp(pre:Path(appear:blue,Side:Front),past:Sofa(dist:0),until:Hatrack(dist:0, side:At))
9. Travel Forward
Explicit Action Route Instructions Model & Actions Next
True map (unknown to Follower)
4. walk along the blue pasta sofa to the hat rack,Travelp(pre:Path(appear:blue,Side:Front),past:Sofa(dist:0),until:Hatrack(dist:0, side:At))
10. Travel Forward
Explicit Action Route Instructions Model & Actions Next
True map (unknown to Follower)
4. walk along the blue pasta sofa to the hat rack,Travelp(pre:Path(appear:blue,Side:Front),past:Sofa(dist:0),until:Hatrack(dist:0, side:At))
11. Travel Forward
Explicit Action Route Instructions Model & Actions Next
True map (unknown to Follower)
4. walk along the blue pasta sofa to the hat rack,Travelp(pre:Path(appear:blue,Side:Front),past:Sofa(dist:0),until:Hatrack(dist:0, side:At))
12. Travel Forward
Explicit Action Route Instructions Model & Actions Next
True map (unknown to Follower)
4. walk along the blue pasta sofa to the hat rack,Travelp(pre:Path(appear:blue,Side:Front),past:Sofa(dist:0),until:Hatrack(dist:0, side:At))
13. Utterance 4 Done
Explicit Action Route Instructions Model & Actions Next
True map (unknown to Follower)
5. turn left,Turnp(dir:Left)
14. Turn Left
Explicit Action Route Instructions Model & Actions Next
True map (unknown to Follower)
5. turn left,Turnp(dir:Left)
15. Utterance 5 Done
Explicit Action Route Instructions Model & Actions Next
True map (unknown to Follower)
6. move into the corner,Travelp(until:Corner(dist:0, side:At))
16. Travel Forward
Explicit Action Route Instructions Model & Actions Next
True map (unknown to Follower)
6. move into the corner,Travelp(until:Corner(dist:0, side:At))
17. Utterance 6 Done
Explicit Action Route Instructions Model & Actions Next
True map (unknown to Follower)
7. this is 5DeclareGoalp(goal:’5’)
18. Declare Goal Utterance 7 Done
Implicit Turns Route Instructions Text & Model Next
Route Instruction Text 3
1. look for the blue hall,
2. go all the way downuntil you hit a spotwith a coat rack,
3. then one movement left.
Compound Action Set 1
1. Findp(until: Path(appear:blue,dist:0,dir:At))
2. Travelp(dir:Forward,until:Hatrack(dist:0,side:At))
3. Travelp(dir:Left, dist:(count:1))
Implicit Turns Route Instructions Model Next
True map (unknown to Follower)
1. Check for Path(appear:Blue,. . . )
1. Findp(until: Path(appear:blue,dist:0,dir:At))
Implicit Turns Route Instructions Model & Actions Next
True map (unknown to Follower)
2. Turn Left
1. Findp(until: Path(appear:blue,dist:0,dir:At))
Implicit Turns Route Instructions Model & Actions Next
True map (unknown to Follower)
3. Travel Forward
1. Findp(until: Path(appear:blue,dist:0,dir:At))
Implicit Turns Route Instructions Model & Actions Next
True map (unknown to Follower)
4. Travel Forward
1. Findp(until: Path(appear:blue,dist:0,dir:At))
Implicit Turns Route Instructions Model & Actions Next
True map (unknown to Follower)
5. Utterance 1 Done
1. Findp(until: Path(appear:blue,dist:0,dir:At))
Implicit Turns Route Instructions Model & Actions Next
True map (unknown to Follower)
6. Turn Right
2. Travelp(dir:Forward,until:Hatrack(dist:0,side:At))
Implicit Turns Route Instructions Model & Actions Next
True map (unknown to Follower)
7. Turn Right
2. Travelp(dir:Forward,until:Hatrack(dist:0,side:At))
Implicit Turns Route Instructions Model & Actions Next
True map (unknown to Follower)
8. Turn Right
2. Travelp(dir:Forward,until:Hatrack(dist:0,side:At))
Implicit Turns Route Instructions Model & Actions Next
True map (unknown to Follower)
9. Travel Forward
2. Travelp(dir:Forward,until:Hatrack(dist:0,side:At))
Implicit Turns Route Instructions Model & Actions Next
True map (unknown to Follower)
10. Travel Forward
2. Travelp(dir:Forward,until:Hatrack(dist:0,side:At))
Implicit Turns Route Instructions Model & Actions Next
True map (unknown to Follower)
11. Travel Forward
2. Travelp(dir:Forward,until:Hatrack(dist:0,side:At))
Implicit Turns Route Instructions Model & Actions Next
True map (unknown to Follower)
12. Travel Forward
2. Travelp(dir:Forward,until:Hatrack(dist:0,side:At))
Implicit Turns Route Instructions Model & Actions Next
True map (unknown to Follower)
13. Travel Forward
2. Travelp(dir:Forward,until:Hatrack(dist:0,side:At))
Implicit Turns Route Instructions Model & Actions Next
True map (unknown to Follower)
14. Utterance 2 Done
2. Travelp(dir:Forward,until:Hatrack(dist:0,side:At))
Implicit Turns Route Instructions Model & Actions Next
True map (unknown to Follower)
15. Turn Left
3. Travelp(dir:Left, dist:(count:1))
Implicit Turns Route Instructions Model & Actions Next
True map (unknown to Follower)
16. Travel Forward
3. Travelp(dir:Left, dist:(count:1))
Implicit Turns Route Instructions Model & Actions Next
True map (unknown to Follower)
17. Utterance 3 Done
3. Travelp(dir:Left, dist:(count:1))
Implicit Turns Route Instructions Model & Actions Next
True map (unknown to Follower)
18. Declare Goal
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 76
Route Instruction Language Modeling
1. Referring Phrase Analysis : noun and verb phrases
2. Hand tag phrase senses in some route instruction texts3. Bootstrap learning of the language model
3.1 Train Probabilistic Context Free Grammar (PCFG) with taggedtrees
3.2 Parse additional texts using trained model3.3 Correct automatically parsed trees3.4 Repeat with expanded training and parsing sets
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 77
Route Instruction Language Modeling
1. Referring Phrase Analysis : noun and verb phrases
2. Hand tag phrase senses in some route instruction texts3. Bootstrap learning of the language model
3.1 Train Probabilistic Context Free Grammar (PCFG) with taggedtrees
3.2 Parse additional texts using trained model3.3 Correct automatically parsed trees3.4 Repeat with expanded training and parsing sets
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 77
Route Instruction Language Grammar
Lexicon for route instructionsI Open set of
I action verbs,I objects,I descriptions
I Closed set ofI action types,I prepositions,I movement adverbs
1. Referring PhrasesI AdjectivesI Prepositions
2. Verbs: Action and Other3. Utterance Types
3.1 Travel (examples)3.2 Turn (examples)3.3 Descriptions
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 78
What to do with the parse trees?
I Parse trees show the syntactic structure of languageI How to extract the meaning?I Remove connector words, irrelevant phrase order, synonym
choiceI Content frame represents the surface semantics of an utterance
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 79
Executing Route Instructions in Situated Context
Compound Actions :Model which low-level actionunder which context
Low-level Actions :Turn, Travel, Declare-Goal,Verify(ViewDesc)
Context while executing :Match view to viewdescriptionMaintain state over route legDistance and Pre-, post-, while
Compound Action Set 1
1. turnp(until:Path(appear:yellow,dist:0:,dir:Front))
2. travelp(dir:forward,distance:(count:3,unit:Structure(type:intersection)))
3. turnp(dir:left)
4. travelp( until:Structure( Type:End,dist:0,dir:At))
5. declare-goalp()
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 80
Executing Route Instructions in Situated Context
Compound Actions :Model which low-level actionunder which context
Low-level Actions :Turn, Travel, Declare-Goal,Verify(ViewDesc)
Context while executing :Match view to viewdescriptionMaintain state over route legDistance and Pre-, post-, while
Compound Action Set 1
1. turnp(until:Path(appear:yellow,dist:0:,dir:Front))
2. travelp(dir:forward,distance:(count:3,unit:Structure(type:intersection)))
3. turnp(dir:left)
4. travelp( until:Structure( Type:End,dist:0,dir:At))
5. declare-goalp()
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 80
Executing Route Instructions in Situated Context
Compound Actions :Model which low-level actionunder which context
Low-level Actions :Turn, Travel, Declare-Goal,Verify(ViewDesc)
Context while executing :Match view to viewdescriptionMaintain state over route legDistance and Pre-, post-, while
Compound Action Set 1
1. turnp(until:Path(appear:yellow,dist:0:,dir:Front))
2. travelp(dir:forward,distance:(count:3,unit:Structure(type:intersection)))
3. turnp(dir:left)
4. travelp( until:Structure( Type:End,dist:0,dir:At))
5. declare-goalp()
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 80
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 81
Route Instruction Language Grammar
Lexicon for route instructionsI Open set of
I action verbs,I objects,I descriptions
I Closed set ofI action types,I prepositions,I movement adverbs
1. Referring PhrasesI AdjectivesI Prepositions
2. Verbs: Action and Other3. Utterance Types
3.1 Travel (examples)3.2 Turn (examples)3.3 Descriptions
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 82
Referring Phrases
OBJ Object: chair, painting easel, a lamp,pictures on the walls
PATH One dimensional path: A blue corridor, longalley, the path
STRUCT Structure of the environment, intersections: thecorner, the end, the shorter portion
POSITION Names of targets: Position 3, seven, p. 6LOC Zero dimensional place: at the chair, at
four, two intersections after the lampVIEW Sensory snapshot from a pose: you see a red
brick floor to your right, you shouldsee blue carpet
REGION 2 dimensional area of the environment: the fisharea, the pink-flowered area
REF Reference: it, that, thereMatt MacMahon University of Texas at Austin Following Natural Language Route Instructions 83
Route Instruction Utterance Types
TURN Change in orientation or switch in paths: turn toface the green hallway, turn right,take a left onto the blue path
TRAVEL Traversal along a path: Follow the grassyhall three segments to the blue-tiledpath, go all the way down the red brickcorridor, walk forward one
FIND Undirected search for object: find the easelin the fish area with the black floorsnear it, go to the flowered flooredarea
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 84
Route Instruction Action Verbs
V/TURN Change in orientation or switch in paths: turn,take (a left), make (a turn), go (left)
V/ORIENT Change in rotation relative to an allocentric frame ofreference. face, put (your back againstthe wall), stand (so that ...), orient
V/TRAVEL Move, changing location: walk, go, move,follow, take (the path), head
V/STOP Cease motion: stop
V/PASS Move past an entity: pass
V/ARRIVE Move to an entity: get to, meet, come to,hit, enter
V/FIND Undirected search: find, look for, go (tothe place where ...)
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 85
Route Instruction Turn Arguments
TURN
I DIR : RIGHT | LEFT
I ORIENT : TOWARD | AWAY | SIDE OBJ | VIEW | STRUCT |DESC
I ONTO : PATH
I LOC : LOCATION
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 86
Route Instruction Turn Arguments
DIR Turn direction relative to agent: left, right
ORIENT Turn so that this condition holds: place yourback to the wall, (turn) so that thewall is on your left, face the hatrack, turn to locate the easel
TOWARD Facing the referentAWAY Facing away from the referentSIDE Facing so that the referent is on the
named side: to your left
ONTO Turn onto this path: into the yellow tiledhall, onto the red brick
LOC Where to turn: left at the coat rack,take a right onto the plain path twointersections after the lamp
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 87
Route Instruction Adjectives
APPEAR Appearance, floor color and texture, pictures:blue, green octagon, fish on the wall,wooden floored
STRUCTURAL Length and shape of path: long, short,winding
STRUCT-TYPE Local topology and geometry of environmentstructure: four-way, ‘T’
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 88
Route Instruction Noun Prepositional Phrases
ON : P/ON PATH Entity is on this path: in thishall, on the long path
BETWEEN : P/BETWEEN OBJ OBJ Entity is between two otherentities: between the chair and thehatrack, between the two intersectionscontaining furniture
SIDE : P/SIDE DIR OBJ? Location of the objects relativeto one other entity’s inherent side in front ofthe easel, to your left, behind you, onone side
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 89
Route Instruction Other Verbs
V/IS Be verb: is, match,are
V/HAS Possession verb: contain,has
V/SEE Local observation verb: see,look
V/LOC Location verb: standing,located
V/STRUCT Verbs encoding environment structure by movementmetaphor: intersect, meet, brings, hits,runs into
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 90
Route Instruction Travel Arguments
TRAVEL
I ALONG : PATH
I DIST : QUAL-DIST | QUANT-DIST
I PAST : OBS
I UNTIL : OBS
I TOWARD : OBS
I AWAY : OBS
I DIR : RIGHT | LEFT | FORWARD | BACK
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 91
Route Instruction Travel Arguments
ALONG Travel on this path: along the grasscarpet, down the hall with the yellowstone floors, take it
QUAL-DIST Qualitative description of distance: all of theway down, far, very end
QUANT-DIST COUNT (DIST-UNIT)? Quantitative description ofdistance: one movement, two alleys awayfrom, about five paths down, first left
PAST Move past entity on this path, in the direction of anobject but further than it: past the stool,passing the lamp, beyond that
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 92
Route Instruction Description ArgumentsDESC
I LOC* (there | where | COND )* OBJ | VIEW | STRUCT | LOC |POSITION Is-V (Appear | Side | Loc)
I (there | where) IS-V OBJ | VIEW | STRUCT | LOC
I PATH STRUCT-V PATH
I (PATH | STRUCT | LOC | REF) HAS-V (PATH | STRUCT | LOC
| REF)I LOC* ARRIVE
COND Conditional for (ARRIVE | VIEW | ORIENT | LOC |DESC) when ..., once ..., if ...
ARRIVE Description of termination of motion you get tothe ..., you come to an intersectionwith ...
Matt MacMahon University of Texas at Austin Following Natural Language Route Instructions 93
EDA: turn to face the green halllway, walkthree times forward, turn left, walk forwardsix times, turn left, walk forward once
EMWC
KLS KXP
TJS WLH
TURN
����
HHHH
TURNV
’turn’
ORIENT
��� HHHORIENTV
’to’’face’
PATH
���
HHH
DET
’the’
APPEAR
’green’
PATHN
’halllway’
PUNCT
’,’
TRAVEL
�����
��
@@
PPPPP
TRAVELV
’walk’
DIST
COUNT
’three’
UNIT
’times’
DIR
’forward’
PUNCT
’,’
TURN
�� HHTURNV
’turn’
DIR
’left’
PUNCT
’,’
TRAVEL
�������
@@
PPPPP
TRAVELV
’walk’
DIR
’forward’
DIST
COUNT
’six’
UNIT
’times’
PUNCT
’,’
TURN
�� HHTURNV
’turn’
DIR
’left’
PUNCT
’,’
TRAVEL
���
HHH
TRAVELV
’walk’
DIR
’forward’
DIST
COUNT
’once’
EMWC: Follow the grassy hall three segmentsto the blue-tiled hall. Turn left. Followthe blue-tiled hall six segments, passingthe chair, stool and bench, to theintersection containing the hatrack. Turnleft. Go one segment forward to the corner.This is Position 5.
EDA KLS
KXP TJS
WLH
TRAVEL
�����������
������
HHHHHH
XXXXXXXXXXX
TRAVELV
’follow’
ALONG
PATH
��� HHHDET
’the’
APPEAR
’grassy’
PATHN
’hall’
DIST��HH
COUNT
’three’
UNIT
’segments’
UNTIL
�� HHUNTILP
’to’
PATH
��� HHHDET
’the’
APPEAR
’blue’’-’’tiled’
PATHN
’hall’
PUNCT
’.’
TURN
�� HHTURNV
’turn’
DIR
’left’
PUNCT
’.’
TRAVEL
���������������
���
���
��
HHH
HHH
HH
XXXXXXXXXXXXXXX
TRAVELV
’follow’
ALONG
��� HHHPATH
��� HHHDET
’the’
APPEAR
’blue’’-’’tiled
PATHN
’hall’
DIST��HH
COUNT
’six’
UNIT
’segments’
PAST
�� HHPASSV
’passing’
OBJ
��� PPP’the’ ’chair’ ’stool’’and’ ’bench’
UNTIL
�� HHUNTILP
’to’
STRUCT
������@@ PPPP
DET
’the’
STRUCTN
’inter-section’
HASV
’con-taining’
OBJ
DET
’the’
OBJN
’hatrack’
PUNCT
’.’
TURN
�� HHTURNV
’turn’
DIR
’left’
PUNCT
’.’
TRAVEL
�������
����
HHHHXXXXXXX
TRAVELV
’go’
DIST��HH
COUNT
’one’
UNIT
’segment’
DIR
’forward’
UNTIL
�� HHUNTILP
’to’
STRUCT
DET
’the’
STRUCTN
’corner’
PUNCT
’.’
NAME
�������
@@
PPPPP
REF
’this’
ISV
’is’
POSITION��HH
POSN
’position’
PN AME
’5’
PUNCT
’.’
KLS: take the green path to the red brickintersection. go left towards the lamp tothe very end of the hall. at the chair,take a right. at the blue pathintersection, take a left onto the bluepath. at the coat rack, take another leftonto the plain cement. at the end of thishall at the corner, you are at position 5
EDA
EMWC
KXP TJS
WLH
TRAVEL
����������
��
��
@@
@@
PPPPPPPPPP
TRAVELV
’take’
ALONG
PATH
��� HHHDET
’the’
APPEAR
’green’
PATHN
’path’
UNTIL��HH
UNTILP
’to’
STRUCT
���
HHH
PATH
���
HHH
DET
’the’
APPEAR
’red’
APPEAR
’brick’
STRUCTN
’intersection’
PUNCT
’.’
TRAVEL
����������
���
��
HHH
HH
XXXXXXXXXX
TRAVELV
’go’
DIR
’left’
TOWARD
�� HHTOWARDP
’towards’
OBJ
DET
’the’
OBJN
’lamp’
UNTIL
�� HHUNTILP
’to’
STRUCT
����HHHH
STRUCTN�� PP’the very end’
STRUCTP
’of’
PATH
DET
’the’
PATHN
’hall’
PUNCT
’.’
TURN
��������
����
HHHH
XXXXXXXX
LOC��HH
LOCP
’at’
OBJ
DET
’the’
OBJN
’chair’
PUNCT
’,’
TURNV
’take’
DIR
DET
’a’
’right’
PUNCT
’.’
TURN
��������������
����
���
HHHHHHH
XXXXXXXXXXXXXX
LOC
�� HHLOCP
’at’
STRUCT
��� HHHPATH
��� HHHDET
’the’
APPEAR
’blue’
PATHN
’path’
STRUCTN
’intersection’
TURNV
’take’
DIR
DET
’a’
’left’
ONTO
�� HHONTOP
’onto’
PATH
���
HHH
DET
’the’
APPEAR
’blue’
PATHN
’path’
PUNCT
’.’
TURN
���������
���
��
HHHHH
XXXXXXXXX
LOC
LOCP
’at’
OBJ
DET
’the’
OBJN
’coat’’rack’
TURNV
’take’
DIR
DET
’another’
’left’
ONTO
�� HHONTOP
’onto’
PATH
��� HHHDET
’the’
APPEAR
’plain’
APPEAR
’cement’
PUNCT
’.’
DESC
�����������
��
��
@@
@@
PPPPPPPPPPP
LOC
�� HHLOCP
’at’
STRUCT
������� @@ PPPPP
DET
’the’
STRUCTN
’end’
STRUCTP
’of’
PATH
DET
’this’
PATHN
’hall’
LOC��HH
LOCP
’at’
STRUCT
DET
’the’
STRUCTN
’corner’
PUNCT
’,’
NAME
���HHH
’you’ ISV
’are’
LOC
�� HHLOCP
’at’
POSITION��HH
POSN
’position’
P-NAME
’5’
KXP: head all the way toward the butterflyhallway, keep going down it until you reacha dead end square area. pos 5 is in thecorner to the left as you enter the squareblock.
EDA
EMWC
KLS TJS
WLH
TRAVEL
�����������
��
��
@@
@@
PPPPPPPPPPP
TRAVELV
’head’
DIST
�� HH’all’ ’the’’way’
TOWARD
���HHH
TOWARDP
’toward’
PATH
��� HHHDET
’the’
APPEAR
’butterfly’
PATHN
’hallway’
PUNCT
’,’
TRAVEL
��������
���
@@@
PPPPPPPP
TRAVELV
AUXV
’keep’
’going’
ALONG
ALONGP
’down’
PATH
REF
’it’
UNTIL
�� HHUNTILP
’until’
ARRIVE
����
HHHH
’you’ ARRIVEV
’reach’
LOC
������ @@ PPPP
DET
’a’
STRUCTN
’dead’’end’
APPEAR
’square’
LOCN
’area’
PUNCT
’.’
NAME
�������
@@
PPPPP
POSITION
POSN
’pos’
P-NAME
’5’
ISV
’is’
LOC
�� HHLOCP
’in’
STRUCT
�����
HHHHH
DET
’the’
STRUCTN
’corner’
SIDE
������
��
@@
PPPPPP
SIDEP
’to’
DET
’the’
DIR
’left’
COND
�� HHCONDP
’as’
ARRIVE
����
HHHH
’you’ ARRIVEV
’enter’
LOC
��� HHHDET
’the’
APPEAR
’square’
LOCN
’block’
PUNCT
’.’
TJS: go all the way down the grassy hall,take a left, go all the way down the bluehall until you see a coat rack, take anotherimmediate left.
EDA
EMWC
KLS KXP
WLHTRAVEL
��������
���
@@@
PPPPPPPP
TRAVELV
’go’
DIST
�� HH’all’ ’the’’way’
ALONG
ALONGP
’down’
PATH
��� HHHDET
’the’
APPEAR
’grassy’
PATHN
’hall’
PUNCT
’,’
TURN
��� HHHTURNV
’take’
DIR
DET
’a’
’left’
PUNCT
’,’
TRAVEL
�������������
��
�����
HH
HHHHH
XXXXXXXXXXXXX
TRAVELV
’go’
DIST
�� HH’all’ ’the’’way’
ALONG
�� HHALONGP
’down’
PATH
���
HHH
DET
’the’
APPEAR
’blue’
PATHN
’hall’
UNTIL
�� HHUNTILP
’until’
VIEW
��� HHH’you’ SEEV
’see’
OBJ
DET
’a’
OBJN
’coat’’rack’
PUNCT
’,’
TURN
���
��
HHH
HH
TURNV
’take’
DIR
���HHH
DET
’another’
DIST
RELDIST
’immediate’
’left’
PUNCT
’.’
WLH: from four face the grass carpet andmove to the hat rack, turn left and moveonto the blue carpet, walk past two chairsand to the lamp, turn left, move into thecorner such that the lamp is behind you andto your right you see a gray carpeted alley
EDA
EMWC
KLS KXP
TJS
TURN
���HHH
LOC��HH
LOCP
’from’
POSITION
P-NAME
’four’
ORIENT
�� HHORIENTV
’face’
PATH
��� HHHDET
’the’
APPEAR
’grass’
PATHN
’carpet’
THEN
’and’
TRAVEL
��� HHHTRAVELV
’move’
UNTIL
�� HHUNTILP
’to’
OBJ
DET
’the’
OBJN
’hat’’rack’
PUNCT
’,’
TURN
TURNV
’turn’
DIR
’left’
THEN
’and’
TRAVEL
��� HHHTRAVELV
’move’
ALONG
��� HHHALONGP
’onto’
PATH
��� HHHDET
’the’
APPEAR
’blue’
PATHN
’carpet’
PUNCT
’,’
TRAVEL
��������
����
HHHH
XXXXXXXX
TRAVELV
’walk’
PAST��HH
PASTP
’past’
OBJ
DET
’two’
OBJN
’chairs’
’and’ UNTIL�� HH
UNTILP
’to’
OBJ
DET
’the’
OBJN
’lamp’
PUNCT
’,’
TURN
�� HHTURNV
’turn’
DIR
’left’
PUNCT
’,’
TRAVEL
��������
���
@@
@
PPPPPPPP
TRAVELV
’move’
UNTIL
�� HHUNTILP
’into’
STRUCT
DET
’the’
STRUCTN
’corner’
�� PP’such’ ’that’
DESC
�� HHOBJ
DET
’the’
OBJN
’lamp’
ISV
’is’
SIDE
SIDEP
’behind’
OBJ
’you’DESC
��
���
HHHHH
SIDE�� HH
SIDEP
’to’
DIR
DET
’your’
’right’
VIEW
���
��
HHH
HH
’you’ SEEV
’see’
PATH
������ @@ PPPP
DET
’a’
APPEAR
’gray’
APPEAR
’carpeted’
PATHN
’alley’
]
EDA: turn to face the green halllway, walkthree times forward, turn left, walk forwardsix times, turn left, walk forward once
KLS
EMWC
KXP TJS
WLH
2666664TURN
266664ORIENT
266664ORIENTV Face
PATH
264DEF +APPEAR GreenPATHN Hall
375377775
3777753777775
2664TRAVEL
2664DIR Forward
DIST
"COUNT 3UNIT Move
#37753775
»TURN
hDIR Left
i–
2664TRAVEL
2664DIR Forward
DIST
"COUNT 6UNIT Move
#37753775
»TURN
hDIR Left
i– 2664TRAVEL
2664DIR Forward
DIST
"COUNT 1UNIT Move
#37753775
EMWC: Follow the grassy hall three segmentsto the blue-tiled hall. Turn left. Followthe blue-tiled hall six segments, passingthe chair, stool and bench, to theintersection containing the hatrack. Turnleft. Go one segment forward to the corner.This is Position 5.
KLS EDA
KXP TJS
WLH
2666666666666666666664
TRAVEL
2666666666666666666664
ALONG
264PATH
264DEF +APPEAR GrassPATHN Hall
375375
DIST
"COUNT 3UNIT Move
#
UNTIL
26666664UNTILP To
PATH
26664DEF +APPEAR BlueAPPEAR TiledPATHN Hall
37775
37777775
3777777777777777777775
3777777777777777777775
»TURN
hDIR Left
i–
2666666666666666666666666666666666666664
TRAVEL
266666666666666666666666666666666666664
ALONG
26664PATH
26664DEF +APPEAR BlueAPPEAR TiledPATHN Hall
3777537775
DIST
"COUNT 6UNIT Move
#
PAST
266666664PASSV Pass
26666664OBJ
26666664OBJ
"DEF +OBJN Chair
#OBJ
hOBJN Stool
iOBJ
hOBJN Bench
i
37777775
37777775
377777775
UNTIL
266666664
UNTILP To
STRUCT
266664DEF +STRUCTN Intersection
OBJ
"DEF +OBJN HatRack
#377775
377777775
377777777777777777777777777777777777775
3777777777777777777777777777777777777775
»TURN
hDIR Left
i–
266666666664TRAVEL
266666666664
DIST
"COUNT 1UNIT Move
#DIR Forward
UNTIL
2664UNTILP To
STRUCT
"DEF +STRUCTN Corner
#3775
377777777775
377777777775
24NAME
"REF thisPOSITION 5
#35
KLS: take the green path to the red brickintersection. go left towards the lamp tothe very end of the hall. at the chair,take a right. at the blue pathintersection, take a left onto the bluepath. at the coat rack, take another leftonto the plain cement. at the end of thishall at the corner, you are at position 5
EDA
EMWC
KXP TJS
WLH
266666666666666664TRAVEL
26666666666666664
ALONG
264PATH
264DEF +APPEAR GreenPATHN Hall
375375
UNTIL
26666664
UNTILP To
STRUCT
266664PATH
264DEF +APPEAR RedAPPEAR Brick
375STRUCTN Intersection
377775
37777775
37777777777777775
377777777777777775266666666666666666664
TRAVEL
26666666666666666664
DIR Left
TOWARD
2664TOWARDP Toward
OBJ
"DEF +OBJN Lamp
#3775
UNTIL
266666664
UNTILP To
STRUCT
266664STRUCTN EndSTRUCTP Of
PATH
"DEF +PATHN Hall
#377775
377777775
37777777777777777775
377777777777777777775
2666664TURN
2666664LOC
2664LOCP At
OBJ
"DEF +OBJN Chair
#3775DIR Right
3777775
3777775
2666666666666666666664
TURN
266666666666666666664
LOC
26666664LOCP At
STRUCT PATH
264DEF +APPEAR BluePATHN Hall
375STRUCTN Intersection
37777775DIR Left
ONTO
266664ONTOP Onto
PATH
264DEF +APPEAR BluePATHN Hall
375377775
377777777777777777775
3777777777777777777775
266666666666666664TURN
26666666666666664
LOC
2664LOCP At
OBJ
"DEF +OBJN CoatRack
#3775DIR Left
ONTO
266664ONTOP Onto
PATH
264DEF +APPEAR PlainAPPEAR Cement
375377775
37777777777777775
377777777777777775
266666666666666666666664
NAME
26666666666666666664
LOC
26666666664
LOCP At
STRUCT
26666664DEF +STRUCTN EndSTRUCTP Of
PATH
"DEF +PATHN Hall
#37777775
377777777752664
LOCP At
STRUCT
"DEF +STRUCTN Corner
#3775
37777777777777777775h
POSITION 5i
377777777777777777777775
KXP: head all the way toward the butterflyhallway, keep going down it until you reacha dead end square area. pos 5 is in thecorner to the left as you enter the squareblock.
KLS EDA
EMWC
TJS WLH
266666664TRAVEL
26666664
DIST ToEnd
TOWARD
266664TOWARDP Toward
PATH
264DEF +APPEAR ButterflyPATHN Hall
375377775
37777775
377777775266666666666666666664
TRAVEL
26666666666666666664
ALONG
24ALONGP Down
PATHh
REF Iti35
UNTIL
2666666666664
UNTILP Until
ARRIVE
2666666664
AGENT YouARRIVEV Reach
LOC
26664DEF -STRUCTN DeadEndAPPEAR SquareLOCN Area
37775
3777777775
3777777777775
37777777777777777775
37777777777777777777526666666666666666666666666664
NAME
2666666666666666666666664
LOC
266666666666666666666664
LOCP In
STRUCT
266666666666666666664
DEF +STRUCTN Corner
SIDE
2666666666666664
SIDEP ToDIR Left
COND
26666666664
CONDP As
ARRIVE
26666664AGENT YouARRIVEV Enter
LOC
264DEF +APPEAR SquareLOCN Block
375
37777775
37777777775
3777777777777775
377777777777777777775
377777777777777777777775
3777777777777777777777775h
POSITION 5i
37777777777777777777777777775
TJS: go all the way down the grassy hall,take a left, go all the way down the bluehall until you see a coat rack, take anotherimmediate left.
KLS EDA
EMWC
KXP WLH
2666664TRAVEL
266664DIST ToEnd
ALONG
264PATH
264DEF +APPEAR GrassPATHN Hall
375375
3777753777775
»TURN
hDIR Left
i–
2666666666666666664
TRAVEL
266666666666666664
DIST ToEnd
ALONG
266664ALONGP Down
PATH
264DEF +APPEAR BluePATHN Hall
375377775
UNTIL
26664UNTILP Until
VIEW
24OBJ
"DEF -OBJN HatRack
#3537775
377777777777777775
3777777777777777775
264TURN
264LOC»
DISTh
RELDIST Immediatei–
DIR Left
375375
WLH: from four face the grass carpet andmove to the hat rack, turn left and moveonto the blue carpet, walk past two chairsand to the lamp, turn left, move into thecorner such that the lamp is behind you andto your right you see a gray carpeted alley
KLS EDA
EMWC
KXP TJS
2666664TURN
2666664LOC
24LOCP From
POSITIONh
P-NAME 4i35
ORIENT»
PATHh
APPEAR Grassi–
3777775
3777775
26664TRAVEL
2664UNTIL
2664UNTILP To
OBJ
"DEF +OBJN HatRack
#37753775
37775»
TURNh
DIR Lefti–
2666664TRAVEL
266664ALONG
266664ALONGP Onto
PATH
264DEF +APPEAR BluePATHN Hall
375377775
3777753777775
2666666666664TRAVEL
266666666664
PAST
2664PASTP Past
OBJ
"NUMBER 2OBJN Chair
#3775
UNTIL
2664UNTILP To
OBJ
"DEF +OBJN Lamp
#3775
377777777775
3777777777775
»TURN
hDIR Left
i–
2666666666666666664
TRAVEL
2666666666666666664
UNTIL
2664UNTILP Into
STRUCT
"DEF +STRUCTN Corner
#3775
ORIENT
266666664
ORIENTP SuchThat
DESC
2666664OBJ
"DEF +OBJN Lamp
#
SIDE
"SIDEP BehindAGENT You
#3777775
377777775
3777777777777777775
3777777777777777775
26666666666664DESC
2666666666664
SIDE
26664SIDEP To
DIR
24RELATIVEh
AGENT Youi
DIRN Right
3537775
VIEW
264PATH
264DEF -APPEAR Gray
Carpet
375375
3777777777775
37777777777775
