(January 2008) : Session on Cognitive Maps

Cognitive MappingShrihari Vasudevan and Roland Siegwart

Autonomous Systems Lab, ETH Zurich, [email protected] , [email protected]

with inputs from Elin A. Topp ([email protected]), KTH Stockholm, Sweden &

Ben Krose ([email protected]), UVA Amsterdam, The Netherlands.

mailto:[email protected]




Shrihari Vasudevan [ [email protected] ]

Outline

Introduction

State-of-the-art (SOA)Robot Mapping

Issues with the SOA

ApproachesETH , KTH & UVA

Conclusion



State of the Art (SOA)

SOA in Robot mapping Metric MapsTopological maps Hybrid mapsHierarchical Representations of SpaceObject MapsCognitive Maps


Robot Mapping = Very well researched problem [Thrun02]


Metric Mapping

Maps the environment using its geometric properties or

features.Occupancy of space => Occupancy Grid Maps [Elfes87, Burgard99]

Picture courtesy: [Elfes87]Picture courtesy: [Burgard99]


Geometric features such as lines, corners etc. => Metric maps [Chatila85, Wijk00, Nguyen06 & Martinelli04].

Picture Courtesy: [Martinelli04]Picture Courtesy: [Nguyen06]


Topological Maps

Usually encodes place related data and how to get from

one place to another.

More abstract than metric mapsLess accurate

Less Voluminous

[Shatkay97, Choset01 & Tapus05]


Picture Courtesy:[Choset01] Picture Courtesy:[Shatkay97]


Fingerprint based Topological Mapping

Fingerprint = circular list of features extracted from the environment. Each fingerprint is a signature of the place.

Features include vertical lines, corners, color blobs etc.

Clustering fingerprints together & placing new nodes when significant differences are observed can lead to a topological map of the environment.


Picture Courtesy:[Tapus05]


contd.

We will try to do a small implementation of this technique in the exercise.

Picture Courtesy:[Tapus05]


Hybrid Maps

Attempt to capture the advantages of both metric and

topological maps

Typically, a global topological map for moving between

“places” and local metric maps for precision navigation.

Removes the need for global metric consistency

particularly difficult in odometry based approaches.

[Thrun98 & Tomatis03]


Hierarchical Representations of Space

Represent space in the form of

a multi-layer representation

[Kuipers00, Martinelli03,

Galindo05, …]

[Kuipers00] – Spatial Semantic

Hierarchy

Sensorimotor – Control –

Causal – Topological - Metric

Picture Courtesy : [Kuipers2000]


contd.

[Galindo05] Two separate hierarchies that are linked.

Spatial hierarchy = hybrid topological –metric (grid) map. Inserts objects in occupancy grid maps

Semantic hierarchy = based on AI language (NeoClassic) Picture Courtesy : [Galindo2005]


[Martinelli03]

• Integration of the relative map work and fingerprint work.

• Proposed a Semantic – Topological – Metric hierarchical representation of space.

Picture Courtesy: [Martinelli03]


Object MapsMostly oriented towards navigation. Basically try to either use or model objects. [Brezetz94, Limeketkai05, Mozos07 Ranganathan07, … ]

[Brezetz94]Uses range scans to segment objects and represent them in a topological framework.

[Limeketkai05]Relational Object maps of walls and doors

[Mozos07]Objects used for interpretation (not object map per se)

[Ranganathan07]Object based representation of space (focus = perception)


contd.

Picture Courtesy: [Brezetz94] Picture Courtesy: [Ranganathan07]


Cognitive MapsImportant works on perception, representation, cognition of space – [Tolman48, Gibson50, Marr82, Yeap2001, McNamara86 & 2003, Haffner2000].Term coined by Tolman in 1948Many research works have since tried to model the cognitive map.

Kuipers (Spatial semantic hierarchy , [Kuipers2000])Haffner (place cells , [Haffner2000])Yeap (Space based & Object based approaches, [Yeap2001])McNamara (not pure hierarchy / not pure non-hierarchy ; both visual memory and inter-object relationships., [McNamara 1986 & 2003])and many more…


Issues with SOA

Only suited to robot navigation.

Doesn’t encode much or most of the semantics.

Spatial awareness of robots – modest.


Approach

ETHMethodology

Implementation- Perception

- Representation

- Cognition

User Studies

KTHMethodology

User Studies

Implementation

UVAAppearance graph based representation.

Navigation


• Sensor information extract high level features such as objects, doors etc.

• From sensory information to increasingly abstract concepts of space.

• Representation Single hierarchical representation wherein, objects are grouped along two dimensions – spatial and semantic.

• Semantic abstraction (groups) Objects are clustered into groups to capture their semantics (functionality / arrangement etc.)

• Spatial abstraction (places) A collection of groups of objects, typically formed by structural/ boundary elements such as doors and walls.

Approach


contd.

Change the feature set !

Represent inter-object relationships using a mathematical and numerical basis

Generate spatial and semantic abstractions of space from the base representation

Reference: [Vasudevan@LNAI07]


Approach contd.

Sensory information [ laser / vision ]

High level features (objects, doors etc.)

Semantic abstractions (groups)

Spatial abstractions (places)

Object recognition, door detection etc.

Conceptualization -functionality / arrangement / other semantics.

Conceptualization / Classification -functionality

Conceptualization / place formation boundary elements (doors / walls.)










POGM

Stereo - SIFT Object recognition

Laser – Door detection Object based map

Robot exploration / mapping Reference: Vasudevan@RAS07


Object Recognition

Using Lowe’s SIFT features

Using a limited set of textured objectsDifferent cartons / chair / mug / shelf / table & book.

SIFT (Lowe2004)“Local features” approachDoes NOT learn general properties of objectsTransforms a naively obtained set of features into a robust feature set that incorporates invariance to scale / rotation and to some extent deals with illumination changes and changes in viewing direction. Our experience – good for textured objects and specifically for object detection.


Object Recognition


Door Detection

a) Track door hypotheses

b) Likelihood based filtering out of false positives

c) Establish door on crossing. Establish reference for place.


Probabilistic graphical representation

Representation encodes objects and relationships between them.

Relationships are encoded in terms of relative spatial information (distance and angle measures in 3D space) between objects. These were meant to metrically capture typical spatial relationships such as nearness, to-the-right-of, above, below etc.

Both existential (discrete belief measure) and position beliefs (Gaussian - mean / covariance matrix) are maintained and treated separately.

Design = extended (over several places) relative metric map or global topological map of local metric maps.


Representation – some detailsOdometry model – Standard differential drive model.

Stereo model – suggested by Jung and Lacroix for their work on

SLAM using stereo vision.

Belief representation of objects in local maps –

1 2

' '1 1 1 2 2 2

1 2

1

2

( ) * (object in local place reference)

(covariancematrix uncertaintyin object position)

( ) & ( )

( , , ) is the robot pose( , , ) is theobject posi

o c RO CR

o

X X

R R R

C C C

X f X where f M M

P F PF F P FwhereF J f F J f

whereX X YX X Y

θθ

= =

= + −

= =

→→

1

2

tion in the camera reference frame is the covariance matrix representing the uncertainty in robot position. is the covariance matrix representing the uncertainty in the object position.

PP


Representation – more details

Belief representation of relative spatial information between objects in local map

1 2

1 1 1 1 2 2 2 2

1 2

1 2' '

1 1 1 1 2 2 2

1 2

X ( , , ) & X ( , , ) re p re se n t tw o o b je c ts( , ) re la tiv e sp a tia l in fo rm a tio n b e tw e e n th e tw o o b je c ts . & u n c e r ta in ty in o b je c t p o s itio n s

( )( )

( ) & (X X

x y z x y zf X XP P

B e l f F P F F P Fp re c is io n

w h e re F J f F J

→→

= +

= =

2

)

( ) m in ( )( ) ( )

f

a n d B e l f b e lie f in e x is te n c e o f o b je c tse x is te n c e fro m O R sy s te m

=


Place Classification (M1)

Spatial Cognition – Place classification & Place recognition.


contd.

















Naïve Approach


Issues

Semantics only represented by the presence of objects.

Place classification was naïve (denote approach as M1)No learning from negative exemplars

Inference based only on evidence present and not on that which was absent.

Could not deal with multi-object occurrences and only worked on the basis of object category presence/absence.

Thus, motivation for the following workImprove place classification procedure

Increase semantic content in robot maps. (i.e. form concepts)


Overview of attempted approaches

Reference: Vasudevan@RAS08


General Approach: Learning

No specific ontology used.

Robot has an object recognition capability.

Human user “teaches” (or bootstrap with a database of exemplars)

the robot different concept instances. (Home Tour scenario).


Conceptualization and Place

Classification : Incremental

process where objects perceived

are grouped into clusters with each

of them being inferred as an

instance of a concept. These

concepts are used to classify the

place.

General Approach: Cognition

Two processes: Clustering & Conceptualization


Clustering

Based on distance and semantic models learnt.

Distance nearest neighbor ; distance to center of cluster

Closest semantic groupMAP estimate of the best case concept suggested by the concept models learnt, for each perceived objectUses models learnt ( M2 ) as the best group implied by the occurrence of the single object is required. M2 modeled the likelihood of the occurrence of an object in an instance of a concept.


contd.

Behavior (in same order)Choose the nearest cluster that has the same concept as the best case concept

Choose the nearest cluster that is conceptually dissimilar but “acceptably likely” with respect to the best case concept.

Create a new cluster of type suggested by best case concept.


Naïve Bayes framework

M4 = Object count (M3) + spatial relationships (distances)

Object model: M3 (object count) with a Gaussian uncertainty applied on the training data.

Relationship model: Gaussian mixture model learnt using EM algorithm.

Approach (M4): Details


Object model

A fixed Gaussian uncertainty added to training input. Every

training exemplar will affect o_i = m_i – 1 , m_i , m_i +1. Attempt

to improve generalization capability by handling conceptually

adjacent cases.

Encodes the likelihood of the occurrence of a specific

number of a particular object towards the formation of a

particular concept.

( | ) (2 )i io m

i i iexemplars

NP o m c N

δδ

= += =

+


Relationship ModelChoose relationships to be modeled

Based on number of occurrences- Both for significance and for practicality of GMM modeling

EM based GMM modeling of relationshipMotivation for GMM – we want to capture characteristic relationships (values) between objects.Use uncertain data (ie. relationship values with associated covariance matrices encoded in the representation).BIC based model selection

Check modeling for selection of relationshipParametric bootstrapping in conjunction with a student’s t-test (two tailed; 1 % level of significance) to check the modeling of the data.


M4: Bayesian Program


Experiments: Sample Output


ExperimentsPhysically measured dataset of 11 offices and 8 kitchens (19 places, 991 objects & ~77 object types).

Clustering outcomesObject in correct cluster (~69 %)Object fused with another cluster (~30 %)Object in singleton cluster (~ 1%)

Conceptualization outcomesObject correctly conceptualizedObject in cluster that has not been classifiedFree object in training assigned a label in testingObject incorrectly conceptualized.


Evaluation of Conceptualization • Nocc = 5, 10, 20, 30 were

attempted. Higher Nocc => fewer relationships => For a while better performance; however M4 will reduce to M3 in limiting case (no relations).

• Overall => reduction in number of false positives (at the expense of more unclassified cases) and an increase in the classification accuracy (correctness amongst classified cases) for a similar number of correctly classified cases in the overall dataset. Inability to classify better than mis-classification =>improvement

Nocc = 20 (M4)

Benchmark (M3) (prior work)

Place Classification accuracy = 100 %

Place Classification accuracy = 100 %


Further experiments on M3 and M4K(=8)FSCV & LK(=2)OCV



Conclusions of C&C approach

Bayesian approach conceptualization and place classification.

Based on Naïve Bayes framework; an object model that models the likelihood of the occurrence of a specific number of instances of an object in that of a concept; and a relationship model based on Gaussian mixture models. Incorporating relationships improves performance of the approach.

- Major effect : Reduce incorrect outcomes by not classifying them- Minor effect : Reduce incorrect outcomes by correctly classifying them

Grounded in underlying representation Not ontology specific.Generative and thus can enable other kinds of reasoning as well.


User Studies

Human centered experiment to validate and enhance representation.52 people ; tour of lab premisesStrictly an AI / robotics / engineering work Not a new theory of the mind

does not answer the question of how the brain represents information.

Not an ontology an attempt to propose a more general framework where the users can teach the robot “ontologies” suited to them.

Reference : Vasudevan@ICRA07-SIR


Places

Objects = desks, drawers, sofa, coffee table….Function = place for rest / meet people / work…Boundaries = many windows, 2-3 doors leading to other rooms, high ceiling,…Size = 40-50 sq. m , big place, …Ambiance = calm, live, congenial, …Luminosity = naturally lighted, very illuminated,…Ground materials = carpet, flooring, …


Places - results

• Representation – objects and boundary elements most referred. • Description – objects and functionality most important

Representation of an office Representation of a living room

Description of an office Description of a refreshment room


Object Arrangement


Change of Place

Boundary elements & object arrangements are key to place formation


GroupsEntrance hall

Laboratory-office


Groups contd. Office

Refreshment room


Groups - results

Zone definition – entrance hall Zone definition – lab-office

Zone definition – refreshment room Zone definition – office

• Zones = spatial (boundary elements – doors / walls / partitions…) and semantic ( arrangements of objects and similarity of purpose / function)

• Entrance hall and lab-office = larger spaces, both semantic and spatial abstractions. In lab-office, boundary elements mainly facilitate zone formation. Refreshment room and office = smaller spaces, semantic abstractions within one spatial abstraction

• Spatial abstractions contain semantics (indicated)


Some ResultsHumans perceive space in terms of high-level information such as objects, states and descriptions.

Inferred from results on place representation, place descriptionand scene interpretation

Boundary elements help in place formationInferred from results on change of place

Existence of Spatial and Semantic abstractions of spaceInferred from experiments on Groups

Spatial abstractions subsume semantic onesIndicated but not clearly shown


Conclusion of User StudyBroad validation of approach.

Cognitive = human compatible and not human like in our

approach.

Human compatible representation of space and a human-

like conceptualization of space.

Some issues need to be better addressed and some

others were discovered during the study

Provided an empirical basis for the intuitive assumptions

we make.


Practical Applications of Work

Context based disambiguation of objects

Context sensitive (intelligent) path planning

Intelligent HRISpatial / Social awareness

In addition to robot navigation.


Overall SummaryRepresentation

Hierarchical probabilistic concept oriented representation of space.Objects and spatial relationships between them.Human compatible representation of space.

CognitionGenerative Bayesian approach to conceptualization and place classification.Can map sensory information to increasingly abstract concepts towards creating a more semantic map for mobile robots.


contd.

Perception and User Studies.

Increase in semantics in robot representations. Increase in Spatial awareness in robots.

Outcome Application

- Towards bringing robots into our homes.

Scientific- Towards bridging symbolic world (traditional AI) and Robotics


Approach

ETHMethodology


- Representation

- Cognition

User Studies

KTHMethodology

User Studies

Implementation


Navigation


Approach

ETHMethodology


- Representation

- Cognition

User Studies

KTHMethodology

User Studies

Implementation


Navigation


Take home points !!!

Cognitive Mapping / Semantic Mapping is a very active

research area in mobile robotics / HRI.Abstraction & Hierarchies are necessary !

Object / Appearance based approaches

Intelligence, Initiative, Personalization & Human Augmentation

You will need it for any intelligent HRI.

You did :reviewed robot mapping literature.

saw three approaches today - all are state-of-the-art and being actively researched on.


Thank You !


References[Thrun2002]

Sebastian Thrun. Exploring Artificial Intelligence in the New Millenium, chapter : Robotic mapping: A survey. Morgan Kaufmann, 2002.

[Elfes1987] Alberto Elfes. Sonar based real world mapping and navigation. IEEE Journal of Robotics and Automation, 3(3):249–265, June 1987.

[Burgard99]W. Burgard, D. Fox, H. Jans, C. Matenar, and S. Thrun. Sonar-Based Mapping of Large-Scale Mobile Robot Environments using EM. In the proceedings of the International Conference on Machine Learning (ICML) 1999.

[Martinelli04]A. Martinelli, A. Svensson, N. Tomatis and R. Siegwart. SLAM Based on Quantities Invariant of the Robot's Configuration. IFAC Symposyum on Inteligent Autonomous Vehicles, 2004.

[Nguyen06]V. Nguyen, A. Harati, N. Tomatis, A. Martinelli, R. Siegwart, Orthogonal SLAM: a Step toward Lightweight Indoor Autonomous Navigation, Proc. of The IEEE/RSJ Intenational Conference on Intelligent Robots and Systems (IROS), 2006.

[Choset01]H. Choset and K. Nagatani. Topological simultaneous localization and mapping (SLAM): toward exact localization without explicit localization. IEEE Transactions on Robotics and Automation, Volume: 17 Issue: 2 , Apr 2001.


contd.[Shatkay97]

H. Shatkay and L.P. Kaelbling, Learning Topological Maps with Weak Local Odometric Information. In the Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI 1997), 1997.

[Tapus05]Adriana Tapus (2005) "Topological SLAM - Simultaneous Localization and Mapping with Fingerprints of Places" PhD Dissertation, Swiss Federal Institute of Technology Lausanne (EPFL), Switzerland, October 2005

[Thrun98]S. Thrun. Learning metric-topological maps for indoor mobile robot navigation. Artificial Intelligence, Volume 99, Issue 1, February 1998, Pages 21-71.

[Tomatis03]N. Tomatis, I. Nourbakhsh and R. Siegwart. Hybrid Simultaneous Localization and Map Building: A Natural Integration of Topological and Metric. Robotics and Autonomous Systems, 44, 3-14., July 2003.

[Kuipers00]B. Kuipers. The Spatial Semantic Hierarchy. Artificial Intelligence, 119: 191-233, May 2000.

[Martinelli03]A. Martinelli, A. Tapus, K.O. Arras and R. Siegwart. Multi-resolution SLAM for Real World Navigation. In Proceedings of the 11th International Symposium of Robotics Research, Siena, Italy, 2003.

[Galindo05]C. Galindo, A. Saffiotti, S. Coradeschi, P. Buschka, J.-A. Fernandez-Madrigal, and J. Gonzalez, “Multi-Hierarchical Semantic Maps for Mobile Robotics,” in IEEE/RSJ Interrnational Conference on Intelligent Robots and Systems (IROS), Edmonton, Canada, 2005, pp. 3492–3497.


contd.

[Brezetz94]S.B. Brezetz, R. Chatila and M. Devy. Natural scene understanding for mobile robot navigation. In the Proceedings of the IEEE International Conference on Robotics and Automation 1994, San Diego, USA

[Mozos07] O.M. Mozos, R. Triebel, P. Jensfelt, A. Rottmann, and W. Burgard. Supervised semantic labeling of places using information extracted from sensor data. Robotics and Autonomous Systems, 55(5):391–402, 2007

[Limeketkai05]B. Limketkai, L. Liao, and D. Fox. Relational Object Maps for Mobile Robots. In the Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), Edinburgh, Scotland, 2005.

[Ranganathan07]A. Ranganathan and Frank Dellaert. Semantic modeling of places using objects. In Robotics Science and Systems (RSS), 2007.

[Tolman48]E. C. Tolman. Cognitive maps in rats and men. Psychological Review, 55:189-208, 1948.

[Yeap2001]Wai-Kiang Yeap and Margaret E. Jefferies. On early cognitive mapping. Spatial Cognition and Computation, 2(2):85–116, 2001.

[McNamara86]T.P. McNamara. Mental representations of spatial relations. Cognitive Psychology, 18:87–121, 1986.


contd.[Haffner2000]

V.V. Hafner. Learning Places in Newly Explored Environments. In the proceedings of the International Conference on Simulation of Adaptive Behavior, September 2000, Paris, France.

[Gibson50]J. J. Gibson. The perception of visual surfaces. American Journal Of Psychology, 63(3):367–384, 1950.

[Marr82]D. Marr. Early processing of visual information. Philosophical Transactions of the Royal Society of London Series B-Biological Sciences, 275(942):483–&, 1976.


[Vasudevan@LNAI07]S. Vasudevan, S. Gachter, A. Harati and R. Siegwart (2007) A hierarchical Concept-oriented Representation for Spatial Cognition in Mobile Robots, In M. Lungarella, F. IIda, J. Bongard and R. Pfeifer (eds.), 50 Years of Artificial Intelligence, Springer Lecture Notes in Artificial Intelligence, Vol. 4850.

[Vasudevan@RAS07]S. Vasudevan, S. Gachter, V.T. Nguyen and R. Siegwart (2007) Cognitive Maps for Mobile Robots -An object based approach. Robotics and Autonomous Systems, Volume 55, Issue 5, From Sensors to Human Spatial Concepts, 31 May 2007, Pages 359-371

[Vasudevan@ICRA07-SIR]S. Vasudevan, S. Gachter and R. Siegwart (2007) Cognitive Spatial Representations for Mobile Robots – Perspectives from a user study. In the proceedings of the IEEE International Conference on Robotics and Automattion - Workshop: Semantic Information in Robotics (ICRA - SIR 2007), Rome, Italy

[Vasudevan@RAS08]S.Vasudevan and R. Siegwart (2008) Bayesian Space Conceptualization and Place Classification for Semantic Maps in Mobile Robotics. Accepted for publication in Robotics and Autonomous Systems.(to be up soon; the IROS-FS2HSC , ECMR and IROS papers have most of the ideas)

http://www.asl.ethz.ch/people/vasudevs/personal OR http://www.asl.ethz.ch/research/asl/cogniron for the papersSome other useful references in the exercise !

References related to this Approach

http://www.asl.ethz.ch/people/vasudevs/personal

http://www.asl.ethz.ch/research/asl/cogniron

Documents

(January 2008) : Session on Cognitive Maps