Grounding Symbols in Multi-Modal Instructions

Yordan Hristov, Svetlin Penkov, Alex Lascarides and Subramanian RamamoorthySchool of Informatics

The University of Edinburgh{yordan.hristov@, sv.penkov@, alex@inf., s.ramamoorthy@}ed.ac.uk

Abstract

As robots begin to cohabit with humansin semi-structured environments, the needarises to understand instructions involv-ing rich variability—for instance, learningto ground symbols in the physical world.Realistically, this task must cope withsmall datasets consisting of a particularusers’ contextual assignment of meaningto terms. We present a method for process-ing a raw stream of cross-modal input—i.e., linguistic instructions, visual percep-tion of a scene and a concurrent traceof 3D eye tracking fixations—to producethe segmentation of objects with a cor-respondent association to high-level con-cepts. To test our framework we presentexperiments in a table-top object manip-ulation scenario. Our results show ourmodel learns the user’s notion of colourand shape from a small number of phys-ical demonstrations, generalising to iden-tifying physical referents for novel combi-nations of the words.

1 Introduction

Effective and efficient human-robot collaborationrequires robots to interpret ambiguous instructionsand concepts within a particular context, commu-nicated to them in a manner that feels natural andunobtrusive to the human participant in the inter-action. Specifically, the robot must be able to:

• Understand natural language instructions,which might be ambiguous in form andmeaning.

• Ground symbols occurring in these instruc-tions within the surrounding physical world.

Figure 1: Combining natural language input witheye tracking data allows for the dynamic labellingof images from the environment with symbols.The images are used in order to learn the meaningof language constituents and then ground them inthe physical world.

• Conceptually differentiate between instancesof those symbolic terms, based on featurespertaining to their grounded instantiation,e.g. shapes and colours of the objects.

Being able to relate abstract symbols to obser-vations with physical properties in the real world isknown as the physical symbol grounding problem(Vogt, 2002); which is recognised as being one ofthe main challenges for human-robot interactionand constitutes the focus of this paper.

There is increasing recognition that the mean-ing of natural language words derives from howthey manifest themselves across multiple modali-ties. Researchers have actively studied this prob-lem from a multitude of perspectives. This in-cludes works that explore the ability of agents tointerpret natural language instructions with respect

to a previously annotated semantic map (Matuszeket al., 2013) or fuse high-level natural language in-puts with low-level sensory observations in orderto produce a semantic map (Walter et al., 2014).Matuszek et al. (2014); Eldon et al. (2016) andKollar et al. (2013) tackle learning symbol ground-ing in language commands combined with gestureinput in a table-top scenario. However, all theseapproaches depend on having predefined specifi-cations of different concepts in the environment:they either assume a pre-annotated semantic mapwith respect to which they ground the linguistic in-put or have an offline trained symbol classifier thatdecides whether a detected object can be labelledwith a specific symbol; e.g. colour and shape in(Matuszek et al., 2014). Thus in order to deploysuch a system, one should have access to an al-ready trained classifier for every anticipated sym-bol, prior to any user interaction.

Multi-modal learning algorithms based on deepneural networks are also popular for groundingnatural language instructions to the shared phys-ical environment (Srivastava and Salakhutdinov,2012; Ngiam et al., 2011). But the majority ofthese algorithms depend crucially on large andpre-labelled datasets, and the challenge is in col-lecting these large-scale labelled datasets so thatthey not only capture the variability in languagebut also manage to represent the nuances (espe-cially across multiple high-bandwidth modalities,such as vision and eye-tracking) of inter-personalvariability in assignment of meaning (e.g., whatone person calls mustard another might call yel-low), which we claim is a key attribute of free-form linguistic instructions in human-robot inter-action applications. If a previously unseen instruc-tion/visual observation is presented to these sys-tems, they might fail to ground or recognize themin the way that the user might have intended in thatspecific setting. Tobin et al. (2017) potentially by-passes the need to collect a big dataset by demon-strating that a model trained in simulation can besuccessfully deployed on a robot in the real world.However, the problem is then shifted to generatingtask-specific training data in a simulator which ap-proximates the real world well enough.

A proposed alternative to this off-line learn-ing approach is to interactively teach an embod-ied agent about its surrounding world, assuminglimited prior knowledge. Al-Omari et al. (2016)demonstrates a model for incrementally learning

the visual representation of words, but relies ontemporally aligned videos with corresponding an-notated natural language inputs. Parde et al.(2015) and Thomason et al. (2016) represent theonline concept learning problem as a variationof the interactive “I Spy” game. However, theseapproaches assume an initial learning/exploratoryphase in the world and extracted features are usedas training data for all concept models associatedwith an object.

Penkov et al. (2017) introduce a method calledGLIDE (see §2.2 for details), which successfullyteaches agents how to map abstract instructions,represented as a LISP-like program, into theirphysical equivalents in the world. Our work buildson this method: it uses it to achieve natural lan-guage symbol grounding, as a by-product of userinteraction in a task-oriented scenario. Our ap-proach achieves the following:

• It maps natural language instructions to aplanned behaviour, such as in a robotic ma-nipulation domain; in so doing it supportsa communication medium that human usersfind natural.

• It learns symbol grounding by exploiting theconcept of intersective modification (Morzy-cki, 2013) —i.e., an object can be labelledwith more than one symbol. The meaning ofthe symbols is learned with respect to the ob-served features of the instances of the object.

In our work the agent assumes some priorknowledge about the world in the form of low-level features that it can extract from objects inthe visual input—e.g. intensities in the primarycolour channels and areas of pixel patches of anyspecific colour. On top of this, we learn classifiersfor performing symbol grounding. Each symbolhas a probabilistic model which is fit to a subsetof the extracted (visual) features. When a new in-struction is received, the classifier for each symbolmakes a decision regarding the object in the world(and their respective features) to which the sym-bol may be grounded. Crucially, the data fromwhich these classifiers are learned is collectedfrom demonstrations at ‘run time’ and not priorto the specific human-robot interaction. Images ofobjects are extracted from the high-frequency eyetracking and video streams, while symbols that re-fer to these objects in the images are extractedfrom the parsed natural language instructions—see Figure 1. Through cross-modal instructions,

Figure 2: Overview of the full system pipeline. Input to the system are natural language instructions,together with eye-tracking fixations and a camera view of the world from above (a) Natural languageinstructions are deterministically parsed to an abstract plan language (b) Using the abstract plan, a setof labelled image patches is produced from the eye-tracking and video data (c) Observable predefinedfeatures are extracted from the image patches (d) Each symbol is grounded to a subset of observablefeatures (e)

the human participant is simultaneously teachingthe robot how to execute a task and what prop-erties the surrounding objects must have for thatexecution to be successful. For instance, whileobserving how to make a fruit salad in a kitchen,apart from learning the sequence of steps, the sys-tem would also gain an initial approximation ofthe visual appearance of different pieces of fruitand their associated natural language symbols.

2 Methods

Figure 2 depicts the architecture of the overall sys-tem. It consists of an end-to-end process, fromraw linguistic and video inputs on the one hand tolearned meanings of symbols that in turn are con-ceptually grouped: i.e., a symbol can correspondeither to an object in the real world, or to a prop-erty of an object. The rest of the section is orga-nized in the following fashion - each subsectioncorresponds to a numbered transition (1 to 4) indi-cated in Figure 2.

2.1 Natural Language Semantic Parsing

The task of the semantic parser is to map nat-ural language requests into instructions repre-

sented in an abstract form. The abstract formwe use is a list of tuples with the format(action target location) (Figure 2b),where action corresponds to an element froma predefined set A, target corresponds to a listof terms that describe an object in the world andlocation corresponds to a single symbol denot-ing a physical location in the environment.

The narration of the plan execution by the hu-man comprises one sentence per abstract instruc-tion. Therefore, given a plan description, our se-mantic parser finds a mapping from each sentenceto a corresponding instruction as defined by ourabstract plan language.

Elementary Dependency Structures (EDS)(Oepen et al., 2004), which are output by parsingthe sentence with the wide-coverage EnglishResource Grammar (Flickinger et al., 2014), areused as an intermediate step in this mappingprocedure. EDS are given as dependency graphs(Figure 3) and are a variable-free reduced formof the full Minimal Recursion Semantics (MRS)(Copestake et al., 2005) representation of the nat-ural language input. Given EDS for a particularsentence, parsing proceeds in two steps:

Figure 3: Example of dependency graphs for input sentences. Red labels denote the top graph nodes.

• The graph is extracted from nodes and theirrespective edges, cleaning up nodes andedges that do not contribute directly to themeaning of the instruction—i.e., anythingthat is not a verb, adjective, preposition or anoun;

• The processed graph is recursively traverseduntil all nodes have been visited at least once.

Knowing the format of our abstract plan lan-guage, we know that action would alwayscorrespond to the verb in the input sentence,target would correspond to a noun phrase andlocation would correspond to a prepositionalphrase (i.e., a combination of a preposition andnoun phrase). For us, the noun phrases all con-sist of a noun, complemented by a possibly emptylist of adjectives. Extracting the action is straight-forward since the top node in the EDS always cor-responds to the verb in the sentence; see Figure 3.The extracted action is then passed through a pre-defined rule-based filter which assigns it one of thevalues from A: e.g. pick, grab, take, get would allbe interpreted as pick.

The target entry can be extracted by identi-fying noun node in the EDS that’s connected tothe verb node. Once such a noun is found, one canidentify its connections to any adjective nodes—this gives a full list of symbols that define the ob-ject referenced by the target.

The location entry can be extracted bysearching for preposition nodes in the EDS thatare connected to the verb node. If there is nosuch node, then the location is constructed di-rectly from the target by concatenating its labels- e.g. for a blue cube the location would bethe symbol blue-cube-location. Extract-ing the location from a prepositional phraseis less constrained since different verbs can berelated to spatial prepositions in varied ways—either the preposition node has an edge connect-

ing it to the verb node or vice versa. Once aprepositional node is visited, we proceed by re-cursively exploring any chains of interconnectednouns, prepositions, and adjectives. The recursioncalls for backtracking whenever a node is reachedwith no unvisited incoming or outgoing edges:e.g., node cube on Figure 3 (bottom). For exam-ple, the symbol on-left-of-cube is producedfor location for the bottom sentence in Figure3.

In this way, the result of parsing is a sequenceof abstract instructions—i.e., an abstract plan—together with a symbol set S, containing all sym-bols which are part of any target entry. At thispoint, the symbols are still not actually groundedin the real world. Together with the raw video feedand the eye-tracking fixations, the abstract planbecomes an input to GLIDE (Penkov et al., 2017).

2.2 Grounding and Learning Instancesthrough Demonstration and Eye tracking(GLIDE)

Penkov et al. (2017) introduce a frameworkfor Grounding and Learning Instances throughDemonstration and Eye tracking (GLIDE). Inthis framework, fixation programs are repre-sented in terms of fixation traces obtained dur-ing task demonstrations combined with a high-level plan. Through probabilistic inference overfixation programs, it becomes possible to inferlatent properties of the environment and deter-mine locations in the environment which corre-spond to each instruction in an input abstract planthat conforms to the format discussed above -(action target location).

2.2.1 3D Eye-TrackingMobile eye trackers provide fixation informationin pixel coordinates corresponding to locations inthe image of a first person view camera. In orderto utilise information from multiple input sensors,

Y1 . . . Yl1 Ys2 . . . Yl2 . . . YsL . . . YT

A1 A2 AL

E1 E2 EL

. . .

. . .

P

T

Figure 4: The proposed probabilistic model forphysical symbol grounding is based on the ideathat “task and context determine where you look”(Rothkopf et al., 2007).

an additional camera may be attached to augmentan eye-tracker, by running mono camera SLAMalgorithm in the background—ORBSLAM (Mur-Artal et al., 2015). The SLAM algorithm pro-vides 6D pose of the eye tracking glasses withinthe world frame; this allows for the fixation lo-cations to be projected into the 3D world by raycasting and finding intersections with a 3D modelof the environment. As a result, fixations can berepresented as 3D locations, enabling the projec-tion of fixations in the frame of any sensor in theenvironment.

2.2.2 Model and Location InferenceIn order to solve the problem of symbol ground-ing, inference is performed using a generativeprobabilistic model, which is shown in Figure 4.The sequence of fixations Y1 : YT depend bothon the current environment state E and the actionbeing executed A. Each action is part of the planP which is determined by the task being demon-strated T . The sequence of fixations is observedand interpreted with respect to a task that is bythis stage already known, while the state of theenvironment and the current action are unknown.The main inference task is to determine the struc-ture of the model and assign each fixation to theaction that is its cause. A crucial feature of thisprocedure is the fact, deriving from human sen-sorimotor behaviour, that the distribution of fix-ations is different when a person is attending tothe execution of an action compared to periods oftransitions between actions in the plan. By utilis-ing this property and using samples from a Dirich-let distribution to describe these transition points,GLIDE is able to infer the correct partitioning of

Algorithm 1: Symbol Meaning LearningInput: σthreshData: I , S, FOutput: K = {(µ1,Σ1), . . . , (µS ,ΣS)},

C = {(F sinvar : s), . . . (FS

invar : S)}1 Data← [s1 : {}, . . . , sS : {}];2 for image i in I do3 symbolsi ← GetSymbols(i);4 featuresi ← ExtractFeatures(i);5 for symbol in symbolsi do6 Append featuresi to Data[symbol];7 for s in S do8 Ks ← FitNormal(Data[s]);9 Ks ← CleanNoise(Data[s]);

10 F sinvar ← FindInvFeat(Ks, σthresh);

11 Ks ← RefitNormal(Ks, Fsinvar);

12 Append (F sinvar : Ks) to C;

the fixation sequence. This information allows usto localise each item of interest in the environmentand extract labelled sensory signals from the en-vironment. A complete description of this modeland inference procedure can be found in (Penkovet al., 2017).

2.3 Feature Extraction

The parser produces a set of symbols S andGLIDE produces a set of image patches I , each ofwhich is labelled with a subset of symbols fromS. We proceed by extracting a number of fea-tures, drawn from a pre-existing set F . The fea-tures are derived from the objects in the imagepatches, after removing the background. This isachieved through a standard background subtrac-tion method—we know that the majority of the im-age will be occupied by a solid object with a uni-form colour, anything else is a background. For in-stance, in the image patches in Figure 2 (c), the ob-jects are the colourful blocks and the backgroundis the black strips around them. Images containingonly or predominantly background are considerednoise in the dataset and are discarded. For eachsymbol s we group the extracted features fromeach image labelled with s resulting in S lists ofMs tuples with F entries in each tuple, where Ms

is the number of images being labelled with s; seeFigure 2 (d, left). The data for each feature is nor-malized to fall between 0 and 1.

Figure 5: Example of feature distributions forblue (top) and cube (bottom) symbols.

2.4 Symbol Meaning Learning

For each symbol s ∈ S and each feature f ∈F we fit a 1-D Normal distribution resultingin a new list of tuples with size F - sj :[(µ

sjf1, σ

sjf1

), . . . , (µsjfF, σ

sjfF

)] for the jth symbol.Taking into account that the object location pro-cess in GLIDE could still produce noisy results—i.e., the label of an image can be associated withthe wrong symbol—we process our distributionsto refit them to data that falls within two standarddeviations from the means of the original distri-butions. We are then seeking observed features fthat are invariant with respect to each token use ofa specific symbol s within the user instructions sofar—i.e. their distributions are ‘narrow’ and withvariance below a predefined threshold σthresh (seeFigure 5). If we have a set of images that are ofblue objects with different shapes and we extracta set of features from them, we would expect thatfeatures with lower variation (e.g. RGB channelsas opposed to area) would explain the colour bluebetter than features with more variation (i.e. pixelarea).

In the last step, we construct a set of the invari-ant features from the discovered narrow distribu-tions for a given symbol l - (F s

invar) - and say thatthis set characterizes the symbol. The parametersfor the symbol are the concatenation of the means

of the features from (F linvar) into a mean vector

and the concatenation of the variances into a diag-onal covariance matrix. The resultant mean vectorand covariance matrix are later used for inferencewhen shown a new set of images.

3 Experiments

We now present results from initial experimentsbased on the framework in Figure 2. We focusour explanation on steps 3 and 4 in that figure, asthese are the pertinent and novel elements intro-duced here. The input data for Figure 2 (c) is de-rived from the process already well described in(Penkov et al., 2017).

3.1 DatasetFor our experiments we used a total of six symbolsdefining S: 3 for colour (red, blue and yellow);and 3 for shape (cell, block, cube). We used fourextracted features for F : R, G, B values and pixelarea. The objects used were construction blocksthat can be stacked together and images of themwere gathered in a tabletop robotic manipulationsetup (see Figure 2 (a)). Based on the empiri-cal statistics of the recognition process in (Penkovet al., 2017), our input dataset to the SymbolMeaning Learning algorithm consists of 75% cor-rectly annotated and 25% mislabelled images. Thetotal training dataset comprised of approximately2000 labelled image patches, each of which islabelled with two symbols—e.g. blue cell,red block, yellow cube, etc.

The additional test set was designed in twoparts: one that would test colour recognition andone that would test shape recognition. Overall, 48objects were presented to the algorithm where thefeatures for each object would fall into one of thefollowing categories:

• Previously seen features (Figure 6 (left))

• Previously unseen features, close to the fea-tures of the training data (Figure 6 (middle))

• Previously unseen features, not close to thefeatures of the training data (Figure 6 (right))

3.2 Experimental Set upInference over new images is performed by thresh-olding the probability density function (PDF) val-ues from the model parameters for each symbol.The idea is to test how well the algorithm can dif-ferentiate the learned concepts with slight varia-tions from concepts it has not seen before: e.g.

Figure 6: Variations in the objects from the test setfor colour (top half) and shape (bottom half)

given that the algorithm was trained on 3 coloursand 3 shapes, we would expect that it should rec-ognize different hues of the 3 colours and objectswith similar shapes to the original 3; however, itmay not be able to recognize objects with com-pletely different features. Moreover, we furthergroup different symbols into concept groups. Ifany two symbols are described by the same fea-tures, it is safe to assume that those two symbolsare mutually exclusive: that is, they can not bothdescribe an object simultaneously. Thus we goover each concept group and if there are symbolsyielding PDF values above a predefined threshold,we assign the new image the symbol from thatgroup with the highest PDF.

3.3 Results

The system successfully learns from the trainingdataset that the colour symbols are being char-acterized by the extracted RGB values, while (incontrast) the shape symbols from the pixel area ofthe image patch—see Figure 7. Given a new testimage with its extracted features, the algorithmrecognises 93% of presented colours and 56% ofpresented shapes. Tables 1 and 2 report the con-fusion matrices for the testing set. This showsthat the system is more robust when recognizingcolours than when recognizing shapes. This canbe attributed to the fact that while RGB values de-scribe the concept of colour well enough, simplythe pixel area is not enough to describe the con-cept of shape. Therefore the algorithm confusesthe rubber duck with a cell, for example, and thearrow with a cube, see Figure 8, principally be-cause they are of a similar size to each other! Infuture work, we would consider a wider range offeatures being extracted from the images, which inturn would support a finer-grained discriminationamong objects.

Table 1: Confusion matrix for colour symbols

Red Yellow Blue UnknownRed 12 1 0 0

Yellow 0 12 0 0Blue 0 0 13 1

Unknown 0 1 0 13Table 2: Confusion matrix for shape symbols

Cell Block Cube UnknownCell 8 1 0 0

Block 1 7 0 0Cube 0 3 8 0

Unknown 5 5 6 4

4 Discussion and Future Work

The experiments in this paper have demonstratedthat it is possible to train classifiers for objectappearance alongside symbols, which are anal-ysed via a semantic parser, to achieve ground-ing of instructions that respect the specificity ofthe scenario within which that association derivesits meaning. Although our framework supportsan entire pipeline, from raw cross-modal input toan interpreted and grounded instruction, the pre-sented scenarios are simple and the specific meth-ods could be made (much) more sophisticated.Despite this, we claim that this provides step-ping stones towards learning more complex lan-guage structures in the future: during the first fewdemonstrations a human could teach a robot fun-damental concepts like colours, shapes, orienta-tion, and then proceed to use this newly gainedknowledge to ground, e.g., prepositional phrases(Forbes et al., 2015; Rosman and Ramamoorthy,2011) or actions (Misra et al., 2016; Zampogian-nis et al., 2015) in an online and context specificmanner. Once the system knows what blue cubeslook like, it would be easier to learn what it meansfor another cube to be on top of/around it.

Another fruitful line of exploration would becontinuous learning of both known and unknownsymbols, using the same conceptual groups thesystem has extracted from the training data. Forinstance, whenever the robot observes a new ob-ject it can either label it with a symbol or deem itas unknown for a particular concept group. When-ever a symbol is assigned, the feature model forthat symbol is updated, taking into account thenew data point. If on the other hand the symbol

Figure 7: Excerpt from the testing dataset of objects whose colour and shape were correctly recognised.

Figure 8: Examples of wrongly assigned knownsymbols to unseen shapes. Leftmost objectsdemonstrate an object from the training data.

is unknown, the system can prompt the human fornew linguistic input which together with its fea-ture model is added to the knowledge base andallows for its future recognition. For example, ifthe robot observes a new hue of blue it would up-date its parameters for blue to account for that;whereas if it observes a new colour (e.g. green) itwould ask the human for the unknown symbol andwould record it for future reference.

The idea of teaching the system about com-pound nouns is also a relevant challenge and apossible extension of this work: our current setuprelies on noun phrases consisting of predicativeAdjs and a Noun (e.g. blue cube), and so weknow that the associated image patch X satisfiesboth the adjective and the noun—i.e., blue(X)and cube(X) are both true. However, this wouldnot apply to a compound noun like steak knife:we know that the associated image patch X sat-isfies knife(X) but does not satisfy steak(X).

Refinements to our model would be necessary inorder to represent more complex symbol relations,e.g. in a hierarchical fashion (Sun et al., 2014).

5 Conclusion

We present a framework for using cross-modal in-put: a combination of natural language instruc-tions, video and eye tracking streams, to simulta-neously perform semantic parsing and groundingof symbols used in that process within the physi-cal environment. This is achieved without relianceon pre-existing object models, which may not beparticularly representative of the specifics of aparticular user’s contextual usage and assignmentof meaning within that rich multi-modal stream.Instead, we present an online approach that ex-ploits the pragmatics of human sensorimotor be-haviour to derive cues that enable the groundingof symbols to objects in the stream. Our prelim-inary experiments demonstrate the usefulness ofthis framework, showing how a robot is not onlyable to learn a human’s notion of colour and shape,but also that it is able to generalise to the recogni-tion of these features in previously unseen objectsfrom a small number of physical demonstrations.

Acknowledgments

This work is partly supported by ERC Grant269427 (STAC), a Xerox University AffairsCommittee grant, and grants EP/F500385/1 andBB/F529254/1 for the DTC in Neuroinformaticsand Computational Neuroscience from the UKEPSRC, BBSRC, and the MRC. We are verygrateful for the feedback from anonymous review-ers.

ReferencesM Al-Omari, P Duckworth, DC Hogg, and AG Cohn.

2016. Natural language acquisition and groundingfor embodied robotic systems. In Proceedings of the31st Association for the Advancement of ArtificialIntelligence. AAAI Press.

Ann Copestake, Dan Flickinger, Carl Pollard, andIvan A Sag. 2005. Minimal recursion semantics: Anintroduction. Research on Language and Computa-tion 3(2-3):281–332.

Miles Eldon, David Whitney, and Stefanie Tellex.2016. Interpreting multimodal referring expres-sions in real time. In International Conference onRobotics and Automation.

Dan Flickinger, Emily M. Bender, and Stephan Oepen.2014. ERG semantic documentation. Accessed on2017-04-29. http://www.delph-in.net/esd.

Maxwell Forbes, Rajesh PN Rao, Luke Zettlemoyer,and Maya Cakmak. 2015. Robot programming bydemonstration with situated spatial language under-standing. In Robotics and Automation (ICRA), 2015IEEE International Conference on. IEEE, pages2014–2020.

Thomas Kollar, Jayant Krishnamurthy, and Grant PStrimel. 2013. Toward interactive grounded lan-guage acqusition. In Robotics: Science and Systems.

Cynthia Matuszek, Liefeng Bo, Luke Zettlemoyer, andDieter Fox. 2014. Learning from unscripted deicticgesture and language for human-robot interactions.In AAAI. pages 2556–2563.

Cynthia Matuszek, Evan Herbst, Luke Zettlemoyer,and Dieter Fox. 2013. Learning to parse natural lan-guage commands to a robot control system. In Ex-perimental Robotics. Springer, pages 403–415.

Dipendra K Misra, Jaeyong Sung, Kevin Lee, andAshutosh Saxena. 2016. Tell me dave: Context-sensitive grounding of natural language to manip-ulation instructions. The International Journal ofRobotics Research 35(1-3):281–300.

Marcin Morzycki. 2013. Modification. Bookmanuscript. In preparation for the Cambridge Uni-versity Press series Key Topics in Semantics andPragmatics. http://msu.edu/ morzycki/work/book.

Raul Mur-Artal, J. M. M. Montiel, and Juan D.Tardos. 2015. ORB-SLAM: a versatile andaccurate monocular SLAM system. IEEETransactions on Robotics 31(5):1147–1163.https://doi.org/10.1109/TRO.2015.2463671.

Jiquan Ngiam, Aditya Khosla, Mingyu Kim, JuhanNam, Honglak Lee, and Andrew Y Ng. 2011. Multi-modal deep learning. In Proceedings of the 28th in-ternational conference on machine learning (ICML-11). pages 689–696.

Stephan Oepen, Dan Flickinger, Kristina Toutanova,and Christopher D Manning. 2004. Research onLanguage and Computation 2(4):575–596.

Natalie Parde, Adam Hair, Michalis Papakostas,Konstantinos Tsiakas, Maria Dagioglou, VangelisKarkaletsis, and Rodney D Nielsen. 2015. Ground-ing the meaning of words through vision and inter-active gameplay. In IJCAI. pages 1895–1901.

Svetlin Penkov, Alejandro Bordallo, and SubramanianRamamoorthy. 2017. Physical symbol groundingand instance learning through demonstration and eyetracking .

Benjamin Rosman and Subramanian Ramamoorthy.2011. Learning spatial relationships between ob-jects. The International Journal of Robotics Re-search 30(11):1328–1342.

Constantin A Rothkopf, Dana H Ballard, and Mary MHayhoe. 2007. Task and context determine whereyou look. Journal of vision 7.

Nitish Srivastava and Ruslan R Salakhutdinov. 2012.Multimodal learning with deep boltzmann ma-chines. In Advances in neural information process-ing systems. pages 2222–2230.

Yuyin Sun, Liefeng Bo, and Dieter Fox. 2014. Learn-ing to identify new objects. In Robotics and Au-tomation (ICRA), 2014 IEEE International Confer-ence on. IEEE, pages 3165–3172.

Jesse Thomason, Jivko Sinapov, Maxwell Svetlik, Pe-ter Stone, and Raymond J Mooney. 2016. Learningmulti-modal grounded linguistic semantics by play-ing i spy. In Proceedings of the Twenty-Fifth inter-national joint conference on Artificial Intelligence(IJCAI).

Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider,Wojciech Zaremba, and Pieter Abbeel. 2017. Do-main randomization for transferring deep neural net-works from simulation to the real world. arXivpreprint arXiv:1703.06907 .

Paul Vogt. 2002. The physical symbol grounding prob-lem. Cognitive Systems Research 3(3):429–457.

Matthew R Walter, Sachithra Hemachandra, BiancaHomberg, Stefanie Tellex, and Seth Teller. 2014.A framework for learning semantic maps fromgrounded natural language descriptions. The Inter-national Journal of Robotics Research 33(9):1167–1190.

Konstantinos Zampogiannis, Yezhou Yang, CorneliaFermuller, and Yiannis Aloimonos. 2015. Learn-ing the spatial semantics of manipulation actionsthrough preposition grounding. In Robotics and Au-tomation (ICRA), 2015 IEEE International Confer-ence on. IEEE, pages 1389–1396.