Interactive Learning of Mappings from Visual Percepts to Actions … · 2018-09-10 · Interactive Learning of Mappings from Visual Percepts to Actions hp t,a t,r t+1,p t+1iof the

Interactive Learning of Mappings from Visual Percepts to Actions

Sebastien Jodogne1 [email protected] H. Piater [email protected]

Montefiore Institute (B28), University of Liege, B-4000 Liege, Belgium

Abstract

We introduce flexible algorithms that canautomatically learn mappings from imagesto actions by interacting with their environ-ment. They work by introducing an imageclassifier in front of a Reinforcement Learn-ing algorithm. The classifier partitions thevisual space according to the presence or ab-sence of highly informative local descriptors.The image classifier is incrementally refinedby selecting new local descriptors when per-ceptual aliasing is detected. Thus, we reducethe visual input domain down to a size man-ageable by Reinforcement Learning, permit-ting us to learn direct percept-to-action map-pings. Experimental results on a continuousvisual navigation task illustrate the applica-bility of the framework.

1. Introduction

Many reactive robotic tasks can be solved by “black-box” mappings from an input space to an outputspace, that directly connect percepts to the appro-priate actions through a given computational mech-anism. Such mappings are usually hard to derive byhand, especially when the input space contains images,since visual domains are high dimensional and noisy.This paper introduces algorithms suitable for buildingimage-to-action mappings using a fully automatic andflexible learning protocol.

Strong neuropsychological evidence suggests that hu-man beings learn to extract useful information fromvisual data in an interactive fashion, without any ex-ternal supervisor (Gibson & Spelke, 1983). By evaluat-ing the consequence of our actions on the environment,we learn to pay attention to visual cues that are be-

Appearing in Proceedings of the 22nd International Confer-ence on Machine Learning, Bonn, Germany, 2005. Copy-right 2005 by the author(s)/owner(s).

haviorally important for solving the task. This processis task-driven, since different tasks do not necessarilyneed to make the same distinctions.

One plausible framework to learn image-to-actionmappings in such an interactive protocol is Reinforce-ment Learning (RL) (Bertsekas & Tsitsiklis, 1996;Sutton & Barto, 1998). RL is a generic frameworkfor modeling the behavior of an agent that learnsa percept-to-action mapping through its interactionswith the environment. The agent is never told whataction it should take. Rather, when it does a goodor a bad action, it only receives from the environmenta reward or a punishment, the reinforcement signal .The major advantages of the RL protocol are that itis fully automatic, and that it imposes very weak con-straints on the environment. However, the price to payfor the flexibility of RL is poor performance when theinput space of the agent is high dimensional or noisy,which makes it unsuitable for building image-to-actionmappings.

However, to take the right decision, it is often sufficientfor the robotic agent to focus its attention on robustand highly informative patterns in the percepts. Thisis actually the basic postulate behind local-appearancemethods, that have had much success in Computer Vi-sion applications such as image matching, image re-trieval and object recognition (Lowe, 1999; Schmid &Mohr, 1997). They rely on the detection of discon-tinuities in the visual signal thanks to interest pointdetectors (Schmid et al., 2000). Similarities in imagesare thereafter identified using a local description of theneighborhood around the interest points (Mikolajczyk& Schmid, 2003): If two images share a sufficient num-ber of matching local descriptors, they are consideredas belonging to the same visual class. Matches are de-tected through a suitable metric (e.g., Mahalanobis’or Euclidean distance), as local descriptors are noth-ing else than vectors of real numbers. Such techniquesare at the same time powerful and flexible, as they

1Research fellow of the Belgian National Fund for Sci-entific Research (FNRS).


are robust to partial occlusions, and do not requiresegmentation or 3D models of objects.

It seems therefore promising to introduce, in front ofthe RL algorithm, an image classifier that translatesthe raw visual input to a visual class according to thepresence of some selected local descriptors at the in-terest points in the image. This preprocessing step isintended to reduce the size of the input domain, thusenhancing the rate of convergence, the generalizationcapabilities as well as the robustness of RL to noise invisual domains. The central difficulty is the dynamicselection of the discriminative local descriptors.

We propose the following algorithm, that will be calledReinforcement Learning of Visual Classes (RLVC).Initially, the agent just knows about one visual class,so that all images are mapped to this class. Of course,this introduces a kind of perceptual aliasing (White-head & Ballard, 1991): The optimal decisions cannotalways be made, since percepts requiring different re-actions are associated with the same class. Then, theagent isolates the aliased classes. Since there is noexternal supervisor, the agent can only rely on a sta-tistical analysis of the earned reinforcements. Oncesome aliased class has been detected, the agent selectsa new local descriptor that is distinctive, i.e. that bestexplains the observed variations in the reinforcementsfor the considered class. The extracted descriptor is fi-nally used to refine the classifier. New local descriptorsare learned until perceptual aliasing vanishes.

RLVC is quite different from classical RL techniquesfor discretizing the input space (Bertsekas & Tsitsik-lis, 1996), since previous methods assume that the in-teresting features (i.e., the local descriptors) or theirnumbers are known in advance. In fact, RLVC can bethought of as performing adaptive discretization of theperceptual space on the basis of an image processingalgorithm. It allows us to select a subset of highly rel-evant visual features in a fully closed-loop, task-drivenlearning process.

2. Theoretical Background

2.1. Reinforcement Learning

In RL, the environment is traditionally modeled as aMarkov Decision Process (MDP). An MDP is a tu-ple 〈S, A, T ,R〉, where S is a finite set of states, Ais a finite set of actions, T is a probabilistic tran-sition function from S × A to S, and R is a rein-forcement function from S × A to R. An MDP obeysthe following discrete-time dynamics: If at time t, theagent takes the action at while the environment lies ina state st, the agent perceives a numerical reinforce-

ment rt+1 = R(st, at), then reaches some state st+1

with probability T (st, at, st+1).

The definition of MDPs assumes the full observabilityof the state space, which means that the agent is ableto distinguish between the states of the environmentusing only its sensors. Let us define the perceptualspace P as the set of possible percepts that the sen-sors can return. In visual tasks, P is a set of images.So, from the point of view of the agent, an interac-tion with the environment is defined as a quadruple〈pt, at, rt+1, pt+1〉, where pt (resp. pt+1) is the perceptfurnished by its sensors in the presence of the state st

(resp. st+1).

A percept-to-action mapping is a fixed probabilisticfunction π : P 7→ A from percepts to actions. Apercept-to-action mapping tells the agent the proba-bility with which it should choose an action when facedwith some percept. In RL terminology, such a map-ping is called a stationary Markovian control policy .For an infinite sequence of interactions starting in astate st, the discounted return is Rt =

∑∞i=0 γirt+i+1,

where γ ∈ [0, 1[ is the discount factor that givesthe current value of the future reinforcements. TheMarkov Decision Problem for a given MDP is to findan optimal percept-to-action mapping that maximizesthe expected discounted return.

MDPs can be solved using Dynamic Programming(DP). Let us call Qπ(s, a), the function giving for eachstate s ∈ S and each action a ∈ A the expected dis-counted return obtained by starting from state s, tak-ing action a, and thereafter following the mapping π:Qπ(s, a) = Eπ {Rt | st = s, at = a}, where Eπ denotesthe expected value if the agent follows the mapping π.Let us also define the H transform from Q functionsto Q functions as:

(HQ)(s, a) = R(s, a) + γ∑s′∈S

T (s, a, s′) maxa′∈A

Q(s′, a′),

(1)for all s ∈ S and a ∈ A. All the optimal mappingsfor a given MDP share the same Q function, denotedQ∗, that always exists and that satisfies Bellman’s so-called optimality equation (Bellman, 1957):

HQ∗ = Q∗. (2)

Once Q∗ is known, an optimal deterministic percept-to-action mapping π∗ is easily derived by lettingπ∗(s) = argmaxa∈A Q∗(s, a) for each s ∈ S.

RL is a set of algorithmic methods for solving MarkovDecision Problems when the underlying MDP is un-known (Bertsekas & Tsitsiklis, 1996). The input ofRL algorithms is basically a sequence of interactions


〈pt, at, rt+1, pt+1〉 of the agent with its environment.RL techniques are often divided in two categories: (i)model-based methods that first build an estimate of theunderlying MDP (e.g., by computing the relative fre-quencies that appear in the sequence of interactions),and then use classical DP algorithms, and (ii) model-free methods such as SARSA, TD(λ), and the popularQ Learning (Watkins, 1989), that do not compute suchan estimate.

2.2. Local-Appearance Methods

Besides RL, the local-appearance paradigm in Com-puter Vision is the other technique required by ouralgorithms. Formally, let us call F the infinite set oflocal descriptors that can be generated by the cho-sen description method (Mikolajczyk & Schmid, 2003).The elements of F will be equivalently referred to asvisual features. Usually, F corresponds to Rn for somen ≥ 1. We assume the existence of a visual feature de-tector , that is a Boolean function F : P × F 7→ Btesting whether a given image exhibits a given localdescriptor at one of its interest points (Schmid et al.,2000). Any suitable metric can be used to test thesimilarity of two visual features, e.g. Mahalanobis’ orEuclidean distance.

3. Learning Visual Mappings

As discussed in the Introduction, we propose to insertan image classifier before the RL algorithm. The clas-sifier will be iteratively refined. At any step k, the clas-sifier, which will be denoted Ck, partitions the visualperceptual space P into a finite number mk of visualclasses {V k

1 , . . . , V kmk}. The classification is done ac-

cording to the local-appearance paradigm, by focusingthe attention of the agent on highly distinctive localdescriptors detected at interest points in the images.The goal of our visual learning algorithms is to iter-atively refine a sequence of classifiers by successivelyselecting new visual features, starting with a classifierC0 that maps all of its input images in a single visualclass V0,1.

3.1. Image Classifier

Because of this incremental process, a natural way toimplement the classifiers is to use binary decision trees.The visual classes correspond to the leaves of the trees.Each internal node is labeled by the visual feature, thepresence of which is to be tested in that node. Toclassify an image, the system starts at the root node,then progresses down the tree according to the resultof the feature detector F for each visual feature foundduring the descent, until reaching a leaf. To refine a

percepts

Image Classifier Reinforcement Learningdetected visual class

reinforcements

aliased visual class

actions

Figure 1. The information flow in RLVC.

visual class using a feature, it is sufficient to replacethe leaf corresponding to this class by an internal nodetesting the presence or absence of this feature.

3.2. Learning Architecture

We are interested in computing an image-to-actionmapping π : P 7→ A. At step k, given a visual perceptp ∈ P , rather than applying RL algorithms directly onthe raw values of the pixels of p, the RL algorithm issupplied with the output of Ck(p). The resulting archi-tecture will be referred to as Reinforcement Learningof Visual Classes (RLVC), and is depicted in Figure 1.

RLVC consists of two interleaved, different learningprocesses: (i) the selection of the distinctive visual fea-tures, and (ii) the Reinforcement Learning of a map-ping from the visual classes to the actions. The twomain challenges in RLVC are therefore (i) to identifywhen a refinement of the image classifier is needed,and (ii) to select visual features that are suitable torefine the classifier.

Until the agent has learned the visual classes requiredto complete its task, visual percepts needing differentreactions may be mapped to the same class. There-fore, the right decisions cannot always be made bythe embedded RL algorithm on the basis of its inputs.From the viewpoint of the embedded RL algorithm,the input space is only partially observable (cf. Sec-tion 2.1). This phenomenon is known as the percep-tual aliasing (or hidden state) problem (Whitehead &Ballard, 1991). Hence, detecting when a refinementis needed is actually equivalent to detecting in whichvisual classes perceptual aliasing occurs. We discusshow this detection is achieved in RLVC in the follow-ing section.

3.3. Detecting the Aliased Visual Classes

3.3.1. Projecting MDPs through a Classifier

Formally, any image classifier Ck converts a sequenceof N interactions 〈pt, at, rt+1, pt+1〉 to a mapped se-quence of N quadruples 〈Ck(pt), at, rt+1, Ck(pt+1)〉.Let use define the mapped MDP Mk as the MDP


〈Sk, A, Tk,Rk〉 obtained from the mapped sequence,where Sk is the set of visual classes that are known toCk, and where Tk and Rk have been computed usingthe relative frequencies in the mapped sequence.

Consider two visual classes V, V ′ ∈ {V k1 , . . . , V k

mk} and

one action a ∈ A. We define the following functions:

• δt(V, a) equals 1 if Ck(pt) = V and at = a, and 0otherwise;

• δt(V, a, V ′) equals 1 if Ck(pt) = V , at = a andCk(pt+1) = V ′, and 0 otherwise;

• η(V, a) is the number of t such that δt(V, a) = 1.

Using this notation, we can write:

• Sk = {V k1 , . . . , V k

mk};

• Tk(V, a, V ′) =∑N

t=1 δt(V, a, V ′)/η(V, a);

• Rk(V, a) =∑N

t=1 rtδt(V, a)/η(V, a).

3.3.2. Optimal Q Function for a Mapped MDP

Each mapped MDP Mk induces an optimal Q func-tion on the domain Sk × A that will be denoted Q′∗

k .Computing Q′∗

k can be difficult: In general, there mayexist no MDP defined on the state space Sk and onthe action space A that can generate a given mappedsequence, since the latter is not necessarily Marko-vian anymore. So, if some RL algorithm is run on themapped sequence, it might not converge toward Q′∗

k ,or not even converge at all. However, when applied ona mapped sequence, any model-based RL method (cf.Section 2.1) can be used to compute Q′∗

k ifMk is usedas the underlying model. Under some conditions, QLearning also converges to the optimal Q function ofthe mapped MDP (Singh et al., 1995).

In turn, the function Q′∗k induces another Q func-

tion on the initial domain S × A through the rela-tion Q∗

k(s, a) = Q′∗k (Ck(ps), a), where ps is an arbi-

trary percept that s can generate. In the absence ofaliasing, the agent could perform optimally, and Q∗

k

would correspond to Q∗, according to Bellman’s theo-rem that states the uniqueness of the optimal Q func-tion (cf. Section 2.1). By Equation (2), the matrixBk = HQ∗

k −Q∗k is therefore a measure of the aliasing

induced by the image classifier Ck. In RL terminology,Bk is Bellman’s residual of the function Q∗

k. The basicidea behind RLVC is to refine the states that have anon-zero Bellman’s residual.

3.3.3. Measuring Aliasing

Unfortunately, in practice, the H transform is un-known, because the agent does not have direct access

to the state space of the MDP modeling the environ-ment. We propose a different approach, by assum-ing temporarily that the transition function T of theenvironment is deterministic. In this case, Bellman’sresidual of Q∗

k can be rewritten as:

Bk(s, a) = R(s, a) + γ maxa′∈A

Q∗k(T (s, a), a′)−Q∗

k(s, a)

= R(s, a)+γ maxa′∈A

Q′∗k

(Ck

(pT (s,a)

), a′

)−Q′∗

k (Ck(ps), a)

(3)

Consider an action a ∈ A and two states s, s′ ∈ S ofthe environment. Interestingly, Equation (3) allows tocompute Bellman’s residual for the pair (s, a), if wehave an interaction 〈pt, at, rt+1, pt+1〉 such that pt hasbeen generated by s, pt+1 has been generated by s′,and at corresponds to a. Indeed, in the latter case,Bk(s, a) equals:

∆t = rt+1 + γ maxa′∈A

Q′∗k (Ck(pt+1), a′)−Q′∗

k (Ck(pt), at).

(4)

According to the previous discussion, the residuals ∆t

measure the perceptual aliasing induced by Ck.2 Anonzero ∆t indicates the presence of perceptual alias-ing in the visual class Vt = Ck(pt) with respect to ac-tion at. Our criterion for detecting the aliased classesconsists in computing the Q′∗

k function, then in sweep-ing again all the 〈pt, at, rt+1, pt+1〉 to identify nonzero∆t.

3.4. Extracting Distinctive Visual Features

Once aliasing has been detected in some visual classV ∈ Sk with respect to an action a, we need to discovera new local descriptor that best explains the variationsin the set of ∆t values corresponding to V and a. Thisis a classification problem, for which we suggest anadaptation of a popular splitting rule used with deci-sion trees (Quinlan, 1993).

Let us denote T the set of time stamps t such thatCk(pt) = V and at = a. Each threshold c ∈ R inducesa binary partition {T c

1 , T c2} of the set T , where:

T c1 = {t ∈ T | ∆t ≤ c},

T c2 = {t ∈ T | ∆t > c}.

Similarly, any visual feature f ∈ F splits T into twosubsets:

T f1 = {t ∈ T | F(pt, f)},

T f2 = {t ∈ T | ¬F(pt, f)}.

2It is worth noticing that αt∆t corresponds to the up-dates that would be applied by Q Learning (Watkins,1989), where αt is known as the learning rate at time t.


Ideally, we would like to identify a visual feature f thatsplits T the same way a threshold c would.

Therefore, for each cutpoint in the sorted sequence of∆t such that t ∈ T , we select the visual feature thatmaximizes a given information-theoretic score for thepartition of images induced by the feature detector F .This is done by iterating over the local descriptors thatdescribe the neighborhood of all the interest points ofthe images in {pt | t ∈ T}. Finally, the visual featurethat has the maximal score among all the extractedfeatures is used to refine the classifier.

Our algorithms assume that the transition function Tbehaves deterministically to be able to directly tar-get Bellman’s residuals. Of course, interesting envi-ronments are in general non-deterministic, which gen-erates variations in Bellman’s residuals that are nota consequence of perceptual aliasing. RLVC can bemade somewhat robust to such a variability by intro-ducing a statistical hypothesis test: For each candidatesplit, a χ2 test is used to decide whether it is signifi-cantly different from a random split. This approach isinspired from decision tree pruning.

3.5. Algorithm

Putting it all together, RLVC can be described as:

1. Begin with a counter k ← 0 and a classifier Ck thatmaps all the percepts to the same visual class.

2. Construct an optimal function Q′∗k for the clas-

sification induced by Ck, e.g. using model-basedmethods. Record in a database the interac-tions 〈pt, at, rt+1, pt+1〉 encountered during thisRL process.

3. Let Ck+1 ← Ck. For each (i, a) ∈ {1, . . . ,mk}×A:

(a) Select all the interactions 〈pt, at, rt+1, pt+1〉in the database such that Ck(pt) = V k

i andat = a.

(b) For each of them, compute the value:

ξt = rt+1 + γ maxa′∈A

Q′∗k (pt+1, a

′).

Note that since Q′∗k (Ck(pt), at) = Q′∗

k (V ki , a)

is a constant c for all the selected interac-tions, ξt is just the ∆t from Equation (4),offset by c.

(c) If some of the ξt are different from c, selecta distinctive feature that best explains thevariations in the set of ξt. If the hypothe-sis test succeeds, refine the classifier Ck+1 bysplitting the class V k

i according to the newlyselected feature.

4. If Ck 6= Ck+1, let k ← k + 1 and go to step 2.

At the end of the algorithm, the classifier Ck is assumedfree of perceptual aliasing, and can be subsequentlyused to control the system.

4. Experiments on a Navigation Task

We have evaluated our system on an abstract task thatclosely parallels a real-world scenario while avoidingany unnecessary complexity. RLVC has succeeded atsolving the continuous, noisy visual navigation taskdepicted in Figure 2. The goal of the agent is to reachas fast as possible one of the two exits of the maze.The set of possible locations is continuous. At eachlocation, the agent has four possible actions: Go up,right, down, or left. Every move is altered by a Gaus-sian noise, the standard deviation of which is 2% thesize of the maze. Glass walls are present in the maze.Whenever a move would take the agent into a wall oroutside the maze, its location is not changed.

The agent earns a reward of 100 when an exit isreached. Any other move, including the forbiddenones, generates zero reinforcement. When the agentsucceeds in escaping the maze, it arrives in a terminalstate in which every move gives rise to a zero reinforce-ment. In this task, γ was set to 0.9. Note that theagent is faced with the delayed reward problem, andthat it must take the distance to the two exits intoconsideration for choosing the most attractive one.

The maze has a ground carpeted with a color image of1280× 1280 pixels that is a montage of pictures fromthe COIL-100 database3. The agent does not have di-rect access to its (x, y) position in the maze. Rather,its sensors take a picture of a surrounding portion ofthe ground. This portion is larger than the blank ar-eas, which makes the input space fully observable. Im-portantly, the glass walls are transparent, so that thesensors also return the portions of the tapestry thatare behind them. Therefore, there is no way for theagent to directly locate the walls. It is obliged to iden-tify them as the regions of the maze in which an actiondoes not change its location.

In this experiment, we have used color differential in-variants as visual features (Gouet & Boujemaa, 2001).The entire tapestry includes 2298 different visual fea-tures. RLVC selected 200 features, corresponding to aratio of 9% of the entire set of possible features. Thecomputation stopped when k reached 84, which took35 minutes on a 2.4GHz Pentium IV using databasesof 10,000 interactions. 205 visual classes were identi-fied. This is a small number, compared to the numberof perceptual classes that would be generated by a dis-

3http://www.cs.columbia.edu/CAVE/coil-100.html


Figure 2. A continuous, noisy navigation task. The exits ofthe maze are indicated by boxes with a cross. Walls of glassare identified by solid lines. The agent is depicted at thecenter of the figure. Each one of the four possible moves isrepresented by an arrow, the length of which correspondsto the resulting move. The sensors return a picture thatcorresponds to the dashed portion of the image.

Figure 3. The resulting deterministic image-to-action map-ping π∗ = argmaxa∈A Q∗

k(s, a), sampled at regularly-spaced points. It manages to choose the correct actionat each location.

(a) (b)

Figure 4. (a) The optimal value function, when the agenthas direct access to its (x, y) position in the maze and whenthe set of possible locations is discretized into a 50×50 grid.The brighter the location, the greater its value. (b) Theoptimal value function obtained by RLVC.

cretization of the maze when the agent knows its (x, y)position. For example, a reasonably sized 20× 20 gridleads to 400 perceptual classes.

The optimal, deterministic image-to-action mappingthat results from the last obtained image classifier Ckis shown in Figure 3. In RL, it is generally instruc-tive to study the optimal value function V ∗(s) of eachstate, that corresponds to the expected discounted re-turn when the agent always chooses the optimal ac-tion in each state, i.e. V ∗(s) = maxa∈A Q∗(s, a).Figure 4 compares the optimal value function of thediscretized problem with the one obtained throughRLVC. The similarity between the two pictures in-dicates the soundness of our approach. Importantly,RLVC operates with neither pre-treatment, nor humanintervention. The agent is initially not aware of whichvisual features are important for its task. Moreover,the interest of selecting descriptors is clear in this ap-plication: A direct, tabular representation of the Qfunction considering all the Boolean combinations offeatures would have 22298 × 4 cells.

The behavior of RLVC on real-word images hasalso been investigated. The navigation rules werekept identical, but the tapestry was replaced by apanoramic photography of 3041× 384 pixels of a sub-way station, as depicted in Figure 5. RLVC took 101iterations to compute the mapping at the right of Fig-ure 5. The computation time was 159 minutes on a2.4GHz Pentium IV using databases of 10,000 interac-tions. 144 distinct visual features were selected amonga set of 3739 possible ones, generating a set of 149visual classes. Here again, the resulting classifier isfine enough to obtain a nearly optimal image-to-actionmapping for the task. Our framework therefore seemsapplicable to realistic Computer Vision applications.


(a) (b)

Figure 5. (a) A navigation task with a real-world image,using the same conventions than Figure 2. (b) The de-terministic image-to-action mapping computed by RLVC.

5. Related Work

The idea of building a decision tree that tests the pres-ence of Boolean features to partition a large state spaceinto a piecewise constant value function goes back tothe G Algorithm (Chapman & Kaelbling, 1991). Mc-Callum’s U Tree algorithm builds upon this idea bycombining this so-called “selective attention” mech-anism with a short-term memory that enables thelearning agent to deal with partially observable en-vironments (McCallum, 1996). Decision trees havealso been used to deal with continuous input spacesin the context of Continuous U Tree (Uther & Veloso,1998) and Variable Resolution Grids (Munos & Moore,2002). However, our work appears to be the first to re-late Reinforcement Learning with the local-appearanceparadigm in Computer Vision. The use of visual fea-tures in fact appears to be a natural and powerful toolfor solving visual, reactive tasks.

A large number of criteria for deciding the presence ofaliasing have been proposed in the literature (White-head & Ballard, 1991; Chrisman, 1992; McCallum,1996; Munos & Moore, 2002). The aliasing criterionused in RLVC is defined independently of any fixed RLalgorithm, and is very close to that used by ContinuousU Tree (Uther & Veloso, 1998), with the major excep-tion that RLVC deals with Boolean features, whereasContinuous U Tree works in a continuous input space.

6. Conclusions

We have introduced Reinforcement Learning of VisualClasses (RLVC). RLVC is designed to learn mappingsthat directly connect visual stimuli to output actionsthat are optimal for the surrounding environment. Thelearning process is closed-loop and flexible. It consistsin taking lessons from interactions with the environ-ment, which is similar to the way living beings learnto solve everyday tasks. RLVC focuses the attentionof an embedded Reinforcement Learning algorithm onhighly informative and robust parts of the inputs bytesting the presence or absence of local descriptors atthe interest points of the input images.

The relevant visual features are incrementally selectedin a sequence of attempts to remove perceptual alias-ing. Importantly, this selection is task-driven: Differ-ent tasks will not necessarily lead to the selection ofthe same subset of features. This is similar to humanvisual learning, for which there is strong evidence thatnew visual classes are learned when the task requiresit (Schyns & Rodet, 1997). In this sense, the paradigmof RLVC can also be motivated from a neuropsycholog-ical point of view, and not only as an ad-hoc machine


learning algorithm.

The area of applications is wide, since nowadaysrobotic agents are often equipped with CCD sensors.Our long-term goal would be to construct a roboticsystem that autonomously learns to structure its vi-sual system to solve a reactive task, such as graspingobjects (Coelho et al., 2001). RLVC could also be po-tentially be applied to Human-Computer Interaction,as the actions need not be physical actions.

However, many questions are still open. For example,it is unclear how RLVC could be applied on continu-ous action spaces, which is required for the exampleof grasping. It would also be interesting to add post-pruning operations to get rid of selected features thatsubsequently prove useless for the task, and that gen-erate overfitting effects. Finally, the efficacy of RLVCclearly depends on the discriminative power of the vi-sual features. If this power is insufficient, the algo-rithm will not be able to completely remove the alias-ing, which will produce a sub-optimal control law. Forfuture research, we therefore suggest the combinationof RLVC with techniques for disambiguating betweenaliased percepts using a short-term memory (McCal-lum, 1996), or for building higher-level geometricalcombinations of visual features that are more robustto noise (Piater, 2001; Scalzo & Piater, 2005).

References

Bellman, R. (1957). Dynamic programming. PrincetonUniversity Press.

Bertsekas, D., & Tsitsiklis, J. (1996). Neuro-dynamicprogramming. Athena Scientific.

Chapman, D., & Kaelbling, L. (1991). Input gener-alization in delayed reinforcement learning: An al-gorithm and performance comparisons. Proc. of the12th International Joint Conference on Artificial In-telligence (IJCAI) (pp. 726–731). Sydney.

Chrisman, L. (1992). Reinforcement learning withperceptual aliasing: The perceptual distinctions ap-proach. National Conference on Artificial Intelli-gence (pp. 183–188).

Coelho, J., Piater, J., & Grupen, R. (2001). Develop-ing haptic and visual perceptual categories for reach-ing and grasping with a humanoid robot. Roboticsand Autonomous Systems, 37, 195–218.

Gibson, E., & Spelke, E. (1983). The development ofperception. Handbook of child psychology vol. iii:Cognitive development, chapter 1, 2–76. Wiley.

Gouet, V., & Boujemaa, N. (2001). Object-basedqueries using color points of interest. IEEE Work-

shop on Content-Based Access of Image and VideoLibraries (pp. 30–36). Kauai (HI, USA).

Lowe, D. (1999). Object recognition from local scale-invariant features. International Conference onComputer Vision (pp. 1150–1157). Corfu, Greece.

McCallum, R. (1996). Reinforcement learning with se-lective perception and hidden state. Doctoral disser-tation, University of Rochester, New York.

Mikolajczyk, K., & Schmid, C. (2003). A performanceevaluation of local descriptors. IEEE Conference onComputer Vision and Pattern Recognition (pp. 257–263). Madison (WI, USA).

Munos, R., & Moore, A. (2002). Variable resolutiondiscretization in optimal control. Machine Learning,49, 291–323.

Piater, J. (2001). Visual feature learning. Doctoraldissertation, University of Massachusetts, ComputerScience Department, Amherst (MA, USA).

Quinlan, J. (1993). C4.5: Programs for machine learn-ing. Morgan Kaufmann Publishers Inc.

Scalzo, F., & Piater, J. (2005). Task-driven learning ofspatial combinations of visual features. Proc. of theIEEE Workshop on Learning in Computer Visionand Pattern Recognition. San Diego (CA, USA).

Schmid, C., & Mohr, R. (1997). Local greyvalue in-variants for image retrieval. IEEE Transactions onPattern Analysis and Machine Intelligence, 19, 530–535.

Schmid, C., Mohr, R., & Bauckhage, C. (2000). Evalu-ation of interest point detectors. International Jour-nal of Computer Vision, 37, 151–172.

Schyns, P., & Rodet, L. (1997). Categorization createsfunctional features. Journ. of Experimental Psychol-ogy: Learning, Memory and Cognition, 23, 681–696.

Singh, S., Jaakkola, T., & Jordan, M. (1995). Rein-forcement learning with soft state aggregation. Ad-vances in Neural Information Processing Systems(pp. 361–368). MIT Press.

Sutton, R., & Barto, A. (1998). Reinforcement learn-ing, an introduction. MIT Press.

Uther, W. T. B., & Veloso, M. M. (1998). Tree baseddiscretization for continuous state space reinforce-ment learning. Proc. of the 15th National Con-ference on Artificial Intelligence (AAAI) (pp. 769–774). Madison (WI, USA).

Watkins, C. (1989). Learning from delayed rewards.Doctoral dissertation, King’s College, Cambridge.

Whitehead, S., & Ballard, D. (1991). Learning to per-ceive and act by trial and error. Machine Learning,7, 45–83.

Documents

Interactive Learning of Mappings from Visual Percepts to Actions … · 2018-09-10 · Interactive Learning of Mappings from Visual Percepts to Actions hp t,a t,r t+1,p t+1iof the