Encyclopedia of Computational Neuroscience || Optic Flow Processing

Optic Flow Processing

Holger G. Krapp*Department of Bioengineering, Imperial College London, London, UK

Synonyms

Directional motion; Matched filters; Motion parallax; Optic flow fields; Optical flow; Relativemotion; Velocity vector fields

Definition

Optic flow describes the geometrical projection of relative motion between the visual environmentand a moving optical system. It is commonly formalized by a distribution of local velocity vectors,observed at many different positions within the optical system’s visual field. Together, the localvelocity vectors constitute a pattern of coherent image motion called an optic flow field. Thedirection and magnitude of the velocity vectors depend on the instantaneous self-motion of theoptical system which may be decomposed into its translation and rotation components. Themagnitude of translation-induced contribution to a given velocity vector depends on the distancebetween the optical system and objects within the visual field, while the rotation-induced contribu-tion is distance invariant. Given its properties, optic flow contains a significant amount of informa-tion about the self-motion and relative distance information that can be exploited for guidance andcontrol tasks.

Detailed Description

The intimate relationship between self-motion and specific patterns of image motion was noticedearly on by J. J. Gibson in the mid-twentieth century. In his book The Perception of the Visual World(1950) and later in The Ecological Approach to Visual Perception (1979), Gibson mainly focused onqualitative properties of optic flow. But it was undoubtedly his work that provided the conceptualfoundation for later quantitative approaches on how optic flow may be exploited to estimate self-motion, which were mainly driven by computer vision and visual neuroscience. The first mathe-matical descriptions of optic flow fields became available in the mid-1970s (e.g., Nakayama andLoomis 1974; Koenderink and van Doorn 1975) and 1980s (e.g., Horn and Schunck 1981). Fromthen on, researchers in computer vision, natural sciences, and engineering started to take advantageof the formal relationship between the self-motion components of optical systems – either eyes orcameras – and the resulting optic flow fields (e.g., Barron et al 1994).

Movements in SpaceAnymovement in space may be described as a combination of translation and rotation, convenientlyparameterized as the projections of their components onto the cardinal axes of a Cartesian coordinate

*Email: [email protected]

Encyclopedia of Computational NeuroscienceDOI 10.1007/978-1-4614-7320-6_332-1# Springer Science+Business Media New York 2014

Page 1 of 22

http://link.springer.com/SpringerLink:ChapterTarget








system that has its origin in the center of the optical system. The components are often referred to asthrust, slideslip, and lift in terms of translations along the x-, y-, and z-axis, respectively, whilerotations around these axes are called roll, pitch, and yaw (Fig. 1). Together, the projections onto thecardinal axes define the translation vector, T ¼ [Tx, Ty, Tz]

T, and the rotation vector, R ¼ [Rx, Ry,Rz]

T, which results in 6 degrees of freedom, where the superscript T denotes the transpose of a vector.

Local Image Shifts Quantified as Parallax VectorsFrom the center of the coordinate system, the lines of sight to any given points in a spherical visualfield may be described by direction vectors di, where the index i (¼ 1, 2, . . ., N) refers to the numberof different points observed. During self-motion, the observed points in the visual field change theirlocation over time relative to the optical system depending on where they are observed and what theself-motion components TandR of the optical system are. The resulting image shifts of the observedpoints are equivalent to the temporal derivative of the direction vectors, di, and are described as localparallax vectors, pi, according to a basic vector equation. Using the notation proposed byKoenderink and van Doorn (1987), each local parallax vector, pi, is defined as

pi ¼@di@t

¼ � T� T � dið Þdið Þ=Di � R� dið Þ, (1)

where T denotes the translation vector,Di gives the distances of the observed points along directionsdi, and R is the rotation vector. The expression (T � di) on the right-hand side of Eq. 1 gives the dot(scalar) product between the translation vector and the direction vectors di, and the operator �

Fig. 1 Self-motion components in a Cartesian coordinate system. (a): Self-motion components T and R may bedescribed in terms of 6 degrees of freedom. Translation components along the x-, y-, and z-axis are called thrust, sideslip,and lift, respectively. Rotation components along the same axes are called roll, pitch, and yaw. (b) The variables yx, yy,and yz describe the angles between the direction, di, upon which the parallax vector, pi, is observed, and the cardinal axesof the coordinate system. We will need those angles later during the discussion of local scaling factors applied to theobservation directions, di. In this case, the parallax vector has been observed during a translation, T, along the positivez-axis. Note that the angles yi, (i ¼ x, y, z), are different from the angleY between the observation directions, di, and theself-motion vectors TorR. In the example given here, where the self-motion vector, T, corresponds to the positive z-axis,the angle Y ¼ yz. As the visual field is spherical, we may represent local image shifts, pi, as tangent vectors on thesurface of a unit sphere. The directions ofR, T, and di may be described in terms of two angles, the horizontal azimuth, a,and the vertical elevation, b. The azimuth ranges from �180� � a � 180�, with a ¼ 0� corresponding to the positivex-axis in Cartesian coordinates and the elevation ranging from 0� � b � 180�, where b ¼ 0� corresponds to a directionalong the positive z-axis; and b ¼ 90� describes the equator of the visual field


Page 2 of 22

symbolizes the cross product, here calculated between the rotation vector and the observationdirections (R � di).

To gain some intuition on how self-motion components and parallax vectors are related to oneanother, we should first consider the translation- and rotation-dependent contributions on the right-hand side of Eq. 1 separately. Let us assume the optical system undergoes a pure unit translationalong the positive x-axis, i.e., T ¼ [1 0 0]T. The resulting image shift is observed in directiondi ¼ [0 1 0]T, which corresponds to the positive y-axis and thus a lateral viewing direction at theequator of the visual field (Fig. 2a). In this case, T and di are orthogonal to one another and the dotproduct (T � di) becomes zero. The result is a parallax vector, pi, that is equal to –T which may beplotted as a tangent vector in a spherical representation of the visual field with its basis originating atdi (Fig. 2a). The translation-dependent term in Eq. 1 states that pi is scaled by the distance, Di,between the center of the optical system and the point in the visual field along direction di. We caneasily relate this distance dependence to our everyday experience. When sitting on a train or in a carmoving along a straight line, close visual objects rush past very quickly, while trees or hills in thedistance seem to hardly move at all.

In contrast to the translation-induced contribution to the parallax vector, the rotation-inducedcontribution does not depend on distance, as is clear from the second term on the right-hand side ofEq. 1. During a pure counterclockwise rotation of the optical system around the positive x-axis,R ¼ [1 0 0]T, we chose the same observation direction as before, di ¼ [0 1 0]T, and calculate thecross product, �(R � di), to obtain the rotation-induced local parallax vector, pi (Fig. 2b). Like ina case of translation-induced relative motion, the resulting pi plotted as a tangent vector on the spherepoints in the direction opposite to the change of di over time (Fig. 2b).

In the next step, we may apply Eq. 1 to compute optic flow fields composed of parallax vectorsmeasured at N different directions, di, in the visual field. Again we consider the global properties oftranslation- and rotation-induced optic flow fields separately. This time we compute a pure lifttranslation, T ¼ [0 0 1]T, and again a pure roll rotation, R ¼ [1 0 0]T. For this purpose, we set inEq. 1 the rotation and translation vectors to zero, respectively. To keep matters simple, we assumea uniform distance distribution, i.e., Di ¼ 1, for all directions, di, when calculating the translation-induced component of the individual parallax vectors.

Figure 3a, b shows the global structure of such translational and rotational optic flow fields – forclarity all individual parallax vectors were multiplied by a scaling factor of 0.1. The flow fields were

Fig. 2 Local motion parallax vectors due to translation and rotation. (a) Translation-induced local parallax vector,pi, observed at direction, di, given a thrust translation, T, along the positive x-axis. The distance to objects in the visualenvironment, Di, was set to unity. (b) Rotation-induced local parallax vector, pi, observed at direction, di, givena counterclockwise roll rotation, R, around the positive x-axis. For further explanation, see text


Page 3 of 22

calculated for a lift translation (a) and roll rotation (b), respectively. In both cases, the directionaldistribution of the local parallax vectors is determined by the self-motion vectors R and T. Theorientation of R and T defines the location of the two singularities within the optic flow fields wherethe magnitude of the local image shifts, or parallax vectors pi, becomes zero. The singularities areseparated by an angle of 180�. Right in between the two singularities, i.e., 90� off the orientation ofRand T, the projection of the relative motion becomes maximum, which is easily verified by revisitingthe rotation- and translation-dependent terms in Eq. 1. At any other location, the magnitude of theparallax vectors follows the function |pi| ¼ sin(Y), where Y is the angle between the self-motioncomponent and the observation direction di. In translation-induced optic flow fields, the twosingularities are often referred to as focus of expansion (FOE), located in the direction of translationT, and the focus of contraction (FOC) in the opposite direction �T. Singularities within a rotation-induced flow field coincide with the orientation of the axis of rotation (AOR), defined by R and�R.

A fundamental difference in the global structure of translation- and rotation-induced optic flowconcerns the respective directional distribution of the parallax vectors. While all image shifts piwithin a translation flow field are aligned along great circles connecting the FOE with theFOC, image shifts in a rotation flow field are oriented along parallel circles centered on the AOR(Fig. 3a, b). Another general feature of translational and rotational parallax vectors is that at anygiven direction di, they are orthogonally oriented to one another when the self-motion axes of

Fig. 3 Global structure of optic flow fields. (a) Optic flow field generated during lift along the positive z-axis. Thedistance to objects in the visual environment, Di, was set to unity. FOE and FOC give the focus of expansion andcontraction, respectively, which are defined by the translation vector, T, and indicate the two singularities in the flowfield. (b) Optic flow field generated during roll rotation, R, around the positive x-axis. AOR stands for axis of rotationwhich defines the two singularities in the flow field. To visualize the entire global flow field, it may be transformed intothe two-dimensional plane using a cylindrical projection. For better intuition when presenting the flow field, theelevation, b, has been redefined to range from +90 � 0 > �90�. This results in the direction a ¼ b ¼ 0� coincidingwith the positive x-axis, i.e., the location in the equator of the visual field in front of the observer (f). Letters d, v, andc stand for dorsal, ventral, and caudal, respectively. Note that in the equatorial region in the right lateral hemisphere (grayarea), the parallax vectors point downward in both flow fields; they are ambiguous with respect to the self-motioncomponent that caused them. For further explanation, see text (Modified from Krapp and Hengstenberg (1996))


Page 4 of 22

translation and rotation are co-aligned. This means that they can be most easily distinguished, whichis a helpful feature when extracting translation- and rotation-induced self-motion components.

For various reasons, it has proven useful to transform the spherical representation of optic flowinto a 2-dimensional cylindrical projection. Such dimensionality reduction naturally comes with anincreasing transformation error toward the pole regions of the sphere where the area is overrepre-sented by a factor of 1/sin(b), with b being the elevation (cf. legend Fig. 2). A cylindrical projection,however, allows us to directly visualize the entire spherical optic flow field in a single figure asa function of azimuth, a, and elevation, b (Fig. 3a, b, right panels).

Estimating Self-Motion Parameters from Noisy Optic Flow FieldsSo far we considered the properties of optic flow assuming that all parameters were perfectly wellknown which enabled us to compute translation- and rotation-induced flow fields. Opticalsystems – or biological visual systems – generally have to solve the inverse problem. To be usefulfor guidance and control of technical systems or, equivalently, the control of posture and gaze inanimals, the task is a continuous estimation of R and T which may be used to generate feedbacksignals driving the appropriate motor plants. I should emphasize that both technical and biologicalsystems have also access to inertial measurements of state changes using accelerometers andgyroscopes or equivalent mechanoreceptive systems. But inertial sensor systems – although beingable to initiate immediate stabilizing action – do not necessarily provide information as to whether ornot compensatory motor activity has yielded the desired effect. Another problem concerns thedynamic range of the sensors. Some inertial measurement sensors and many mechanoreceptorsystems are not particularly sensitive in the low end of their dynamic range resulting in a low gainfor sensing slow drifts or angular rotations. Vision, on the other hand, covers the low-end dynamicrange but is notoriously slow due to long delay times and heavy computational overheads whenprocessing local image shifts. Biological visual systems also have limited bandwidth for theprocessing of image shifts due to high linear and angular velocities. We will return to the topic ofdynamic range fractioning further below.

Why exploit optic flow to estimate self-motion? For instance, one of the components in humangaze control is driven by the vestibuloocular reflex (VOR) where the oculomotor pathway receivesinputs from the fast vestibular system that measures angular accelerations of the head (e.g., Angelakiand Cullen 2008). Ideally, small head rotations should be quantitatively compensated for by motoraction of the eye muscles counterrotating the eye balls in their orbits. But the only way of knowingwhether the counterrotation fully compensated for the head or body movement is to measure theremaining relative motion between the eyes and the visual surroundings. Adjusting the motorcommands sent to the eye muscles requires an estimate of the rotational optic flow componentanalyzed by the motion vision system. Another vision-dependent mechanism enabling the control ofposture and gaze is the oculomotor reflex. To support an equilibrium posture and a stable gaze alsorequires an estimate of self-motion parameters R and T. In general, whenever animals – includinghumans – are endowed with vision, they will have developed the analysis of optic flow asa mechanism that complements mechanoreception for the control of their motor actions.

In the following, we will discuss the theoretical limits of estimating R and T based on noisy opticflow fields. Noise, in this context, may be introduced by internal or external factors, for instance, asa result of imperfections in measurements of local parallax vectors or uncertainties in the distancedistribution within the cluttered visual surroundings (see below). As a starting point, it will be usefulto revisit the work by Koenderink and van Doorn (1987) who proposed an iterative least squaresolution for the recovery of R and T, also known as the KvD algorithm. As Eq. 1 states, we aredealing with an expression that contains one known quantity on the left-hand side – at least under the


Page 5 of 22

assumption that the optical system is able to measure the parallax vectors, pi. On the right-hand side,the only parameter known is the distribution of directions, di, at which the parallax vectors areobserved. The unknown parameters on the right-hand side of Eq. 1 are the distances to objects in thevisual field, Di, as well as T and R. This clearly is an ill-posed problem due to the fact that thecontribution to pi induced by T is under-defined. The same pi may result from a slow translation atclose distance to visual objects or from a fast translation in an environment where the objects are atlarge distances. Although the contribution to pi based on R is sufficiently well defined, R and Tcannot be estimated based on a single parallax vector induced during a combination of translationand rotation.

Koenderink and van Doorn (1987) tackled this problem by introducing a local scaling factor mi, or“reduced nearness,” defined as mi ¼ T/Di, where T denotes the speed of the translation vector Talong the direction of the unit vector t according to the relationship T ¼ tT. By introducing reducednearness, the KvD turns an ill-posed problem into a tractable one. It allows for the retrieval of thedirection of translation vector, t, rotation vector, R, and the local reduced nearness, mi, given theparallax vectors, pi, are measured at N � 5 different directions, di. The introduction of reducednearness modifies Eq. 1 to

pi ¼ �mi t� t � dið Þdið Þ � R� dið Þ: (2)

The KvD is based on minimizing the squared error, E, between a set of measured parallax vectors,pi, and a set of parallax vectors, pi0, calculated from fitted parameters t0, R0, and mi0, under theconstraint that |t 0| ¼ 1. The error E is formalized by

E ¼ 1=NX

pi � pi0j jj jð Þ2 þ l t0 � t0ð Þ � 1ð Þ, (3)

where l is the Lagrange multiplier ensuring the constraint |t0| ¼ 1 is met. After working out thepartial derivatives @/@R0, @/@t 0, and @/@mi0 for the right-hand side of Eq. 3, three equations areobtained which can be used in an iterative procedure to estimate t0, R0, and mi0:

t 0 ¼ �n I� m0idi � m0

idi� �� 1 mi

0pih i þ R0 � mi0dih ið Þ½

n o, (4a)

R0 ¼ I� di � dih i½ �1 pi � dih i þ t 0 � mi0dih ið Þ½ , (4b)

mi0 ¼ �t 0 � pi � di � R0ð Þð Þ= 1� t 0 � dið Þ2

� �, (4c)

where I is the identity matrix, n is a factor forcing |t0| ¼ 1,� symbolizes the dyadic (tensor) product,and arguments in pointed brackets, h . . . i, are averages over all N observation directions. It is worthdiscussing the terms on the right-hand side of the iteration equations for the estimate of t 0 and R0

more closely.Apparent translation and rotation: It may come as a surprise to find the cross product between the

estimated rotation and the average across all observation directions multiplied by the reducednearness, (R0 � hmi0dii), in the iteration equation for the estimate t 0. Similarly, in the equation forthe rotation estimate, R0, the term (t 0 � hmi0 dii) is included. These terms in Eq. 4a and 4b describethe so-called “apparent translation” and “apparent rotation” terms, due to the rotational andtranslational components of self-motion, respectively (Koenderink and van Doorn 1987). Theyare the inevitable consequence of the fact that a parallax vector observed in direction di could have


Page 6 of 22

been the result of a translation or a rotation. This is easily illustrated by comparing the same region oftwo optic flow fields where one was caused by lift translation and the other one by clockwise rollrotation (Fig. 4). The parallax vectors in this region of the equatorial visual field are quite similar and,for the position a ¼ 90�, point in exactly in the same direction. The apparent terms, however, doalso suggest a practical solution to this problem. The estimates t 0 and R0 are both applied to theaverage hmi0dii. If the observation directions are appropriately chosen so that their average is equal tozero, then the cross product with the estimated translation and rotation, and thus the “apparent”terms, also equals zero. The best way to achieve this is a pair-wise observation of parallax vectors inopposite directions. The reason being that parallax vectors in a translation flow field observed atopposite locations in the visual field assume the same direction, but those observed at equivalentlocations in a rotation flow field point in opposite directions (cf. Fig. 3), which allows us todistinguish between the two cases. The best strategy would certainly be to observe parallax vectors

Fig. 4 Processing and integration of directional motion information. (a) The sign and amplitude of the output signalproduced by an elementary movement detector (EMD) depends on the direction of motion. Motion in the detector’spreferred direction (PD) and anti-preferred direction (anti-PD) results in positive and negative outputs, respectively.M stands for multiplication and t is the time constant of a filter modeling the delay stage. For further information, see textmodified from Borst and Egelhaaf (1989). (b) In the fly hexagonal eye lattice directional motion is analyzed by sets ofEMDs (thick black arrows) along the ommatidial rows. Along the vertical rows mostly next neighbor interactions areimplemented. The analysis of the two oblique rows is combined to result in horizontal motion sensitivity (thin arrows).Individual detectors will not be able to distinguish between motion due to translation or rotation. (c) Red lines indicatethe orientation of “vertical” rows in the one quadrant of the right eye plotted over azimuth and elevation. The localpreferred directions (LPDs, blue vectors) of vertical cells (VS cells) follow roughly the orientation changes of thevertical rows which are most conspicuous in the dorsofrontal part of the eye. Data from Petrowitz et al. (2000).(d) A lobula plate tangential cell (LPTC) integrates the outputs of retinotopically arranged arrays of EMDs. The VS6cell receives input only from those EMDs, the preferred direction of which is aligned with the direction of local parallaxvectors in an optic flow field induced during roll rotation. The VS6 cell receptive field may be considered a matched filterfor roll-induced optic flow. For further explanation, see text (Redrawn from Egelhaaf et al. (2002))


Page 7 of 22

at many paired locations over as large a solid angle as possible – ideally 4p –which would enable theanalysis of optic flow generated in both visual hemispheres. We will later show examples of opticflow processing in insects which reflect an approximation of exactly that strategy.

The matrices of observation directions: The results of the averaged dyadic products includingnearness estimates hmi0 di � mi0dii in Eq. 4a and hdi � dii in Eq. 4b are 3 � 3 matrices, subtractedfrom the unit matrix I. These matrices are independent of either t0 or R0 and simply indicate theoverall distribution of the chosen directions, di, at which parallax vectors pi are observed. The samereasoning put forward in the last paragraph as a way to reduce the apparent terms may be appliedhere, which introduces a very useful simplification of iteration Eqs. 4a and 4b. Assuming a suffi-ciently high number of pair-wise observation directions reasonably covering the visual field willsignificantly reduce the off-diagonal elements of the matrices, or even make them disappear(Dahmen et al. 2001). We can then rewrite Eqs. 4a and 4b which may be used as the initial estimates,t00 and R0

0:

mi0px, i

D E= m02

i sin 2 yx, i� � �

t00 ffi � mi

0py, iD E

= m02i sin 2 yy, i

� � �

mi0pz, i

D E= m02

i sin 2 yz, i� � �

(5a)

and

py, idz, i � pz, idy, iD E

= sin 2 yx, i� � �

R00 ffi pz, idx, i � px, idz, i

D E= sin 2 yy, i

� � �

px, idy, i � py, idx, iD E

= sin 2 yz, i� � �

(5b)

where yx,i, yy,i, and yz,i give the angles between di and the cardinal axes of the coordinate system(cf. Fig. 1b). These initial estimates are obtained in a single calculation step. Given an appropriatedistribution of observation directions which minimizes the apparent term in Eq. 4a, the translationestimate, t00, is simply obtained from the average across the nearness-scaled parallax vectorcomponents, normalized by averages of the products of the squared nearness and observationdirection components. Similarly, under the same assumptions regarding the distribution of obser-vation directions, the rotation estimate, R0

0, in Eq. 4b is based on the averages of the cross productsbetween the parallax vector components and the components of the observation directions, normal-ized by the averages of the squared components of the observation direction. These initial estimatesprovide already a fairly decent approximation of the actual self-motion parameters. But in case localparallax vector observations are noisy, the distribution of observation directions has been lessfavorably chosen, and the distance or nearness distribution is inhomogeneous, the full iterationprocess using Eqs. 4a, 4b, and 4c clearly improves the results. Choosing a reasonable number ofobservation directions, say 50, distributed evenly over the 4p visual field enables stable estimates ofthe three parameters t 0, R0, and mi0 within about 10–20 iterations. Obviously, the exact number ofiterations required also depends on the overall noisiness of the observed parallax vectors, pi.

So far we considered the general case, independent of the nature of the optical system movingrelative to a static environment. On the one hand side, it is the beauty of the KvD algorithm that itdoes not bother about questions concerned with the implementation of mechanisms which enablethe measurement of local parallax vectors. The KvD algorithm and later applications of it simply


Page 8 of 22

point out the principle limits of retrieving self-motion parameters from optic flow fields. In the nextsection we will discuss the mechanisms required for detecting the direction of motion in biologicalvisual systems and how specific adaptations in the visuomotor pathway of a simple model system,the common blowfly, shed some light on the strategies underlying neuronal processing of optic flow.

Matched Filter for Optic Flow ProcessingWhen estimating self-motion components from optic flow fields, the visual system does not have theluxury of spending time on an iterative procedure. It also has to solve the problem of the ambiguityof local parallax vectors which may potentially increase the apparent motion terms in Eqs. 4a and 4b,and it has to establish a neural mechanism that measures the pi in the first place. Flying insects, withtheir compound eyes endowing them with nearly 4p vision, are in a prime position to process opticflow. Depending on the species, they analyze visual motion at around hundreds (fruit flies),thousands (blowflies), or even tens of thousands (dragonflies) of locations spread across the visualfield (e.g., Land and Nilsson 2012). This enables them to establish a favorable distribution ofobservation directions, di, which helps to keep the apparent motion terms small.In addition – except for situations where they are subjected to unpredictable disturbances of theirflight attitude and trajectory – they may structure their self-motion by alternating phases oftranslation and rotation, which further reduces the apparent motion terms during the different phases.Such alternating flight patterns have been studied in blowflies and were proposed to play animportant role in estimating relative distance from analyzing translational optic flow (cf. “▶VisualProcessing in Free Flight,” this volume).

But how is the local parallax vector analyzed? Ideally, what the visual system needs to measure isthe direction and magnitude of the pi. We should start with a brief introduction of elementarymovement detectors (EMDs) which were proposed by Hassenstein and Reichardt (1953) based ona behavioral input–output analysis in beetles several decades ago. The phenomenological modelderived from the experimental results reflects the necessary and sufficient conditions required togenerate a time-averaged output, the sign of which indicates the direction of motion with respect tothe measuring axis of the EMD (rev., Reichardt 1961, 1987). An EMD has to (i) sample lightintensity changes at two neighboring locations of the eye, (ii) process the resulting signals in anasymmetric way, and (iii) combine the two signals in a nonlinear way, e.g., approximated bya multiplication operation (e.g., Reichardt 1987; Borst and Egelhaaf 1989, 1993). The fullyopponent EMD consists of two mirror-symmetrical half-detectors, the outputs of which aresubtracted from one another at a common integration stage. As a result, at the integration stage,motion in the preferred (PD) and anti-preferred direction (AD) of the detector produces on averagea positive and a negative signal. Figure 5b shows a simplified version of an EMD. The modelbasically performs a spatiotemporal correlation of light intensity changes at adjacent positions in theeye. A mathematical description of the computational properties of EMDs in the invertebrate (e.g.,Reichardt 1987) and the vertebrate visual system (e.g., Adelson and Bergen 1985) suggests that thebasic functional structure of correlation-based EMDs is quite ubiquitous (Borst and Euler 2011).

In terms of the potential implementation of the EMDs in the visual system of flies, two stages wereidentified a long time ago. As confirmed by quantitative behavioral experiments in fruit flies(Buchner 1976) and later electrophysiological studies in blowflies (Riehle and Franceschini1984), the inputs to the EMDs in the compound eyes of insects are provided by individual facets,or ommatidia, arranged in the hexagonal eye lattice. Ample electrophysiological evidence obtainedin blowflies suggested that the subtraction stage may correspond to the dendrites of wide-fieldmotion-sensitive interneurons in the posterior part of the third visual neuropile, the lobula plate (e.g.,Hausen 1984). How and where the core operation of the EMD, i.e., the nonlinear combination of the


Page 9 of 22



two input signals, may be implemented remained unclear but is currently under heavy investigationin the fruit fly using neurogenetical approaches (cf. “▶Motion Detection Models in Drosophila,”this volume).

Regarding optic flow processing in insects, the key message is that in any observation directiona set of EMDs with different preferred directions analyze local image motion and thus the system hasaccess to a quantity that is related to the parallax vectors pi. I will later discuss the issue that theoutput of an EMD is not linear in the velocity domain, but depends on several parameters describingthe spatiotemporal properties of the motion stimulus. The next question is: Howmay the self-motionparameters t and R be recovered in a non-iterative fashion?

We should revisit Eqs. 5a and 5b which provide the initial estimates of the self-motion parameters.An intuitive interpretation would be based on the following example proposed by Dahmenet al. (2001) in a slightly different notation: To retrieve the rotation component Ra around axis a,

Fig. 5 Receptive field organization of VS cells in the fly motion vision system. The upper panels showa morphological reconstruction based on injections of fluorescent dye during intracellular recordings from the cells.(a, bottom): Local preferred directions (LPDs) and motion sensitivities (LMSs) were determined from visual motionstimulation within small fractions of the receptive field at different positions in terms of azimuth, a, and elevation, b.LPDs and LMS define the orientation and length of each individual vector. The LPD distribution of the VS6 cells ishighly similar to the distribution of the local parallax vectors, pi, in an optic flow field induced by a counterclockwise rollrotation. There is a conspicuous dorsoventral asymmetry in the LMS distribution which differs from the symmetricaldistribution of magnitudes in a roll-induced flow field. (b, bottom): LPD and LMS distribution within the receptive fieldof the VS8 cell. While the LPDs reflect the preference of this cell for a body rotation in between roll and pitch(cf. singularity at a ¼ 45� and b ¼ 0�), the LMS distribution shows a similar asymmetry as found in the receptive fieldof the VS6 cell (Redrawn from Krapp et al. (1998))


Page 10 of 22


we design a directional template, UaR, which consists of a set of unit vectors ua,i

R that are all alignedwith flow vectors induced by the rotation around a observed at directions di:

URa ¼ � a� di½ = sin yi, (6)

where yi gives the angle between a and di. To evaluate the contributions of the local parallax vectorsof the current optic flow field to Ra, all pi are projected onto the local template vectors ua,i

R :

mRi ¼ pi � uRa, i ¼ pi � di½ � a= sin yi: (7)

Averaging over all contributions mRi multiplied by 1/sin yi, we make sure we obtain properly

scaled contributions of the flow vectors pi to the rotational componentRa. The best one-step estimateof the alignment between the current rotation and a rotation around the specific axis, a, is obtained byapplying weighting factors wi

R ¼ sin2yi to the average of the local projections, which is consistentwith the result of Eq. 5b.

Similarly, we may construct a directional template Uat to estimate the component ta along unit

translation awhere the unit vectors ua,it at observation directions di are parallel to the parallax vectors

induced by a:

Uta ¼ � di � a½ � di= sin yi: (8)

The projection of the current pi onto the local template directions ua,it results in local contributions:

mti ¼ pi � uta, i ¼ pi � a= sin yi: (9)

We again scale, this time by 1/mi sin yi, taking into account the relative nearness or inversedistance, and average across all contributions after applying local weightings wi

t ¼ mi2 sin2yi to

arrive at Eq. 5a.These considerations on best one-shot estimates – based on specific directional templates – allow

us to make some predictions about the properties of any neural mechanisms underlying theprocessing of optic flow to recover self-motion information. For one, neurons involved in thisfunctional context should respond to visual motion in a directional-selective way. Secondly, theyshould have extended receptive fields, ideally comprising both visual hemispheres, to reduce theambiguity inherent to local parallax vectors (apparent terms). And finally, to be able to act astemplates matching the directional distribution of local parallax vectors in a specific optic flow field,their directional preference should depend on the directions in which they analyze local visualmotion.

Response Properties of Lobula Plate Tangential CellsI mentioned earlier that the common integration stage of the two EMD half-detectors was localizedon the dendritic input arborization of large motion-sensitive interneurons in the fly lobula plate.These lobula plate tangential cells (LPTCs) have long been studied in dipteran flies with respect totheir morphology and physiological response properties, mostly in blowflies (Hausen 1984, 1993;“▶Visual Processing in Free Flight,”, this volume) and more recently in fruit flies (“▶MotionDetectionModels inDrosophila,” this volume). In each half of their visual system, blowflies employabout 60 individually identified LPTCs (e.g., Hausen 1993) which receive input from retinotopicallyarranged local motion-sensitive neurons, the function of which was suggested to be equivalent to the


Page 11 of 22




elements of the EMD providing half-detector outputs (e.g., Borst and Egelhaaf 1989). According totheir morphology, physiology, or function, LPTCs may be divided into three classes: cells tuned tothe movement of small moving objects that are involved in figure-ground discrimination (FD cells,e.g., Egelhaaf 1985), intrinsic cells contributing to the neural mechanisms underlying figure-grounddiscrimination (CH cells, e.g., Hausen 1984), and heterolateral cells propagating motion informationto the contralateral lobula plate bymeans of action potentials (H cells and V cells, e.g., Hausen 1993)and two distinct subpopulations constituting the horizontal system, HS (Hausen 1982), and thevertical system, VS (Hengstenberg 1982). The three HS and ten VS cells in the blowfly encodemotion information in a mixed signal mode, a combination of graded membrane potentials andaction potentials (Hengstenberg 1977). They are chemically and electrically connected to multi-modal descending neurons (Strausfeld and Seyan 1985; Gronenberg and Strausfeld 1990;Gronenberg et al. 1995; Haag et al. 2007) and motor neurons (Strausfeld et al. 1987) which, inturn, provide various motor systems with sensory feedback in the context of stabilization reflexes aswell as object detection and collision avoidance (e.g., Taylor and Krapp 2007; Krapp and Wicklein2008; “▶Visual Processing in Free Flight,” this volume).

There are an excessive number of publications available on the response properties of LPTCswhich were used as model systems to study a large variety of fundamental questions in neuroscienceranging from the nature of the neural code and circadian rhythms in information processing todendritic integration and sensorimotor control. In this context, I will focus on the HS and VS cellsand their role in optic flow processing. These cells have extended receptive fields and respond tovisual motion in a fully opponent way: They produce graded depolarizing membrane potential shiftsand action potentials when the cells are stimulated in their preferred direction and are hyperpolarizedupon visual motion in their anti-preferred direction (e.g., Hausen 1984). Because of their physio-logical properties, these cells were identified as ideal candidates for processing optic flow andestimating the fly’s self-motion parameters.

A detailed electrophysiological analysis of the distribution of local preferred directions at manylocations within the receptive fields of tangential cells has, in most cases, shown a high degree ofsimilarity with the distribution of local velocity vectors in specific optic flow fields (Krapp andHengstenberg 1996). This suggested that the ten VS cells, for instance, are involved in estimatingbody rotations of the fly around horizontal axes, while the receptive fields of two out of the three HScells were found to be matched to optic flow fields induced during rotations about slightly differentvertical rotation axes (Krapp 2000; Taylor and Krapp 2007). More recent research provides evidencethat HS cells are also involved in global distance estimation based on the processing of optic flowduring phases of translational self-motion (e.g., Lindemann et al. 2005; “▶Visual Processing in FreeFlight,” this volume).

Figure 5 shows the receptive field organization of two VS cells, VS6 and VS8. Here, eachindividual vector plotted over azimuth and elevation indicates the local preferred direction (LPD)and local motion sensitivity (LMS) to a locally confined directional motion stimulus (Krapp andHengstenberg 1997). By applying the KvD algorithm, the specific rotation axes these cells are tunedto could be calculated. The VS6 cell would respond best during a roll rotation of the animal(cf. Figs. 3a and 5a), while the VS8 cell prefers rotation around an axis between the roll and thepitch axis of the animal (Krapp et al. 1998). The results suggested that the receptive fields of themajority of LPTCs studied so far can be interpreted as matched filters for optic flow where eachindividual LPD may represent a weighted local unit vector, ua,i

R , of a directional template, UaR, as

described in the previous section. By selectively integrating the outputs of EMDs, the preferreddirection of which is aligned with the directions of local parallax vectors in a specific optic flow field,the output of the LPTC signals the very self-motion component that has caused it (Fig. 4d).


Page 12 of 22




Several studies on the receptive field organization of LPTCs suggested that the LPDs are virtuallyset in stone (Krapp 2014). They are innate and do not require any early visual experience for thedevelopment of their specific distribution (Karmeier et al. 2001) and are related to the orientation ofthe ommatidial rows within the hexagonal eye lattice (Fig. 4) along which directional motion isanalyzed (Buchner 1984). The stimulus-invariant conservation of the LPDs is in fact a necessarycondition that enables LPTCs to estimate self-motion parameters from the optic flow fields the flyexperiences (Krapp et al 2012).

A very different situation is found for the LMS distribution within the receptive fields. The firstmarked difference between the magnitude distribution of parallax vectors in rotational optic flowfields and the LMS distribution found in VS cells is that the former is symmetric with respect to theflow field singularity, while the latter is not (cf. Figs. 3a and 6a). The LMS distribution basicallyindicates the sensitivity of the cell to directional motion determined as the difference betweenmotion in the preferred and anti-preferred direction – essentially quantifying the height ofa directional tuning curve. Many studies have shown in the past that the sensitivity to directionalmotion in LPTCs depends on a number of stimulus parameters (e.g., Hausen 1984) and theadaptational state of the cell (e.g., Harris et al. 2000) as well as the animal’s locomotor state (e.g.,Longden and Krapp 2009; Maimon et al. 2010).

Fig. 6 Optimized matched filters for optic flow processing. Panels (a) and (b) show experimental data and modelpredictions regarding the weighting of local motion information. In LPTCsmatched to rotational optic flow fields (VS6),the sensitivity to ventral motion is reduced. In cells matched to translational optic flow (Hx), the sensitivity distribution isinverted with higher sensitivity in the ventral visual field. The model predicts a weighting of local parallax vectormeasurements which is in agreement with the experimental data. Optimal weights may reflect assumptions the visualsystem makes regarding the mean relative distance distribution within the visual environment. For further explanation,see text (Data redrawn from Franz and Krapp (2000))


Page 13 of 22

How can we interpret the LMS distribution in LPTCs? Taking an engineering perspective andconsidering the LPTCs as optimized matched filters for optic flow to estimate the translation androtation components of self-motion may provide an answer to this question.

Optimized Matched Filters for Optic FlowTo appreciate the LMS distribution in LPTCs, we combine aspects of the KvD algorithm with theconcept of the one-step estimate outlined in the section before the last. Taking into account Eqs. 6, 7,8, and 9, we may rewrite Eqs. 4a and 4b to quantify the output of matched filters tuned to a unit self-motion a:

ta, 0 ¼ n m2i sin yi pi � uta, iD E

, (10)

Ra, 0 ¼ n m2i sin yi pi � uRa, iD E

, (11)

where n is a suitable normalization factor. As described in the section on one-step estimation, ua,it and

ua,iR are unit vectors aligned in parallel with optic flow vectors induced by translation or rotation

along or around a, respectively (Dahmen et al 2001). This simple model predicts a dependence of theLMSs on sin yi, where yi is again the angle between the self-motion axis, a, and the observationdirections, di, here represented by the unit vectors ua,i. As mentioned in the last section, such sin yi-dependence was not found in the receptive field organization of the VS cells where the LMSdistributions show a pronounced dorsoventral asymmetry.

A generalized model of an optimized matched filter proposed by Franz and Krapp (2000)implements the basic structure given by Eqs. 10 and 11 but introduces two modifications: (i) Itapplies weightings to the local contributions replacing the factors sin yi, and (ii) it assumes a velocityfunction, f, modeled after the temporal frequency dependence of the EMD outputs (cf. previoussection) to result in a filter response:

r ¼ n wi f pi � ua, i� � �

: (12)

In a model described by Eqs. 10 and 11, the dot product between the parallax vectors, pi, and thetemplate vectors, ua,i, implies a linear relationship between response and velocity of the formf(x) ¼ x. This approximation is only valid within a range where the response of LPTCs isa monotonic function of velocity. Beyond that velocity range, the responses remain at a peak levelbefore dropping off as the stimulus velocity is further increased. One way of modeling the nonlinearvelocity characteristic of the EMDs is to introduce a threshold value, P, below which the output is setto zero and above which the output is normalized to unity. Such a plateau model provides equalizedresponses in those cases where |(pi � ua,i)| > P and is formalized by

r ¼ n wi pi � ua, i�

= pi � ua, i� � �

: (13)

We can now derive an analytical solution for the optimal weight applied in Eq. 13 which can beinterpreted as the LMS distribution within the receptive fields of the VS cells. To define anoptimality criterion, the following properties of optic flow fields in connection with the environmentthe filters will be used in may be considered: Translation-induced optic flow contributions depend onthe distance to visual objects in the surroundings. In case we are interested in estimating the rotationparameter, those contributions should be minimized so that the output of the optimized matched


Page 14 of 22

filter robustly indicates rotation. Assuming that the rotation of interest always occurs in combinationwith a broad variety of possible translations in a simplified “world model” approximating theaverage distance distribution of the environment, we could try to find the sets of weights, wi, thatminimizes the variance of the filter’s output. To describe the translations, a symmetrical 2-dimensional von Mises distribution with a center of mass in the thrust direction may be used. Theworld model assumes the average distances in the dorsal visual field to be twice as large as thedistances in the ventral visual field, with an isotropic distribution of local variances, whichessentially looks like a sphere, the bottom half of which is flattened. Finally, an additive uniformnoise variance, si, is included locally that takes into account stochastic variations in the outputsignals of the EMDs (for details, see Franz and Krapp 2000).

Under these assumptions, the optimal weight set for a matched filter indicating rotation is given by

wi ¼ n pi � ua, i� 2D E

= Dt2i þ Ds2i�

, (14)

where n ensures that the sum of all wi in the plateaumodel causes a filter output of 1, Dti2 is the local

variance induced by the distance variability, and Dsi2 gives the local noise variance. In observation

directions where the average distance is small, the relative local distance variability is larger, and dueto a larger contribution to the local parallax vector caused by translation, t, along ua,i, the term Dti

2

becomes particularly large. As a result, local weights assigned to those directions, which are in theventral visual field, are small. This corresponds to the LMS distribution found in the VS cells(Fig. 6a). A w2 test to assess the goodness of fit between the model-derived weights and the measuredLMSs based on w2-values did not indicate a significant difference between the two distributions.

Depending on the velocity function, f, implemented in the model, different optimal weights werepredicted with respect to their dependence on the angleY between the filter’s preferred axis, a, andthe observation directions. Although both models produced a dorsoventral asymmetry for theweights, in case of the linear model, the weights showed a sinY-dependence, which is in agreementwith Eq. 13 but significantly different from the experimental data. The plateau model, assuming unitcontributions from EMDswhen the threshold value Pwas exceeded, resulted in a sin2Y-dependencewhich, as mentioned above, was consistent with the LMS distributions found in the receptive fieldsof the VS cells (Franz and Krapp 2000).

The same approach applied to predict the optimal weights of a matched filter that indicatesa specific horizontal translation produced a distribution similar to that of LMSs found within thereceptive field of an LPTC that resembles a translational optic flow field. In this case, the dorso-ventral asymmetry was inverted. Locations in the ventral part of the receptive field were assignedhigher weights than those in the dorsal part – a result that, again, was in agreement with theexperimental findings (Fig. 6).

Taking into account the modeling results, the dorsoventral asymmetry of the LMS distribution inLPTCs can be easily interpreted from the functional point of view. An optimal matched filter thatindicates the presence of a specific rotation component should do so no matter what translation theanimal is engaging on at the same time. As the fly is likely to maneuver closer to the ground than tothe sky, it will introduce substantial translation-induced contributions to the local parallax vectorsespecially in the ventral visual field. Reducing the weights in the ventral visual field will reducethose contributions. The higher weight given to measurements of parallax vectors in dorsal visualfield where the distances are larger makes perfect sense as translation-induced optic flow is scaleddown by its distance dependence. Applying a similar rationale, the inverted sensitivity or weightdistribution in matched filters estimating translation components may also be readily explained. In


Page 15 of 22

this case, more reliable signals are expected in the ventral visual field, while the dorsal visual fieldwould feature stronger contributions to the filter output by unwanted rotation-induced parallaxvectors. An interesting observation regarding the LMS distribution of all ten VS cells is that theirdorsoventral asymmetry shows the same dependence on elevation. It is tempting to assume that thisasymmetry in the LMS distribution reflects an internal model of the average distances in the worldthe fly has developed over the time course of evolution.

Conclusions

The beauty of the concept of optic flow is that it allows us to clearly point out what can and whatcannot be retrieved from visual relative motion in terms of self-motion parameters. Inspired byGibson (1950) and thoroughly developed formally by theoretician and computer vision scientists(e.g., Koenderink and van Doorn 1987; Barron et al. 1994), it has certainly benefited research invisual neuroscience over the last few decades, in particular the work on arthropods. Althoughvertebrates such as primates and birds have long been studied regarding the neural mechanismsunderlying optic flow processing (e.g., Lappe 2000), in these animals, a straightforward applicationof the mathematical framework to the local receptive field properties of individual cells has hardlybeen achieved. One of the reasons is that cells in visual areas concerned with higher-order motionvision in monkeys, such as MST and MSTd, have highly nonlinear integration properties. Localmotion stimuli, which were successfully used in flies, are mostly ineffective to induce measurableresponses in otherwise directional-selective cells. Only if comparatively large areas are presentedwith motion patterns (Ø > 30�) the cells respond to rotational or translational flow field approxi-mations but may as well not show a high degree of specificity (e.g., Tanaka et al 1989, 1993; DuffyandWurtz 1991a, b). A characterization regarding flow field specificity in the pigeon basal optic roothas been quite successful (e.g., Wylie and Frost 1999) – in particular the evidence of a commonreference frame for optic flow processing neurons and the measuring axes of the semicircular canalsin the vestibular system (Wylie et al. 1998). But again, detailed mappings of local directionalpreferences, as achieved in the fly, have not yet been obtained.

Despite their small size which has been a major limitation for electrophysiological studies, one ofthe great advantages of flies is that they are complex enough to address fundamental questions inneuroscience and yet simple enough to obtain conclusive answers. Their limited behavioral reper-toire, which in many cases allows for a quantitative input–output characterization, provides a well-defined context for what the nervous system is required to achieve. In particular with respect tovisuomotor control, flies are among the most successful model systems used. Their motion visionpathway supporting flight and gaze control has been – and still is – exceptionally well studied usinga combination of neuroanatomical, electrophysiological, and computational approaches which maybe directly linked to specific behavioral tasks. By applying neurogenetics in fruit flies, majorprogress has been made over recent years to identify the neural correlates of the EMD (“▶MotionDetection Models in Drosophila,” this volume), the holy grail not only for invertebrate researchersbut also regarding the question of how neurons multiply.

Despite the considerable level of detail at which motion vision in flies has been studied and ourrecent advances in understanding the neural mechanisms underlying optic flow processing, there arestill a number of unsolved issues to consider. In the context of the VS cells, one of the questions is:How is the complex receptive field organization in these neurons achieved? Comparatively, recentresearch suggested that the VS cells are coupled among each other, either directly by means ofelectrical synapses or through another interneuron (Haag and Borst 2004). As a consequence, their


Page 16 of 22



receptive fields based on retinotopic inputs are smaller along the azimuth if determined in thedendrite than they are when measured in the axon (Elyada et al. 2009). The coupling between the VScells nicely explains the enormous receptive field extent, which is far bigger than it would bepredicted solely based on synaptic input to their dendritic arborization. A question that still needsanswering is: How do the dendritic branches and the axons of the local input elements find each otherin the correct directional-selective input layers (Buchner and Buchner 1984) of the lobula plate toestablish a specific optic flow template?

Another proximate question concerns the nonlinear properties of the EMDs in the velocitydomain (Reichardt 1987), the fact that motion and contrast adaptation changes the cells’input–output characteristic (Maddess and Laughlin 1985; Harris et al. 2000) and that local inputsare not linearly integrated on their dendrites (Borst et al. 1995). How useful are the LPTC forbehavioral control, if they do not provide a feedback signal that represents absolute velocity?

Before attempting an answer, there is an ultimate question that should be addressed, which waslong puzzling people working in the area.Why are there somany LPTCs? The number of VS and HScells per brain hemisphere is already 13. The two cellular systems alone comprise 26 cells – andthere are many more that could function as matched filters for optic flow. From the technical point ofview, six cells would be enough to retrieve all possible self-motion components. A potential answermay be related to another heavily researched area in neuroscience concerned with sensorimotortransformations (e.g., Krapp 2010). Sensory signals are normally obtained in local coordinatesystems, which – in this case – would be defined by the orientation of the ommatidial rows in theeye lattice (cf. Fig. 4b, c). But what needs to be controlled are motor systems, the pulling planes ofwhich may be described in entirely different coordinates. The mechanisms underlying the transfor-mation from sensory signals into appropriate motor commands could be quite complicated, anda small number of degrees of freedom regarding the motor system may already allow for a vastnumber of possible solutions, if the problem is not constrained.

The LPTCs seem to perform exactly this task. They integrate local motion information on theirdendrites, but their output reflects exactly what is needed to control the behavior: information aboutspecific self-motion components or – more technically – about state changes. Each of them hasa preferred self-motion axis, and together the LPTCs set up a high-dimensional coordinate system,where the number of dimensions is equal to the number of different matched filter axes they cover.The distribution of those axes seems not to be random. After studying the arrangement, it wasrecently hypothesized that the preferred rotation axes of the VS cells are ideally suited to sense oneof the natural modes of motion that flying systems encounter, a “Dutch roll” (Taylor andKrapp 2007). The basic idea is that the sensory coordinate system set up by the LPTCs correspondsto the motor coordinate systems it controls – a design that would considerably simplifya sensorimotor transformation required for flight stabilization (Krapp et al 2012). For this to workefficiently, one crucial condition has to be fulfilled: The visual system needs to stay aligned with theinertial vector, which means the head has to be stabilized in the horizontal plane.

Evidence in support of this “mode-sensing hypothesis” comes from studies of motor neuronssupplying the fly neck muscles which control the animal’s gaze (Huston and Krapp 2008). Neckmotor neurons receive input from LPTCs such as the HS and VS cells, either directly or viadescending neurons (Gronenberg and Strausfeld 1990; Gronenberg et al. 1995). Neck motorneurons have large visual receptive fields and respond to visual motion in almost the same way asLPTCs, but some of them require mechanosensory input simultaneously to generate action poten-tials, which is a necessary condition for muscle contraction (Huston and Krapp 2009). Figure 7ashows parts of a neckmotor neuron receptive field that responds to horizontal motion induced duringyaw rotation. Its visual receptive field looks almost identical to the receptive field of a so-called HSE


Page 17 of 22

cell. The only difference is that it receives stronger input from the contralateral eye. This largebinocular receptive field helps to reduce apparent terms due to translation (cf. Eq. 4b) which makesperfect sense for a neuron that controls head rotations. Using the KvD algorithm on the receptivefield organization of each of the neck motor neurons studied, their preferred rotation axes werecomputed. The resulting distribution of axes is compared in Fig. 7b with the distribution of thepreferred rotation axes of all 26 HS and VS cells and shows a high degree of overlap (Huston andKrapp 2008). Almost all neck muscles in flies are innervated only by a single motor neuron. It islikely, therefore, that the pulling planes of the muscles are predicted by the preferred rotation axes ofthe neck motor neurons, which still needs to be tested experimentally. The results so far arecompatible with the idea that the high number of LPTCs may be interpreted along the lines of themode-sensing hypothesis (Krapp et al. 2012): There should be roughly as many matched filters forspecific optic flow fields as there are axes in the motor systems that need to be controlled.

To get back to an earlier question regarding the suitability of LPTC signals for motor control, thereis another essential aspect that should be considered. Motion vision alone would not be sufficient tosupport the high maneuverability and gust tolerance of flies. The limited bandwidth and longprocessing delays of the visual system are complemented by fast and low latency mechanosensorysystems, such as the halteres in dipteran flies signaling angular rotation rates (Hengstenberg 1993) orthe antennae measuring air flow changes (Taylor and Krapp 2007). One of the current and futurechallenges will be to understand how optic flow-based and mechanosensory signals are integrated tocontrol flight and gaze as efficiently as dipteran flies do. A proof of concept study where matchedfilters for optic flow were implemented in a linear control engineering framework did indeed enableautonomous flight of a quadcopter (Hyslop et al. 2010). But only if we understand multisensory gazeand flight stabilization well enough to extract the underlying biological control design, we mayeventually reach a performance level in artificial micro air vehicles their natural counterpartsachieve.

Fig. 7 Visual receptive field of a fly neck motor neuron and the preferred rotation axis distributions of neckmotor neurons and LPTCs. (a) The neck motor neuron receptive field is matched to an optic flow field generated bya rotation around a slightly tilted vertical body axis. (b) Preferred axes of neck motor neurons (red arrows) and LPTCs(blue arrows) show a high degree of overlap, suggesting that the sensory coordinate system (LPTC) and the coordinatesystem controlling the fly’s gaze are co-aligned. For further explanation, see text (Redrawn from Huston andKrapp (2008))


Page 18 of 22

References

Adelson EH, Bergen JR (1985) Spatiotemporal energy models for the perception of motion. J OptSoc Am 2:284–299

Angelaki DE, Cullen KE (2008) Vestibular system: the many facets of a multimodal sense. AnnuRev Neurosci 31:125–150

Barron JL, Fleet DJ, Beauchemin SS (1994) Performance of optical flow techniques. Int J Comp Vis12:43–77

Borst A, Egelhaaf M (1989) Principles of visual motion detection. Trends Neurosci 12:297–306Borst A, Egelhaaf M (1993) Detecting visual motion: theory and models. In: Miles FA, Wallman

J (eds) Visual motion and its role in the stabilization of gaze, vol 5, Reviews of oculomotorresearch. Elsevier, Amsterdam/London/New York/Tokyo, pp 3–27

Borst A, Euler T (2011) Seeing things in motion: models, circuits, and mechanisms. Neuron71:974–994

Borst A, Egelhaaf M, Haag J (1995) Mechanisms of dendritic integration underlying gain control infly motion-sensitive interneurons. J Comput Neurosci 2:5–18

Buchner E (1976) Elementary movement detectors in an insect visual-system. Biol Cybern24:85–101

Buchner E (1984) Behavioural analysis of spatial vision in insects. In: Ali MA (ed) Photoreceptionand vision in invertebrates. Plenum Press, New York, pp 623–634

Buchner E, Buchner A (1984) Neuroanatomical mapping of visually induced nervous activity ininsects by 3H-deoxyglucose. In: Ali MA (ed) Photoreception and vision in invertebrates. PlenumPress, New York, pp 561–621

Dahmen H, Franz MO, Krapp HG (2001) Extracting egomotion from optic flow: limits of accuracyand neural matched filters. In: Zanker MJ, Zeil J (eds) Motion vision. Computational, neural, andecological constraints. Springer, Berlin/Heidelberg/New York/Tokyo, pp 143–168

Duffy CJ, Wurtz RH (1991a) Sensitivity of MST neurons to optic flow stimuli. I. A continuum ofresponse selectivity to large-field stimuli. J Neurophysiol 65:1329–1345

Duffy CJ, Wurtz RH (1991b) Sensitivity of MST neurons to optic flow stimuli. II. Mechanisms ofresponse selectivity revealed by small-field stimuli. J Neurophysiol 65:1346–1359

Egelhaaf M (1985) On the neuronal basis of figure-ground discrimination by relative motion in thevisual-system of the fly.2. Figure-detection cells, a new class of visual interneurones. Biol Cybern52:195–209

Egelhaaf M, Kern R, Krapp HG, Kretzberg J, Kurtz R, Warzecha AK (2002) Neural encoding ofbehaviourally relevant visual-motion information in the fly. Trends Neurosci 25:96–102

Elyada YM, Haag J, Borst A (2009) Different receptive fields in axons and dendrites underlie robustcoding in motion-sensitive neurons. Nat Neurosci 12:327–332

Franz MO, Krapp HG (2000) Wide-field, motion-sensitive neurons and matched filters for opticflow fields. Biol Cybern 83:185–197

Gibson JJ (1950) The perception of the visual world. Houghton Mifflin, BostonGibson JJ (1979) The ecological approach to visual perception. Houghton Mifflin, BostonGronenberg W, Strausfeld NJ (1990) Descending neurons supplying the neck and flight motor of

Diptera: physiological and anatomical characteristics. J Comp Neurol 302:973–991Gronenberg W, Milde JJ, Strausfeld NJ (1995) Oculomotor control in calliphorid flies: organization

of descending neurons to neck motor neurons responding to visual stimuli. J Comp Neurol361:267–284


Page 19 of 22

Haag J, Borst A (2004) Neural mechanism underlying complex receptive field properties of motion-sensitive interneurons. Nat Neurosci 7:628–634

Haag J, Wertz A, Borst A (2007) Integration of lobula plate output signals by DNOVS1, anidentified premotor descending neuron. J Neurosci 27:1992–2000

Harris RA, O’Carroll DC, Laughlin SB (2000) Contrast gain reduction in fly motion adaptation.Neuron 28:595–606

Hassenstein B, Reichardt W (1953) Der Schluss von Reiz-Reaktions-Funktionen auf System-Strukturen. Z Naturforsch B 8:518–524

Hausen K (1982) Motion sensitive interneurons in the optomotor system of the fly.1. The horizontalcells – structure and signals. Biol Cybern 45:143–156

Hausen K (1984) The lobula-complex of the fly: Structure, function and significance in visualbehaviour. In: Ali MA (ed) Photoreception and vision in invertebrates. Plenum Press, New York,pp 523–559

Hausen K (1993) Decoding of retinal image flow in insects. In: Miles FA, Walman J (eds) Visualmotion and its role in the stabilization of gaze, vol 5, Reviews of oculomotor research. Elsevier,Amsterdam/London/New York/Tokyo, pp 203–235

Hengstenberg R (1977) Spike responses of ‘non-spiking’ visual interneurone. Nature 270:338–340Hengstenberg R (1982) Common visual response properties of giant vertical cells in the lobula plate

of the blowfly Calliphora. J Comp Physiol A 149:179–193Hengstenberg R (1993) Multisensory control in insect oculomotor systems. In: Miles FA, Walman

J (eds) Visual motion and its role in the stabilization of gaze, vol 5, Reviews of oculomotorresearch. Elsevier, Amsterdam/London/New York/Tokyo, pp 285–298

Horn BKP, Schunck BG (1981) Determining optic flow. J Artif Intell 17:185–204Huston SJ, Krapp HG (2008) Visuomotor transformation in the fly gaze stabilization system. PLoS

Biol 6:1468–1478Huston SJ, Krapp HG (2009) Nonlinear integration of visual and haltere inputs in fly neck motor

neurons. J Neurosci 29:13097–13105Hyslop A, Krapp HG, Humbert JS (2010) Control theoretic interpretation of directional motion

preferences in optic flow processing interneurons. Biol Cybern 103:353–364Karmeier K, Tabor R, Egelhaaf M, Krapp HG (2001) Early visual experience and the receptive-field

organization of optic flow processing interneurones in the fly motion pathway. Vis Neurosci18:1–8

Koenderink JJ, van Doorn AJ (1975) Invariant properties of motion parallax field due to movementof rigid bodies relative to an observer. Opt Acta 22:773–791

Koenderink JJ, van Doorn AJ (1987) Facts on optic flow. Biol Cybern 56:247–254Krapp HG (2000) Neuronal matched filters for optic flow processing in flying insects. Int Rev

Neurobiol 44:93–120Krapp HG (2010) Sensorimotor transformation: from visual responses to motor commands. Curr

Biol 20:R236–R239Krapp HG (2014) Flies, optic flow, and multisensory stabilization reflexes. In: Bleckmann H,

Coombs S, Mogdans J (eds) Flow sensing in air and water. Springer Heidelberg New York,Dondrecht, London pp 215–243

Krapp HG, Hengstenberg R (1996) Estimation of self-motion by optic flow processing in singlevisual interneurons. Nature 384:463–466

Krapp HG, Hengstenberg R (1997) A fast stimulus procedure to determine local receptive fieldproperties of motion-sensitive visual interneurons. Vis Res 37:225–234


Page 20 of 22

Krapp HG, Wicklein M (2008) Central processing of visual information in insects. In: Basbaum AI,Kenako A, Shepherd GM, Westheimer G (eds) The senses: a comprehensive reference. MaslandIR, Albright TD (eds) Vision I, vol 1. Academic, San Diego, pp 131–204

Krapp HG, Hengstenberg B, Hengstenberg R (1998) Dendritic structure and receptive-field orga-nization of optic flow processing interneurons in the fly. J Neurophysiol 79:1902–1917

Krapp HG, Taylor GK, Humbert JS (2012) The mode-sensing hypothesis: matching sensors,actuators and flight dynamics. In: Barth FG, Humphrey JAC, Srinivasan MV (eds) Frontiers insensing – from biology to engineering. Springer, Wien, pp 101–114

Land MF, Nilsson DE (2012) Animal eyes. Oxford University Press, OxfordLappe M (2000) Neuronal processing of optic flow, vol 44, International review of neurobiology.

Academic, San DiegoLindemann JP, Kern R, van Hateren JH, Ritter H, EgelhaafM (2005) On the computations analyzing

natural optic flow: quantitative model analysis of the blowfly motion vision pathway. J Neurosci25:6435–6448

Longden KD, Krapp HG (2009) State-dependent performance of optic-flow processing interneu-rons. J Neurophysiol 102:3606–3618

Maddess T, Laughlin SB (1985) Adaptation of the motion-sensitive neuron H-1 is generated locallyand governed by contrast frequency. Proc R Soc B 225:251–275

Maimon G, Straw AD, Dickinson MH (2010) Active flight increases the gain of visual motionprocessing in Drosophila. Nat Neurosci 13:393–399

Nakayama K, Loomis JM (1974) Optical velocity patterns, velocity-sensitive neurons, and spaceperception: a hypothesis. Perception 3:63–80

Petrowitz R, Dahmen H, Egelhaaf M, Krapp HG (2000) Arrangement of optical axes and the spatialresolution in the compound eye of the female blowfly. J Comp Physiol A 186:737–746

Reichardt W (1961) Autocorrelation, a principle for the evaluation of sensory information by thecentral nervous system. In: Rosenblith WA (ed) Sensory communication. MIT Press, Cambridge,pp 303–317

Reichardt W (1987) Evaluation of optical motion information by movement detectors. J CompPhysiol A 161:533–547

Riehle A, Franceschini N (1984) Motion detection in flies: parametric control over ON-OFFpathways. Exp Brain Res 54:390–394

Strausfeld NJ, Seyan HS (1985) Convergence of visual, haltere, and prosternal inputs at neck motorneurons of Calliphora erythrocephala. Cell Tissue Res 240:601–615

Strausfeld NJ, Seyan HS, Milde JJ (1987) The neck motor system of the fly Calliphora-Erythrocephala. 1. Muscles and motor neurons. J Comp Physiol A 160:205–224

Tanaka K, Fukada Y, Saito HA (1989) Underlying mechanisms of the response specificity ofexpansion/contraction and rotation cells in the dorsal part of the medial superior temporal areaof the macaque monkey. J Neurophysiol 62:642–656

Tanaka K, Sugita Y, Moriya M, Saito HA (1993) Analysis of object motion in the ventral part of themedial superior temporal area of the macaque visual-cortex. J Neurophysiol 69:128–142

Taylor GK, Krapp HG (2007) Sensory systems and flight stability: What do insects measure andwhy? Adv Insect Physiol Insect Mech Control 34:231–316

Wylie DR, Frost BJ (1999) Responses of neurons in the nucleus of the basal optic root totranslational and rotational flow fields. J Neurophysiol 81:267–276

Wylie DR, Bischof WF, Frost BJ (1998) Common reference frame for neural coding of translationaland rotational optic flow. Nature 392:278–282


Page 21 of 22

Further ReadingEgelhaaf M, Borst A (1993)Movement detection in arthropods. In: Miles FA,Walman J (eds) Visual

motion and its role in the stabilization of gaze, vol 5, Reviews of oculomotor research. Elsevier,Amsterdam/London/New York/Tokyo, pp 53–77

Krapp HG (2009) Sensory integration: neuronal adaptations for robust visual self-motion estima-tion. Curr Biol 19:R413–R416

Krapp HG, Hengstenberg R, Egelhaaf M (2001) Binocular contributions to optic flow processing inthe fly visual system. J Neurophysiol 85:724–734

Parsons MM, Krapp HG, Laughlin SB (2010) Sensor fusion in identified visual interneurons. CurrBiol 20:624–628

Schwyn D, Hernadez Heras FJ, Bolliger G, Parsons MM, Krapp HG, Tanaka RI (2011) Interplaybetween feedback and feed forward control in fly gaze stabilization. In: 18th World Congress ofInternational Federation of Automated Control (IFAC), Milan, pp 9674–9679


Page 22 of 22

Documents

Encyclopedia of Computational Neuroscience || Optic Flow Processing