IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS, VOL. SMC-6, NO. 2, FEBRUARY 1976 127

On Models for Line DetectionLARRY S. DAVIS, STUDENT MEMBER, IEEE, AZRIEL ROSENFELD, FELLOW, IEEE, AND

ASHOK K. AGRAWALA, MEMBER, IEEE

Abstract-Several models for line detection in noise are discussed, 4 axis of orientationbased on models for the "simple" and "complex" cells that have been -found in the visual cortex. The possibility of obtaining experimentalevidence for deciding among the models is briefly considered.

I. INTRODUCTION

T IS NOW generally accepted that the human visual inhibitory excititory inhibitory1 ~~~~ ~ ~~~~~ ~~~~~~~~~~zonezolizone

l system contains "feature detectors" that respondselectively to bright or dark lines. Various investigators,e.g., [5], [7], have proposed visual pattern recognitionsystems based on simulations of such detectors. This paperdiscusses possible computer models for these detectors andfor the analysis of their output. Some experiments are sug-gested which should cast light on the relative plausibilitiesof such models.

II. SIMPLE AND COMPLEX CELLS

A. Simple Cells

In their classic neurophysiological experiments, Hubeland Wiesel [8]-[l 1] found specialized feature detectorsin the visual pathways of the cat and spider monkey. In Wparticular, they found a type of detector, called a "simple Fcell," that responds optimally to straight lines of a partic-ular position, length, and orientation [8]. Psychophysicalexperiments on human observers have produced strong exposed to diffuse light, it does not fire-the excitatoryevidence for the existence of similar type cells in humans and inhibitory fields are therefore weil balanced. For a[2]. In the discussion below, we consider only cells that more complete description of simple cells see [10]; for arespond to bright lines ("slits"); analogous remarks apply review of other recent work in neurophysiology see [12].to cells that respond to dark lines. Many writers have suggested that simple cells are approx-

Associated with a simple cell is the set of retinal receptor imately linear devices which add the inputs from the receptorelements (i.e., rods, cones) that when stimulated affect cells in their excitatory zones and subtract the inputs fromthe firing activity of the simple cell. This set of receptor the cells in their inhibitory zones. Such a device would, inelements is called the receptive field (RE) of the simple fact, respond maximally to a (bright) line through itscell. Fig. 1 shows the receptive field of the type of simple excitatory zonecell we are interested in. It is composed of a central Hubel and Wiesel report that a simple cell responds to aexcitatory zone and two flanking, symmetric inhibitory bright spot anywhere in its elongated excitatory receptivezones. The boundary between the excitatory and inhibitory field and that this response becomes stronger if the spotzone defines the axis of orientation of the cell. Empirically, is enlarged, as long as the neighboring inhibitory zones arethis cell responds optimally when its excitatory zone is not illuminated [10]. They do not report whether thestimulated, but the inhibitory zones are not. Thus, for response becomes stronger when the spot becomes brighterexample, a straight line through the excitatory zone, paral- while remaining the same size. Such a result would seem tolel to the axis of orientation of the cell, will excite the follow from the assumption usually mace about simplecell (i.e., cause it to fire); however, a straight line orthog- cells, namely, that their response is a result of linear sum-onal to the axis of orientation and extending across the mation (over both the excitatory and inhibitory zones);entire RE will not excite the cell. Also, when the cell is see, however, below.

B. Complex CellsManuscript received January 20, 1975; revised July 14, 1975. This

work was supported by the Division of Computer Research, National Hubelan eslhvasodcrbd"mpxcls.Science Foundation, under Grant GJ-32258X. Such a cell responds to a brightlinS e of a given orientationThe authors are with the Computer Science Center, University of

Maryland, College Park, MD 20742. fre th assumption but note about sple

128 IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS, FEBRUARY 1976

nor to a line of incorrect orientation. This response is always give a higher response than a spot, no matter howsaid to increase with line length, but is no greater for bright.two (parallel) lines than for a single line [10]. They do Suppose now, similarly, that a complex cell has excita-not report the existence of an inhibitory zone associated tory inputs from a set of simple cells, but that its outputwith a complex cell, such that some stimulus present in depends on the number of these inputs that are above somethis zone could inhibit response even when the proper line threshold, rather than on the sum of the inputs. This wouldstimulus is present in the field. account for the observation that the complex cell does notThe properties of complex cells just described would respond to a scattering of spots, no matter how bright,

seem to be inconsistent with a straightforward interpreta- but does respond to a line. There is no response to a settion of complex cells as ORS, i.e., logical disjunctions, of of below-threshold inputs, but there is a response to ansimple cells (as Hubel and Wiesel [10] suggest), since the above-threshold input. A long line would yield a strongerlatter fire when stimulated by spots, but the former do not. response, since more of the simple cells are excited byThis cannot simply be a matter of thresholding (based on it. To explain why two parallel lines give no strongerthe fact that simple cells respond more weakly to dots than response than a single line, a small refinement is neces-to lines), for two reasons. sary. One can assume that the complex cell's receptive

1) If simple cells are truly linear, they should respond field is divided into zones perpendicular to the preferredmore strongly to very bright spots than to less bright lines. direction and that only the number of zones having above-

2) In any case, a complex cell should fire when several threshold inputs is counted (so that the complex cell takesof its input simple cells are excited by spots. It should thus an OR within each zone and a sum across the zones).respond to scatterings of spots, even when these are not It should be pointed out that the thresholding opera-collinear. tions described above must somehow take place between

It has been suggested (e.g., [6]) that one can account output from the concentric cells and input to the simplefor the nonresponse of complex cells to spots by assuming cells, and between output from the latter and input to thethat they have strong inhibitory input from simple cells complex cells. Moreover, one cannot postulate additionalwhose fields are oriented perpendicularly to the preferred layers of neurons that perform these operations, since suchline direction. Thus a spot would not cause the complex layers have not been detected-no recordings have beencell to fire, since these orthogonal simple cells would be made from cells that behave like thresholded concentric orfiring and would inhibit it. simple cells. Operations of this sort could be performed in

This model can be questioned on the grounds that there a number of ways in the course of transmitting informationdoes not appear to be any spatial association, in the cor- from the concentric to the simple to the complex cells, e.g.,tex, between pairs of perpendicular directions. Note also by self-inhibitory fibers. A more detailed discussion of thesethat the model requires that the fields of the inhibiting simple speculations will not be attempted here.cells must "cover" the fields of the exciting simple cells.If some point on the retina lies in one of the exciting fields, III. MODELS FOR LINE DETECTION USING SIMPLE CELLSbut not in any of the inhibiting fields, the complex cell We now consider how to formulate models for line detec-would have to respond to a spot at that point. Conversely, tion using simple cells. This involves not only choosing aif some point lies in an inhibiting field but in no exciting model (linear or nonlinear) for the cells themselves, butfield, a bright spot at that point should inhibit the complex also devising a model for the manner in which the outputscell even when a line in the excitatory orientation is present. of the modeled cells are analyzed in arriving at a detectionNote, however, that no such inhibitory effects have been decision.reported. Suppose that we want to model the detection of dotted

straight lines embedded in a background of "noise," suchC. Nonlinear Models for Simple and Complex Cells as shown in Fig. 2. These particular patterns are 255 byThe observed responses of simple and complex cells can 255 arrays in which the probability of an "on" dot is 0.5.

be accounted for, at least qualitatively, if one assumes At one of the two positions indicated by the tick marks, athat a thresholding operation has been performed on their dotted line of dot density d (d = probability of an "on"inputs. Specifically, let us suppose that a simple cell has dot) has been logically oRed with the background array. Theexcitatory inputs from a set of concentrically organized detection task is to determine in which of the two positionscells (i.e., cells whose receptive fields have excitatory the line is located. We assume that each "on" dot in thecenters and inhibitory surrounds), as suggested by Hubel pattern has the same brightness.and Wiesel [10]. We assume, however, that its output is The dimensions of the "receptive fields" in our simplerelated to the number of these inputs that are above some cell can be defined in terms of dots in the pattern (seethreshold, rather thanl to the sum of the inputs. This would Fig. 3). We shall assume that the model has as many dotsbe consistent with the observation that the simple cell's in its excitatory zones as it does in both of the inhibitoryoutput increases with the area of the stimulating light spot. zones together. The excitatory and inhibitory zones areNote, however, that it would imply no increase with the rectangularly shaped and are h dots long. The excitatorybrightness of the spot. In particular, a bright line would zone has NE dots in it, and each inhibitory zone has NE!2.

DAVis et al.: MODELS FOR LTNE DETECTION 129

70, then a detector height often dots corresponds to a visualangle of about 10/255 of 70, or about (1/4)°, which is areasonable value based on current conjectures about thehuman visual system.The response of a detector should be proportional to the

difference between the numbers of "on" dots in the excita-tory zones and in the (two) inhibitory zones. This assumesthat the detector is deterministic; it does not take intoaccount the noisiness of simple cells. However, even if we

(a) (b) assumed that each detector had a normal distribution ofresponses, we could still use the same type of analysis aswe have given below, based on normal distributions.

A. The Averaging ModelWe now consider models for combining detector out-

puts in order to arrive at a line detection decision. Thefirst model is based on the average firing rate of a familyof detectors. If a line is known to be present in one of

(c) (d) two positions, we choose that position for which the familyof simple cells whose receptive fields overlap the position

(d) d = 0.20 at top.(ottom have the greater average firing rate. We use only detectorswhose orientations are parallel to the line and whosereceptive fields are entirely contained in the pattern (wedo not consider any border effects).

Suppose now that a single dot in the background-whenno line is present-has probability p of being "on" and

0 h (1- p) of being "off." Thus the probability of exactly kdots out of N being "on" in the background is

inhibitory inohibitory pkN\-k(z e

_ E/zone

_ p)NkNE/2 QXtaOy NE/2 k o

dots O citatory dotst zone In other words, the distribution of the number of back-ground dots that are "on" is binomial. The mean of this

O distribution is Np, and the variance is Np(1 - p).Now the excitatory zone and the inhibitory zones of our

0 detectors are just families of NE dots. Since we are asso-° | ciating the output of the excitatory (inhibitory) zone(s)O with the number of dots in the zone(s) that are "on," the

NE output of the excitatory (inhibitory) zone(s) in the back-Odots ground is a random variable XE(XI) binomially distributed

0 / with mean NEP and variance NEp(l - p).For sufficiently large N, we can approximate a binomial

3ig.. Sim1ledete ctor. distribution with mean p and variance a2 by a normalFig. 3. Simple detector. distribution with mean , and variance a2 (denoted N(y,o)).Thus the distribution of outputs of the excitatory zone is

We make the assumption that the area of each inhibitory N(NEp,VNEp(l - p)), and the distribution of the outputszone is the same as the excitatory zone, but we sample only of the (two) inhibitory zones is also N(NEP, NEp(l- p).half of the dots in each inhibitory zone. We shall further Now, the firing rate of the detector is just the differenceassume that the cross section of each zone is uniform (there between the outputs of the excitatory zone and the in-is no weighting of the responses to the dots). hibitory zones. However, this is just the difference of theWe shall refer to our model below as a "detector." In two random variables XE and XI. If XE is N(/1E,aE) and

the examples to be given below, we have used 11 = 10 and XI is N(f1,a1), then XE - XI is N(PE - P,uV1E2 + 712).NE = 30; this corresponds to a detector width of nine dots, Thus the firing rate of a detector when no line is presentthree in each zone, so that the detector is roughly square. is N(0,V2NEp(l - p)).If the patterns in Fig. 3 are viewed from an appropriate Superimposing a signal of level d on one row of the noisedistance, so that a pattern subtends a visual angle of about background complicates the analysis of the excitatory

130 IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS, FEBRUARY 1976

5G

48

44

40 \ AVERAGE MODEL

36 \\ NE = 30 h = 10

P[xdiff>] 32

diff~~~~~~~~~~~~~

28

24-

20-

16.

12,

8-

4

0 4 6 1i 16 26 24 28 32 36 40 44 48 52 56 60 4d

Fig. 4.

zone. The probability of a dot being "on" in noise p + Fig. 4 shows several curves of signal d versus prob-signal d is s = p + d - pd. We can analyze the excitatory ability [Xdiff > 0], for p = 0.5. The different curveszone in signal plus noise, then, by considering it to be represent different sample sizes (m) of detectors, havingcomposed of two parts: 1) a family of h dots in signal plus dimensions NE = 30 and h = 10. (On the choice of thesenoise, and 2) a family of NE- h dots in noise alone. Now, values, see above.) The curves are qualitatively reasonable,in a noise background of level p, the output of the family since we would expect the probability to drop from 0.5of NE- h dots is a random variable with mean (NE- h)p at d = 0 to near zero as d rises to 0.5 or so (correspondingand variance (NE- h)p(l - p), and the output of the to a dot density of 0.75 on the line, compared with p = 0.5family of h cells has mean hs and variance hs(l - s). The off the line).output of the excitatory zone is a random variable that is Great caution must be exercised in interpreting thethe sum of these two outputs. Its distribution is N(NEp + sample size required by the model. The model is based onhs,j(NE- h)p(l - p) + hs(l - s)). The output of the independent detectors; that is, when we speak of 20 de-inhibitory zone is still N(NEp,J/NEp(l - p)). Finally, the tectors, we mean that the RF's of the detectors are alloutput of a detector in signal, d, plus noise is N(hs, disjoint. In the context of the models one cannot discussW'2NEp(l - p) + hs(l - s)-the distribution of the dif- overlapping receptive fields. In fact, nothing would beference of the two random variables. gained by an overlap in the RF's of the detectors, because

two detectors having the same RF give no more informationXm the average of samples of sizeX of X, is distributed together than each one does separately.

-the average of samples of sizemofXisdistributed

Now, in the vertebrate visual system there is a greatas N(y,a in1m. Thus, the average firing rate of a family deal of overlap in the RF's of simple cells. However, realof m detectors in noise alone is a random variable XN simple cells are very noisy devices, and so the overlap herewith distribution N(0,V12NEp(l - p)/m); in signal plus does have a great utility because, e.g., if each simple cellnoise it is a random variable XS, with distribution N(hs, has a normal distribution of responses, then the averageV[(2NE - h)p(l - p) + hs(I - s)]/m). Now, Xdiff = response of ni simple cells with the same RF would also beXN - XS is the difference between the average response of normal. However, the variance of the average responsem detectors in noise alone and the average response of m would be reduced by a factor of 1/n over the variance ofdetectors in signal plus noise. Xd jff is distributed as N(-hs, any individual cell. Again, since our detectors are notlI[4(NE -h)p(l -p) + hs(l -s)]/m)- Finally, Pr [Xdiff > noisy devices, there is no utility in overlap. Consequently,0] = Pr [XN -X > 0] is the probability that the average we can at best make a prediction about the lower boundresponse of in detectors in noise alone is greater than the on the number of simple cells needed to perform the detec-average response of mn detectors in signal plus noise. This tion task. The visual system needs at least as many simpleis the probability of making an error in deciding where the cells as the model needs detectors, but we cannot predictdotted line is. how many more. Similar remarks hold for the other Inodels.

DAVIS et al.: MODELS FOR LINE DETECTION 131

5C

4E

44. MAX MODEL

p(pma <0) =pmapx<O ) = < N =30 h = 10

p[[max in 4E

s+n -

max in 3 \noise]

3.

2E

2-

2C

14

12m = 1,000

8 m = 8,000

4

5 10 15 20 25 30 35 40 45 50 55 60 65

signal

Fig. 5.

B. The Max Model _ _ _ _

A second model for the task of detecting a line in oneof two positions is to choose the position which contains hAkthe maximally responding detector. In general, if a randomvariable X has distribution function F(X), then the max- k-zoneimum of a sample of size n drawn from the original popula- \tion will have distribution function F5(X) and density _ _function dF'(X)/dx = nF '-(X) dF(X)/dX.

Let N(X) be the distribution function for the detectors Excitatory Inhibitorythat see noise alone and S(X) the distribution in signal E| receptor zn/2receptoplus noise. Then h

Pmax = P[max in signal + noise of n detectors

> max in noise alone]

- !; N5(X) nS8-'(X) X) dx.

However, we are interested in evaluating this integralfor large values of n, and this is computationally unfeasible.Thus, we need a limit distribution for the distribution of themaxima. We use the following form of Kendall and Stuart Fig. 6. Nonlinear detector.[13]. For samples of size n drawn from a normal populationN(j,u), the distribution function of the maximum is

which can be approximated byG(X) = exp [-exp [-(2 log n)'12 .((X -,u)/ ~ N,E

*(2 log n)"12)]]. E GN(-NV + (I - 1)e)If GN(X) iS the distribution in noise and G5(X) that in * [Qs(-N + ic) -GS(-N + (i -))/signal plus noise, then

which we can easily compute.Pmax = + NX dGs(X) dX Fig. 5 shows graphs of signal (d) versus probability

_, ~dX [Pmax < 0] for samples of various sizes. The dimensions

132 IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS, FEBRUARY 1976

5Q48

P (Xdiff > 0)

4NONLINEAR MODEL

NE = 30 h = 10 k = 3

36-

32\

28

24-

20

161

12-

8L

4

m=50

.=200 ,,, .IIIM=100

4 8 12 16 20 24 28 32 36 40 44 48 52 56 60 64

signal

Fig. 7.

of the detectors are the same as for the averaging model ing model, it suffices to examine the number of dots that(NE = 30, h = 10). Note that the sample size required by are "on" because we are assuming that the responses ofthe max model in order to yield curves similar to those the k-zones are linear with respect to brightness.in Fig. 4 is several orders of magnitude larger than for In the following analysis we use essentially an averagingthe average model. In fact, the curve for m = 15 000 is model, and we treat the special case p = 1/2. We recallso close to the curve for m = 8000 that it is not plotted. that this implies s = 1/2 + d/2, where d is the signal level.The large number that would be required by this model in The firing rate of the linear detector in this case is distributedorder to yield curves similar to those in Fig. 4 seems un- asreasonable, and for this reason it appears that the maxmodel is not viable. where s = 1/2 + d/2.

C. Models Based on Nonlinear Simple Cells Noise alone is the special case with d = 0. Now, each ofthe k-zones has NE/k receptors and h/k height. So its

Both the averaging and max models are based on the firing rate is a random variable Xd with distributionnotion of a simple cell whose firing rate is a linear function N(hd/2k,1(2NE - h)/4k + hs(I - s)/k) = N(Pk,T). (Ifof brightness. However, to view simple cells as specialized Nk is not lare enouh to ermit the use of the normalfeature detectors rather than slmply as orlentatlon-specific E/

mg g p

brightness detectors entails abandoning the idea that simple aproximatin tengthe dbuino funtion ca becells relinardevces (se Secton II)directly computed using the binomial distribution of re-cells are linear devices (see Section II).spne.

Our nonlinear detectors (NLD) are diagrammed in sponses.)Fig. 6. Each detector is partitioned into k nonoverlapping Now, let p' = Pr [NLD fires with "yes" response] =

gPr [half k-zones fire with rate > 0] = Pr [Xd > 0]. Thenzones, called k-zones. These k-zones should not be con-fue'ihteecttr n niioyznso h the probability that at least half of the k-zones fire iSfused with the excitatory and inhibitory zones of the

detector; the new k-zones are separated by borders orthog- -N/lk (N! t N/k-ional to the axis of orientation of the detector. Each of the P = E i PA( -P )i=N/2k+ 1k-zones behaves (has a firing function) like our originallinear detectors-its firing rate is the difference between the Then q" = 1- p" = Pr [NLD fires with "no"9numbers of "on" dots in its excitatory and inhibitory zones. response] = Pr [Xd . 0]. Then the probability of exactlyOur new nonlinear detector will fire with either a "yes"~ore rNLD's out of m firing in signal plus noise has the binomiala "no"~ response depending on whether or not more than distributionhalf of its k-zones fire with a rate greater than 0. What we m rm-are interested in determining is the probability that the 1,r)P qnumber of detectors firing with a "'yes"~response in signalplus noise is greater than the number of detectors firing For large values of m, we can approximate this distributionwith a "yes" response in noise alone. Just as in the averag- by N(mp",>lmp"q").

DAVIS et al.: MODELS FOR LINE DETECTION 133

Let Nm(Md,Ud) denote the distribution of "yes" responses but our data did not appear to be reliable. We hope that theof a sample of m NLD's in signal (d) plus noise (for noise publication of this paper will encourage others, betteralone, d = 0). Let Xd' be the random variable with distri- qualified in the design of perceptual experiments, to pursuebution N(jid,ad), and let X0 be the random variable with these ideas further.distribution N(yo,ao). Then Xdiff' = X0 - Xd' has the dis-tribution N(uo - ld,1 o002 + Cd2). Thus

The authors wish to thank Prof. Jacob Beck for manyPr [more NLD's fire "yes" in noise than in signal plus noise] helpful discussions.

= Pr [XO - Xd' > 0] REFERENCES= Pr [Xd ffI > 0]. [1] H. B. Barlow, R. Narasimhan, and A. Rosenfeld, "Visual pattern

recognition in machines and animals," Science, vol. 177, pp.567-575, Aug. 18, 1972.

A slightly different form of the non-linear model was pro- [2] C. Blakemore and F. W. Campbell, "On the existence in thegrammed: here an NLD would fire only if all of its k-zones human visual system of neurones selectively sensitive to the

fhsusP X > 0] orientation and size of retinal images," J. Physiol. London,fired. Fig. 7 shows graphs of signal (d) versus Pr [Xdiff' vol. 203, pp. 237-260, 1969.for NLD's of the same dimensions as used earlier and with [3] G. S. Brindley, Physiology of the Retina and Visual Pathway,three zones. Sample sizes in the vicinity of m = 100 yield [4

London: Edward Arnold, 1970.[4] L. S. Davis, A. Rosenfeld, and A. K. Agrawala, "On models forcurves similar to those in Fig. 4. line detection," Computer Science Center, University of Maryland,

Tech. Rep. 258, Aug. 1973.[5] S. Deutsch, "Conjectures on mammalian neuron networks for

IV. CONCLUSIONS visual pattern recognition," IEEE Trans. Syst. Sci. Cybern.,vol. SSC-2, pp. 81-85, Dec. 1966.

This paper has examined several alternative models for [6] U. T. Eysel and 0. J. Gruisser, "Neurophysiological basis ofpattern recognition in the cat's visual system, in Pattern Recog-

line detection in noise. In Section II, we saw that different nition in Biological and Technical Systems, 0. J. Grusser andmodels for the "simple" and "complex" cells in the cortex R. Klinke, Eds. New York: Springer Verlag, 1971.

lead to differentpredictionabouttheresponse[7] K. Fukushima, "Visual feature extraction by a multilayeredlead to different predictions about the responses to various network of analog threshold elements," IEEE Trans. Syst. Sci.stimuli that can be expected from these cells and that a non- Cybern., vol. SSC-5, pp. 322-333, Oct. 1969.

[8] D. H. Hubel and T. N. Wiesel, "Intergrative action in the cat'slinear model seems to be more consistent with the reported lateral geniculate body," J. Physiol., vol. 155, pp. 385-398, 1961.data than are the commonly assumed models. [9] , "Receptive fields, binocular interaction and functional

architecture in the cat's visual cortex," J. Physiol., vol. 160,Section III considered models for combining (linear or pp. 106-123.nonlinear) simple cell outputs to arrive at a detection [10] , "Receptive fields and functional architecture in two non-

striate visual areas of the cat," J. Neurophysiol., vol. 28, pp.decision. These models led to rather different bounds on the 228-289, 1965.numbers of cells required for a given detection performance. [11] , "Receptive fields and functional architecture of monkey

striate cortex," J. Physiol., vol. 195, pp. 215-243, 1968.It would be of interest to obtain experimental data from [12] G. Jacobs, "Receptive fields in visual systems," Brain Res., vol. 14,human subjects corresponding to the predictions of these pp. 553-573, 1969.

models. We havemadsomintiaattmptto[13] M. C. Kendall and A. Stuart, The Advanced Theory of Statistics,models. We have made some initial attempts to do so [4], vol. 2. New York: Hafner, 1967.