Low Level Visual Processing

Low Level Visual Processing

Information Maximization in the Retina

• Hypothesis: ganglion cells try to transmit as much information as possible about the image.

• What kind of receptive field maximizes information transfer?


• In this particular context, information is maximized for a factorial code:

• For a factorial code, the mutual information is 0 (there are no redundancies):

1

n

ii

P P r

R

1 2( , ,..., ) 0nI I r r r R


Independence is hard to achieve. Instead, we can look for a code which decorrelates the activity of the ganglion cells. This is a lot easier because decorrelation can be achieved with a simple linear transformation.


We assume that ganglion cells are linear:

The goal is to find a receptive field profile, Ds(x), for which the ganglion cells are decorrelated (i.e., a whitening filter).

*

s s

s

L a dxD x a s x

D s a


• Correlations are captured by the crosscorrelogram (all signals are assumed to be zero mean):

,LL LL s sQ a b Q a b L a L b

• The crosscorrelogram is also a convolution

s sdxdyD x a D x b s x s y

Fourier Transform

• The Fourier transform of a convolution is equal to the product of the individual spectra:

• The spectrum of a Dirac function is flat.

*h x f g x f x g d

H F G


• To decorrelate, we need to ensure that the crosscorrelogram is a Dirac function, i.e., its Fourier transform should be as flat as possible.

2LL LQ a b a b

2LL LQ k


2s s LdxdyD x a D x b s x s y a b

2 2s ss LD k Q k

Ls

ss

D kQ k


2 20

k

ss

eQ k

k k

Ls

ss

D kQ k

2 20

kLe k k


• If we assume that the retina adds noise on top of the signal to be transmitted to the brain, the previous filter is a bad idea because it amplifies the noise.

2E dk D k s k k s k

D k

• Solution: use a noise filter first, :


ss

ss

Q kD k

Q k Q k

Ls

ss

D k D kQ k

L ss

ss

Q k

Q k Q k


• The shape of the whitening filter depends on the noise level.

• For high contrast/low noise: bandpass filter. Center-surround RF.

• For low contrast/high noise: low pass filter. Gaussian RF


j

+


0 00 0

55 10 1015

1 1

2 2

3 3

44

temporal frequency (Hz) temporal frequency (H

Information Maximization beyond the Retina

• The bottleneck argument can only work once…

• The whitening filter only decorrelates. To find independent components, use ICA: predicts oriented filter

• Use other constrained beside infomax, such as sparseness.

Center Surround Receptive Fields

• The center surround receptive fields are decent edge detectors

+

Center Surround Receptive Fields

Feature extraction: Energy Filters

2D Fourier Transform

Frequency

Orientation



Motion Energy Filters

Space

Tim

e


• In a space time diagram 1st order motion shows up as diagonal lines.

• The slope of the line indicates the velocity

• A space-time Fourier transform can therefore recover the speed of motion




• 1st order motionT

ime

Space


• 2nd order motion


• In a space time diagram, 2nd order motion does not show up as a diagonal line…

• Methods based on linear filtering of the image followed by a nonlinearity cannot work

• You need to apply a nonlinearity to the image first


• A Fourier transform returns a set of complex coefficients:

( ) ji x

j

j j j

f x c e

c a ib


• The power spectrum is given by

2 2 2

2 2 2 2

,

cos sin

cos , sin

j

j j j j j j

i

j j j j j

j jj j

j j j j

c a ib c a b

c c e c i

a b

a b a b

2

jc


( )

( ) cos sin

( )cos

( )sin

i xj

j j

j

j

c f x e dx

f x x i x dx

a ib

a f x x dx

b f x x dx


2 2 2

( )cos

( )sin

j

j

j j j

a I x x dx

b I x x dx

c a b

cos

sin+

aj

bj

cjx2

x2

I


2 2 2

( )cos

( )sin

j

j

j j j

a I x x dx

b I x x dx

c a b

cos

sin+

aj

bj

cjx2

x2

I

t

x


• Therefore, taking a Fourier transform in space time is sufficient to compute motion. To compute velocity, just look at where the power is and compute the angle.

• Better still, use the Fourier spectrum as your observation and design an optimal estimator of velocity (tricky because the noise is poorly defined)…


• How do you compute a Fourier transform with neurons? Use neurons with spatio-temporal filters looking like oriented sine and cosine functions.

• Problem: the receptive fields are non local and would have a hard time dealing with multiple objects in space and multiple events in time…


• Solution: use oriented Gabor-like filters or causal version of Gabor-like filters.

• To recover the spectrum, take quadrature pairs, square them and add them: this is what is called an Energy Filter.

2

2( ) exp sin2x

f x x


x2

x2

+

From V1 to MT

• V1 cells are tuned to velocity but they are also tuned to spatial and temporal frequencies

t

x

From V1 to MT

• MT cells are tuned to velocity across a wide range of spatial and temporal frequencies

t

x

MT Cells

Pooling across Filters

• Motion opponency: it is not possible to perceive transparent motion within the same spatial bandwidth. This suggests that the neural read out mechanism for speed computes the difference between filters tuned to different spatial frequencies within the same temporal bandwidth.

Pooling across Filters

+ Flicker

Energy Filters

• For second order motion, apply a nonlinearity to the image and then run a motion energy filter.

Motion Processing: Bayesian Approach

• The energy filter approach is not the only game in town…

• Bayesian integration provides a better account of psychophysical results

Energy Filters: Generalization

• The same technique can be used to compute orientation, disparity, … etc.


• The case of stereopsis: constant disparity correspond to oriented line in righ/left RF diagram.



-50 0 50-1

-0.5

0

0.5

1

20 40 60 80 100

20

40

60

80

100

20 40 60 80 100

20

40

60

80

100

10 20 30 40

10

20

30

40

Orientation Selectivity

• At first sight: a simple, if not downright stupid, problem.

• Use an orientation energy filter

• The only challenge: finding out exactly how the brain does it…

• Two classes of models: feedforward and lateral connection models.

Orientation Selectivity: Feedforward (Hubel and Wiesel)

LGN

CTX

Orientation Selectivity: The Lateral Connection Model

LGN

CTX -+

Orientation Selectivity: Feedforward (Hubel and Wiesel)

--+


• Take a quadrature pair, rectifiy and square their outputs, sum and you get a complex cell tuned to orientation.


• Most people think this model is wrong, yet the evidence in its favor are overwhelming.– No further tuning over time– Aspect ratio consistent with this model– LGN input to layer 4 cells as tuned as output– LGN/Cortex connectivity as predicted by

feedforward model

• Why are there lateral connections?

Documents

Low Level Visual Processing