Low Level Visual Processing

Low Level Visual Processing

Information Maximization in the Retina

• Hypothesis: ganglion cells try to transmit as much information as possible about the image.

• What kind of receptive field maximizes information transfer?

Information Maximization in the Retina

• In this particular context, information is maximized for a factorial code:

• For a factorial code, the mutual information is 0 (there are no redundancies):

1

n

ii

P P r

R

1 2( , ,..., ) 0nI I r r r R

Information Maximization in the Retina

Independence is hard to achieve. Instead, we can look for a code which decorrelates the activity of the ganglion cells. This is a lot easier because decorrelation can be achieved with a simple linear transformation.

Information Maximization in the Retina

We assume that ganglion cells are linear:

The goal is to find a receptive field profile, Ds(x), for which the ganglion cells are decorrelated (i.e., a whitening filter).

*

s s

s

L a dxD x a s x

D s a

Information Maximization in the Retina

• Correlations are captured by the crosscorrelogram (all signals are assumed to be zero mean):

,LL LL s sQ a b Q a b L a L b

• The crosscorrelogram is also a convolution

s sdxdyD x a D x b s x s y

Fourier Transform

• The Fourier transform of a convolution is equal to the product of the individual spectra:

• The spectrum of a Dirac function is flat.

*h x f g x f x g d

H F G

Information Maximization in the Retina

• To decorrelate, we need to ensure that the crosscorrelogram is a Dirac function, i.e., its Fourier transform should be as flat as possible.

2LL LQ a b a b

2LL LQ k

Information Maximization in the Retina

2s s LdxdyD x a D x b s x s y a b

2 2s ss LD k Q k

Ls

ss

D kQ k

Information Maximization in the Retina

2 20

k

ss

eQ k

k k

Ls

ss

D kQ k

2 20

kLe k k

Information Maximization in the Retina

• If we assume that the retina adds noise on top of the signal to be transmitted to the brain, the previous filter is a bad idea because it amplifies the noise.

2E dk D k s k k s k

D k

• Solution: use a noise filter first, :

Information Maximization in the Retina

ss

ss

Q kD k

Q k Q k

Ls

ss

D k D kQ k

L ss

ss

Q k

Q k Q k

Information Maximization in the Retina

• The shape of the whitening filter depends on the noise level.

• For high contrast/low noise: bandpass filter. Center-surround RF.

• For low contrast/high noise: low pass filter. Gaussian RF

Information Maximization in the Retina

j

+

Information Maximization in the Retina

0 00 0

55 10 1015

1 1

2 2

3 3

44

temporal frequency (Hz) temporal frequency (H

Information Maximization beyond the Retina

• The bottleneck argument can only work once…

• The whitening filter only decorrelates. To find independent components, use ICA: predicts oriented filter

• Use other constrained beside infomax, such as sparseness.

Center Surround Receptive Fields

• The center surround receptive fields are decent edge detectors

+

Center Surround Receptive Fields

Feature extraction: Energy Filters

2D Fourier Transform

Frequency

Orientation

2D Fourier Transform

2D Fourier Transform

Motion Energy Filters

Space

Tim

e

Motion Energy Filters

• In a space time diagram 1st order motion shows up as diagonal lines.

• The slope of the line indicates the velocity

• A space-time Fourier transform can therefore recover the speed of motion

Motion Energy Filters

Motion Energy Filters

Motion Energy Filters

• 1st order motionT

ime

Space

Motion Energy Filters

• 2nd order motion

Motion Energy Filters

• In a space time diagram, 2nd order motion does not show up as a diagonal line…

• Methods based on linear filtering of the image followed by a nonlinearity cannot work

• You need to apply a nonlinearity to the image first

Motion Energy Filters

• A Fourier transform returns a set of complex coefficients:

( ) ji x

j

j j j

f x c e

c a ib

Motion Energy Filters

• The power spectrum is given by

2 2 2

2 2 2 2

,

cos sin

cos , sin

j

j j j j j j

i

j j j j j

j jj j

j j j j

c a ib c a b

c c e c i

a b

a b a b

2

jc

Motion Energy Filters

( )

( ) cos sin

( )cos

( )sin

i xj

j j

j

j

c f x e dx

f x x i x dx

a ib

a f x x dx

b f x x dx

Motion Energy Filters

2 2 2

( )cos

( )sin

j

j

j j j

a I x x dx

b I x x dx

c a b

cos

sin+

aj

bj

cjx2

x2

I

Motion Energy Filters

2 2 2

( )cos

( )sin

j

j

j j j

a I x x dx

b I x x dx

c a b

cos

sin+

aj

bj

cjx2

x2

I

t

x

Motion Energy Filters

• Therefore, taking a Fourier transform in space time is sufficient to compute motion. To compute velocity, just look at where the power is and compute the angle.

• Better still, use the Fourier spectrum as your observation and design an optimal estimator of velocity (tricky because the noise is poorly defined)…

Motion Energy Filters

• How do you compute a Fourier transform with neurons? Use neurons with spatio-temporal filters looking like oriented sine and cosine functions.

• Problem: the receptive fields are non local and would have a hard time dealing with multiple objects in space and multiple events in time…

Motion Energy Filters

• Solution: use oriented Gabor-like filters or causal version of Gabor-like filters.

• To recover the spectrum, take quadrature pairs, square them and add them: this is what is called an Energy Filter.

2

2( ) exp sin2x

f x x

Motion Energy Filters

x2

x2

+

From V1 to MT

• V1 cells are tuned to velocity but they are also tuned to spatial and temporal frequencies

t

x

From V1 to MT

• MT cells are tuned to velocity across a wide range of spatial and temporal frequencies

t

x

MT Cells

Pooling across Filters

• Motion opponency: it is not possible to perceive transparent motion within the same spatial bandwidth. This suggests that the neural read out mechanism for speed computes the difference between filters tuned to different spatial frequencies within the same temporal bandwidth.

Pooling across Filters

+ Flicker

Energy Filters

• For second order motion, apply a nonlinearity to the image and then run a motion energy filter.

Motion Processing: Bayesian Approach

• The energy filter approach is not the only game in town…

• Bayesian integration provides a better account of psychophysical results

Energy Filters: Generalization

• The same technique can be used to compute orientation, disparity, … etc.

Energy Filters: Generalization

• The case of stereopsis: constant disparity correspond to oriented line in righ/left RF diagram.

Energy Filters: Generalization

Energy Filters: Generalization

-50 0 50-1

-0.5

0

0.5

1

20 40 60 80 100

20

40

60

80

100

20 40 60 80 100

20

40

60

80

100

10 20 30 40

10

20

30

40

Orientation Selectivity

• At first sight: a simple, if not downright stupid, problem.

• Use an orientation energy filter

• The only challenge: finding out exactly how the brain does it…

• Two classes of models: feedforward and lateral connection models.

Orientation Selectivity: Feedforward (Hubel and Wiesel)

LGN

CTX

Orientation Selectivity: The Lateral Connection Model

LGN

CTX -+

Orientation Selectivity: Feedforward (Hubel and Wiesel)

--+

Orientation Selectivity

• Take a quadrature pair, rectifiy and square their outputs, sum and you get a complex cell tuned to orientation.

Orientation Selectivity

• Most people think this model is wrong, yet the evidence in its favor are overwhelming.– No further tuning over time– Aspect ratio consistent with this model– LGN input to layer 4 cells as tuned as output– LGN/Cortex connectivity as predicted by

feedforward model

• Why are there lateral connections?