Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the

Low Level Visual Processing

Information Maximization in the Retina

• Hypothesis: ganglion cells try to transmit as much information as possible about the image.

• What kind of receptive field maximizes mutual information I(s,r)?


• In this particular context, information is maximized for a factorial code:

• For a factorial code, the mutual information is 0 (there are no redundancies):

1

n

ii

P P r

r

1 2( , ,..., ) 0nI I r r r r


Independence is hard to achieve. Instead, we can look for codes that decorrelate the activity of the ganglion cells. This is a lot easier because decorrelation can be achieved with a simple linear transformation.


We assume that ganglion cells are linear:

The goal is to find a receptive field profile, Ds(x), for which the ganglion cells are decorrelated (i.e., a whitening filter).

*

s s

s

L a dxD x a s x

D s a


Intuition: take a white noise image and filter it with a gaussian filter. To decorrelate the resulting image, you need to deconvolve. The deconvolving filter would be a mexican hat (band pass) filter in this case.


• Correlations are captured by the crosscorrelogram (all signals are assumed to be zero mean):

,LL LL s sQ a b Q a b L a L b

• The crosscorrelogram is also a convolution (in 4D)

s sdxdyD x a D y b s x s y

Fourier Transform

• The Fourier transform of a convolution is equal to the product of the individual spectra:

• The spectrum of a Dirac function is flat.

*h x f g x f x g d

H F G


• To decorrelate, we need to ensure that the crosscorrelogram is a Dirac function, i.e., its Fourier transform should be as flat as possible.

2LL LQ a b a b

2LL LQ k


2s s LdxdyD x a D y b s x s y a b

2 2s ss LD k Q k

Ls

ss

D kQ k


2 20

k

ss

eQ k

k k

Ls

ss

D kQ k

2 20

kLe k k

Grows with k.. Amplifies noise!


• If we assume that the retina adds noise on top of the signal to be transmitted to the brain, the previous filter is a bad idea because it amplifies the noise.

2E dk D k s k k s k

D k

• Solution: use a noise filter first, :


ss

ss

Q kD k

Q k Q k

Ls

ss

D k D kQ k

L ss

ss

Q k

Q k Q k

Goes to 0 as k goes to infinity


• The shape of the whitening filter depends on the noise level.

• For high contrast/low noise: bandpass filter. Center-surround RF.

• For low contrast/high noise: low pass filter. Gaussian RF


j

+


0 00 0

55 10 1015

1 1

2 2

3 3

44

temporal frequency (Hz) temporal frequency (H

Information Maximization beyond the Retina

• The bottleneck argument can only work once…

• The whitening filter only decorrelates. To find independent components, use ICA: predicts oriented filter

• Use other constrained beside infomax, such as sparseness.

Information Maximization beyond the Retina

• One big problem with the previous theory: Receptive field size goes down for very low luminance level…

• Alternative: the retina tries to transmit edges.

Center Surround Receptive Fields

• The center surround receptive fields are decent edge detectors

+



• At high luminance: small RFs are fine because the SNR is high

• At intermediate luminance: large RFs are needed for signal averaging

• At very low luminance, larger RF would not be good because most of the info would be lost. Use small RFs instead and look for support in the cortex for contiguous contours

Feature extraction: Energy Filters

2D Fourier Transform

x

y


Frequency

Orientationx

y


x

y

Motion Energy Filters

Space

Tim

e


• In a space time diagram 1st order motion shows up as diagonal lines.

• The slope of the line indicates the velocity

• Therefore, a space-time Fourier transform can recover the speed of motion




• 1st order motionT

ime

Space


• 2nd order motion

http://www.psypress.com/mather/resources/swf/Demo11_9.swf


• In a space time diagram, 2nd order motion does not show up as a diagonal line…

• Methods based on linear filtering of the image followed by a nonlinearity cannot work

• You need to apply a nonlinearity to the image first


• A Fourier transform returns a set of complex coefficients:

( )

cos sin

ji x

j

j j j

j j j

I x c e

c x i x

c a ib


• The power spectrum is given by

2 2 2

2 2 2 2

,

cos sin

cos , sin

j

j j j j j j

i

j j j j j

j jj j

j j j j

c a ib c a b

c c e c i

a b

a b a b

2

jc


( )

( ) cos sin

( )cos

( )sin

ji x

j

j j

j j

j j

j j

c I x e dx

I x x i x dx

a ib

a I x x dx

b I x x dx


2 2 2

( )cos

( )sin

j j

j j

j j j

a I x x dx

b I x x dx

c a b

cos

sin+

aj

bj

cj2x2

x2

I


• Therefore, taking a Fourier transform in space time is sufficient to compute motion. To compute velocity, just look at where the power is and compute the angle.

• Better still, use the Fourier spectrum as your observation and design an optimal estimator of velocity (tricky because the noise is poorly defined)…


• How do you compute a Fourier transform with neurons? Use neurons with spatio-temporal filters looking like oriented sine and cosine functions.

• Problem: the receptive fields are non local and would have a hard time dealing with multiple objects in space and multiple events in time…


• Solution: use oriented Gabor-like filters or causal version of Gabor-like filters.

• To recover the spectrum, take quadrature pairs, square them and add them: this is what is called an Energy Filter.

2

2( ) exp sin2x

f x x


x ( )cosj ja I x x dxx2

Local cosine filter: Gabor filter


( )sinj jb I x x dxx x2

Local sine filter: Gabor filter


x2

x2

+

Neurons do not have negative activity: use quadrature pairs

V1 energy filters

t

x

Constant velocity

t

x

Bank of filters

y

x

From V1 to MT

• V1 cells are tuned to velocity but they are also tuned to spatial and temporal frequencies

t

x

2

2( , ) exp sin2 x t

x tr x t x t

From V1 to MT

t

x

Constant velocity

From V1 to MT

• MT cells are tuned to velocity across a wide range of spatial and temporal frequencies

t

x

MT Cells

Pooling across Filters

• Motion opponency: it is not possible to perceive transparent motion within the same spatial bandwidth. This suggests that the neural read out mechanism for speed computes the difference between filters tuned to different spatial frequencies within the same temporal bandwidth.

Pooling across Filters

t

+ Flicker x

Energy Filters

• For second order motion, apply a nonlinearity to the image and then run a motion energy filter.

Energy Filters: Generalization

• The same technique can be used to compute orientation, disparity, … etc.


• Stereopsis: constant disparities correspond to oriented lines in right/left RF diagram.


-50 0 50-1

-0.5

0

0.5

1

20 40 60 80 100

20

40

60

80

100

20 40 60 80 100

20

40

60

80

100

10 20 30 40

10

20

30

40


Orientation Selectivity

• At first sight: a simple, if not downright stupid, problem.

• Use an orientation energy filter

• The only challenge: finding out exactly how the brain does it…

• Two classes of models: feedforward and lateral connection models.

Orientation Selectivity: Feedforward (Hubel and Wiesel)

Pooled LGN afferences

LGN

Cortex

Orientation (deg)

-45 0 450

20406080

100

Act

ivity

spi

kes/

s

Orientation (deg)

-45 0 450

2468

10

Pooled LGN Tuning Curves

Orientation Selectivity: The Lateral Connection Model

Pooled LGN afferences

Output Tuning Curve

LGN

Cortex

Orientation (deg)

-45 0 450

20406080

100

Act

ivity

spi

kes/

s

Orientation (deg)

-45 0 450

2468

10



• How does it work? Simple consequence of Fourier transform…

Orientation Selectivity: Feedforward (Hubel and Wiesel)

--+


• To get a complex cell: take a quadrature pair, rectify, square and sum.


• The evidence in favor for the Hubel and Wiesel model are overwhelming– No further tuning over time– Aspect ratio consistent with this model– LGN input to layer 4 cells as tuned as output– LGN/Cortex connectivity as predicted by

feedforward model

• Why are there lateral connections?


• But which model is the most efficient?

Fisher information

• Fisher information, one neuron:

• For a population of neurons with independent noise:

2

2

fI

xyu

T rrw w

xyu

T rrw w

2

21 1

N Ni

ii ii

fI I

Current view

Common conclusion:

Increasing the slope leads to better discrimination (or more information)

Two ways to increase the slope:

1. Sharpening

2. Increase in gain

xyu

T rrw w

xyu

T rrw w

2

21

Ni

i i

fI

Main Problems

1- Neurons are not independent

2

21

Ni

i i

fI

xyu

T rrw w

1 1 1trTI f Q f' Q Q Q Q

Q: Covariance matrix of the noise

Source of the variability

Where does the noise come from?

• The variability comes from the dynamics of balanced recurrent networks

xyu

T rrw w

xyu

T rrw w

Source of the variability

• Tuning curves and variability are intertwined

xyu

T rrw w

• So what?

f1()

f2()

f3()

Changing tuning curves requires changing the synaptic weights

Changing the synaptic weights changes the noise

distribution


Output Tuning Curve

Cortex

Orientation (deg)

-45 0 450

20406080

100

Act

ivity

spi

kes/

s


Output Tuning Curve

LGN

Cortex

Orientation (deg)

-45 0 450

20406080

100

Act

ivity

spi

kes/

s

Orientation (deg)

-45 0 450

2468

10


Orientation (deg)

-45 0 450

2468

10

No-Sharpening Sharpening

LGN

2- Is this a bad representation of orientation?

2- Does sharpening help?

1- Are the output representations equally informative?1- Are the output representations equally informative?

3- Does sharpening help?


• The no-sharpening model extracts a lot more information

xyu

T rrw w

0

1

2

3

4

5

6

NS S

Info OUT

Info

rmat

ion

(deg

-2)


• Why are the models extracting different amount of information? It’s all in the output covariance matrices…

xyu

T rrw w

xyu

T rrw w

SNS

180

0

90180

90Preferred Orientation (deg) Preferr

ed O

rientat

ion (deg

)180

0

90180

90Preferred Orientation (deg) Preferr

ed O

rientat

ion (deg

)

Conclusions

• Fisher information: what matters is the ratio of the slope over the variance of the noise

• However, in the presence of correlations, increasing the slope does not necessarily translate into better representations.

Object Recognition

Nature of the problem

• Given an image, recognize (identify) the objects that are currently seen.

• The hard part: the same object can look incredibly different in different images due to differences in view points, illumination, color, occlusions,… etc.

• A robust recognizer must develop invariant recognition (translation, size, rotation?)

What do we know from imaging

• Are they specialized areas? Face and scene areas? Or areas for expertise?

• How distributed are the representations?

Match Filter/Nearest Neighbor

• Store images of the objects under different viewpoint, scale, orientation, etc.

• To recognize, find the template that matches the image the best.

• Optimal under a gaussian noise assumption which is clearly wrong in this case….

• Not invariant • Works surprisingly well… (especially with

tangent distance)

Object as nonlinear manifolds

• The set of points corresponding to the same object forms a nonlinear manifold in pixel space (of whatever space is used to represent the object)

• Tangent distance: compute distance to the tangent plane to the manifold.

• Tangent distance is not unlike relaxation in line attractor networks…

Wavelets Templates

• Filter the image with wavelets

• Store objects as vector of wavelets activation

• Slightly more invariant

Radial Basis Functions

• A set of functions is said to form a basis set if most other functions can be approximated as linear combinations of the basis functions.

• Ex: sine and cosine functions (Fourier transform) or gaussians (RBF).

1

n

i ii

f x c B x


-100 0 1000

50

100

150

200

250

Direction (deg)

Act

ivity

-200 -100 0 100 2000

0.2

0.4

0.6

0.8

1

Preferred Direction (deg)

Act

ivity


• Paper clip recognition. Parameterize the object with the angles between adjacent segments.

• Recognition is a nonlinear mapping from angle space onto identity.

• Use an RBF network to implement the mapping


Object 1

Object 2


Object 1

View 1

View Based Unit

Invariant unit forObject 1


Object 1

View 1

2 2 2

1 1 2 2 3 3

1 2 3 2, , exp

2

i i i

ia


• RBF network: not unlike nearest neighbours but can be applied to high level features, not just pixel values

• Illustrate the value of overcomplete set: good recognition can be performed with a simple linear computation


• Learning: use a gradient descent procedure to tune the network parameters

• The RBF units can be tuned with an unsupervised learning rule.


• Problems: how does the cortex extract angles?

• How does this generalize to other objects?

Vie w-tune d c e lls

M AX

we ig hte d su m

Sim p le c e lls (S1)

C o m p le x c e lls (C 1)

C o m p le x c o m p o site c e lls (C 2)

C o m p o site fe a ture c e lls (S2)


• Basis sets can be complete (sine) or overcomplete (Wavelets, RBF).


• Representation using complete sets are often very distributed (e.g. Fourier transform) and global. This means that they can be hard to decode and poor at dealing with multiple objects

• Overcomplete sets are memory intensive but they lead to sparse representations (linearly decodable), they work with multiple objects and they are efficient for learning.

Convolution Networks

LeNet• Multilayer networks with hierarchical organization

and built-in translation invariance • Fine to coarse analysis provide some scale and

rotation invariance, plus noise resistance.• No natural description of the features used for the

decomposition… Bad news for neurophysiologists…

Geons

• Two ideas:– Objects can be decomposed in terms of

combinations of simple shapes (geons)– Object representations involve a list of geons

and relative position in object-centered coordinates

• Finally an approach that does not turn the brain into a lookup table…

Geons

• Numerous problems:– No convincing experimental support and lots of

data against it – No good theory on how to build such

representations (current implementations cannot deal with real images)

– The list of Geons seems awfully arbitrary…

Documents

Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the