View
242
Download
0
Embed Size (px)
Citation preview
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Itti: CS564 - Brain Theory and Artificial Intelligence
University of Southern California
Lecture 28. Overview & Summary
Reading Assignment:
TMB2 Section 8.3
Supplementary reading: Article on Consciousness in HBTNN
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
You said “brain” theory??
First step: let’s get oriented!
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Major Functional Areas
Primary motor: voluntary movementPrimary somatosensory: tactile, pain, pressure, position, temp., mvt.Motor association: coordination of complex movementsSensory association: processing of multisensorial informationPrefrontal: planning, emotion, judgementSpeech center (Broca’s area): speech production and articulationWernicke’s area: comprehen-
sion of speechAuditory: hearingAuditory association: complex
auditory processingVisual: low-level visionVisual association: higher-level
vision
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Neurons and Synapses
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
http://www.radiology.wisc.edu/Med_Students/neuroradiology/fmri/
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Limbic System
Cortex “inside” the brain.Involved in emotions, sexual behavior, memory, etc(not very well known)
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Major Functional Areas
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Some general brain principles
Cortex is layered
Retinotopy
Columnar organization
Feedforward/feedback
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Layered Organization of Cortex
Cortex is 1 to 5mm-thick, folded at the surface of the brain (grey matter), and organized as 6 superimposed layers.
Layer names:1: Molecular layer2: External granular layer3: External pyramidal layer4: internal granular layer5: Internal pyramidal layer6: Fusiform layer
Basic layer functions:Layers 1/2: connectivityLayer 4: InputLayers 3/5: Pyramidal cell bodiesLayers 5/6: Output
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Retinotopy
Many visual areas are organized as retinotopic maps: locations next to each other in the outside world are represented by neurons close to each other in cortex.Although the topology is thus preserved, the mapping typically is highly non-linear (yielding large deformations in representation).
Stimulus shown on screen… and corresponding activity in cortex!
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Columnar Organization
Very general principle in cortex: neurons processing similar “things” are grouped together in small patches, or “columns,” or cortex.
In primary visual cortex… as in higher (object recognition) visual areas…
and in many, non-visual, areas as well (e.g., auditory, motor, sensory, etc).
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Felleman & Van Essen, 1991
Interconnect
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Neurons???
Abstracting from biological neurons to neuron models
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
The "basic" biological neuron
The soma and dendrites act as the input surface; the axon carries the outputs.
The tips of the branches of the axon form synapses upon other neurons or upon effectors (though synapses may occur along the branches of an axon as well as the ends). The arrows indicate the direction of "typical" information flow from inputs to outputs.
Dendrites Soma Axon with branches and
synaptic terminals
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Transmenbrane Ionic Transport
Ion channels act as gates that allow or block the flow of specific ions into and out of the cell.
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Action Potential and Ion Channels
Initial depolarization due to opening sodium (Na+) channels
Repolarization due to opening potassium (K+) channelsHyperpolarization happens because K+ channels stay open longer than Na+ channels (and longer than necessary to exactly come back to resting potential).
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
A McCulloch-Pitts neuron operates on a discrete time-scale, t = 0,1,2,3, ... with time tick equal to one refractory period
At each time step, an input or output is
on or off — 1 or 0, respectively.
Each connection or synapse from the output of one neuron to the input of another, has an attached weight.
Warren McCulloch and Walter Pitts (1943)
x (t)1
x (t)n
x (t)2
y(t+1)
w1
2
n
w
w
axon
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
From Logical Neurons to Finite Automata
AND
1
1
1.5
NOT
-10
OR
1
1
0.5
Brains, Machines, and Mathematics, 2nd Edition, 1987
X Y
Boolean Net
X
Y Q
Finite Automaton
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Leaky Integrator Neuron
The simplest "realistic" neuron model is a continuous time model based on using
the firing rate (e.g., the number of spikes traversing the axon in the most recent 20 msec.)
as a continuously varying measure of the cell's activity
The state of the neuron is described by a single variable, the membrane potential.
The firing rate is approximated by a sigmoid, function of membrane potential.
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Leaky Integrator Model
= - m(t) + h
has solution m(t) = e-t/ m(0) + (1 - e-t/)h
h for time constant > 0.
We now add synaptic inputs to get the
Leaky Integrator Model:
= - m(t) + i wi Xi(t) + h
where Xi(t) is the firing rate at the ith input.
Excitatory input (wi > 0) will increase
Inhibitory input (wi < 0) will have the opposite effect.
m(t)
m(t)
m(t)
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Models of what?
We need data to constrain the models
Empirical data comes from various experimental techniques:
PhysiologyPsychophysicsVarious imagingEtc.
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Electrode setup
- drill hole in cranium under anesthesia- install and seal “recording chamber”- allow animal to wake up and heal- because there are no pain receptors in brain, electrodes can then
be inserted & moved in chamberwith no discomfort to animal.
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Receptive field
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Example: yes/no task
Example of contrast discrimination using yes/no paradigm.
- subject fixates cross.- subject initiates trial by pressing space bar.- stimulus appears at random location, or may not appear at all.
- subject presses “1” for “stimulus present” or “2” for “stimulus absent.”
- if subject keeps giving correct answers, experimenter decreases contrast of stimulus (so that it becomes harder to see).
time
+
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Staircase procedure
Staircase procedure is a method for adjusting stimulus to each observer such as to find the observer’s threshold. Stimulus is parametrized, and parameter(s) are adjusted during experiment depending on responses.
Typically:- start with a stimulus that is very easy to see.
- 4 consecutive correct answers make stimulus more difficult to see by a fixed amount.- 2 consecutive incorrect answers make stimulus easier to see by a fixed amount.
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Example SPECT images
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Reconstruction using coincidence
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Example PET image
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
BOLD contrast
Bo Bo
with deoxygenated blood
with oxygenatedblood
The magnetic properties of blood change withthe amount of oxygenation
resulting in small signal changes
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Vascular System
arteries arterioles(<0.1mm)
capillaries venules(<0.1mm)
veins
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Oxygen consumpsion
O 2
O 2
O 2
O 2
O 2
O 2
O 2
O 2
O 2
O 2
O 2
O 2
O 2
O 2
O 2
O 2
The exclusive source of metabolic energy of the brain is glycolysis:
C6H12O6 + 6 O2 6 H2O + 6 CO2
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
BOLD Contrast
stimulation
neuronal activation
metabolic changes
hemodynamic changes
local susceptibility changes
MR-signal changes
data processing
functional image
signal detection
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Example of Blocked paradigm
Gandhi et al., 1999
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
First BOLD-effect experiment
Kwong and colleagues at Mass. General Hospital (Boston).
Stimulus: flashing light.
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Summary 2
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Case study: Vision
Vision is the most widely studied brain function
Our goals:
- analyze fundamental issues- Understand basic algorithms that may address those issues
- Look at computer implementations- Look at evidence for biological implementations- Look at neural network implementations
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Eye Anatomy
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Visual Pathways
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Retinal Sampling
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Origin of Center-Surround
Neurons at every location receive inhibition from neurons at neighboring locations.
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Origin of Orientation Selectivity
Feedforward model of Hubel & Wiesel: V1 cells receive inputs from LGN cells arranged along a given orientation.
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Oriented RFs
Gabor function:product of a grating anda Gaussian.
Feedforward model:equivalent to convolvinginput image by sets ofGabor filters.
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Cortical Hypercolumn
A hypercolumnrepresents one visuallocation, but manyvisual attributes.
Basic processing “module” in V1.
“Blobs”: discontinuitiesin the columnar structure.Patches of neurons concernedmainly with color vision.
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
From neurons to mind
A good conceptual intermediary between patterns of neural activity and mental events is provided by the schema theory
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
From Schemas to Schema Assemblages
The Famous
Duck-Rabbit
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Bringing in Context
For Further Reading:TMB2:Section 5.2 for the VISIONS system for schema-based interpretation of visual scenes.HBTNN: Visual Schemas in Object Recognition and Scene Analysis
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
A First “Useful” Network
Example of fully-engineered neural net that performsUseful computation: the Didday max-selector
Issues:
- how can we design a network that performs a given task
- How can we analyze non-linear networks
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Winner-take-all Networks
Goal: given an array of inputs, enhance the strongest (or strongest few) and suppress the others
No clear strong input yieldsglobal suppression
Strongest input is enhancedand suppresses other inputs
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Didday’s Model
= copy of input
= inhibitory inter-neurons
= receives excitationfrom foodness layerand inhibition fromS-cells
retinotopicinput
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
NN & Physics
Perceptrons = layered networks, weights tuned to learnA given input/output mapping
Winner-take-all = specific recurrent architecture for specific purpose
Now: Hopfield nets = view neurons as physical entities and analyze network using methods inspired from statistical physics
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Hopfield Networks
A Hopfield net (Hopfield 1982) is a net of such units subject to the asynchronous rule for updating one neuron at a time:
"Pick a unit i at random.
If wij sj i, turn it on.
Otherwise turn it off."
Moreover, Hopfield assumes symmetric weights:
wij = wji
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
“Energy” of a Neural Network
Hopfield defined the “energy”:
E = - ½ ij sisjwij + i sii
If we pick unit i and the firing rule (previous slide) does not change its si, it will not change E.
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
si: 0 to 1 transition
If si initially equals 0, and wijsj i
then si goes from 0 to 1 with all other sj constant,
and the "energy gap", or change in E, is given by
E = - ½ j (wijsj + wjisj) + i
= - ( j wijsj - i) (by symmetry)
0.
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
si: 1 to 0 transition
If si initially equals 1, and wijsj < i
then si goes from 1 to 0 with all other sj constant
The "energy gap," or change in E, is given, for symmetric wij, by:
E = j wijsj - i < 0
On every updating we have E 0
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Minimizing Energy
On every updating we have E 0
Hence the dynamics of the net tends to move E toward a minimum.
We stress that there may be different such states — they are local minima. Global minimization is not guaranteed.
B
C
A
Basin of
Attraction for C
D
E
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Attractors
1. The state vector comes to rest, i.e. the unit activations stop changing. This is called a fixed point. For given input data, the region of initial states which settles into a fixed point is called its basin of attraction.
2. The state vector settles into a periodic motion, called a limit cycle.
For all recurrent networks of interest (i.e., neural networks comprised of leaky integrator neurons, and containing loops), given initial state and fixed input, there are just three possibilities for the asymptotic state:
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Strange attractors
3. Strange attractors describe such complex paths through the state space that, although the system is deterministic, a path which approaches the strange attractor gives every appearance of being random.
Two copies of the system which initially have nearly identical states will grow more and more dissimilar as time passes.
Such a trajectory has become the accepted mathematical model of chaos,and is used to describe a number of physical phenomena such as the onset of turbulence in weather.
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
The traveling salesman problem 1
There are n cities, with a road of length lij joining city i to city j.
The salesman wishes to find a way to visit the cities that is optimal in two ways: each city is visited only once,
and the total route is as short as possible.
This is an NP-Complete problem: the only known algorithms (so far) to solve it have exponential complexity.
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Associative Memories
http://www.shef.ac.uk/psychology/gurney/notes/l5/l5.html
Idea: store:
So that we can recover it if presented with corrupted data such as:
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Associative memory with Hopfield nets
Setup a Hopfield net such that local minima correspondto the stored patterns.
Issues:-because of weight symmetry, anti-patterns (binary reverse) are stored as well as the original patterns (also spurious local minima are created when many patterns are stored)
-if one tries to store more than about 0.14*(number of neurons) patterns, the network exhibits unstable behavior
- works well only if patterns are uncorrelated
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Learning
All this is nice, but finding the synaptic weights that achieve a given computation is hard (e.g., as shown in the TSP example or the Didday example).
Could we learn those weights instead?
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Simple vs. General Perceptrons
The associator units are not interconnected, and so the simple perceptron has no short-term memory.
If cross-connections are present between units, the perceptron is called cross-coupled - it may then have multiple layers, and loops back from an “earlier” to a “later” layer.
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Linear Separability
A linear function of the form
f(x) = w1x1+ w2x2+ ... wdxd+wd+1 (wd+1 = - )
is a two-category pattern classifier.
f(x) = 0 w1x1+ w2x2+ ... wdxd+wd+1 =
gives a hyperplane as the decision surface
Training involves adjusting the coefficients (w1,w2,...,wd,wd+1) so that the decision surface produces an acceptable separation of the two classes.
Two categories are linearly
separable patterns if in fact
an acceptable setting of such
linear weights exists.
A
A
A
A
AA
AA
A
AA
A
A
A
A
A
AAA
AA
BB
BB
BB
B
BBB
B
B
B
B
B
BB
B
B
B
B
B
BB
Bx1
x2
f(x) 0
f(x) < 0
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Classic Models for Adaptive Networks
The two classic learning schemes for McCulloch-Pitts
formal neurons i wixi
Hebbian Learning (The Organization of Behaviour 1949)
— strengthen a synapse whose activity coincides with the firing of the postsynaptic neuron
[cf. Hebbian Synaptic Plasticity, Comparative and Developmental Aspects (HBTNN)]
The Perceptron (Rosenblatt 1962)
— strengthen an active synapse if the efferent neuron fails to fire when it should have fired;
— weaken an active synapse if the efferent neuron fires when it should not have fired.
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Hebb’s Rule
where synapse wij connects a presynaptic neuron with firing rate xj to a postsynaptic neuron with firing rate yi.
Peter Milner noted the saturation problem
von der Malsburg 1973 (modeling the development of oriented edge detectors in cat visual cortex [Hubel-Wiesel: simple cells])
augmented Hebb-type synapses with:
- a normalization rule to stop all synapses "saturating"
wi = Constant
- lateral inhibition to stop the first "experience" from "taking over" all "learning circuits:” it prevents nearby cells from acquiring the same pattern thus enabling the set of neurons to "span the feature space"
xj yi
The simplest formalization of Hebb’s rule is to increase wij by: wij = k yi xj (1)
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Perceptron Learning Rule
The best known perceptron learning rule - strengthens an active synapse if the efferent neuron fails to
fire when it should have fired, and - weakens an active synapse if the neuron fires when it should not have:
wij = k (Yi - yi) xj (2)
As before, synapse wij connects a neuron with firing rate xj to a neuron with firing rate yi, but now
Yi is the "correct" output supplied by the "teacher."
The rule changes the response to xj in the right direction: If the output is correct, Yi = yi and there is no change, wij = 0. If the output is too small, then Yi - yi > 0, and the change in wij will add wij xj = k (Yi - yi) xj xj > 0 to the output unit's response to (x1, . . ., xd).
If the output is too large, wij will decrease the output unit's response.
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Backpropagation: a method for training a loop-free network which has three types of unit:
input units;hidden units carrying an internal representation;output units.
Back-Propagation
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Example: face recognition
Here using the 2-stage approach:
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Non-Associative and Associative Reinforcement Learning
Non-associative reinforcement learning, the only input to the learning system is the reinforcement signal
Objective: find the optimal actionAssociative reinforcement learning, the learning system also receives information about the process and maybe more.
Objective: learn an associative mapping that produces the optimal action on any trial as a function of the stimulus pattern present on that trial.
[Basically B, but with new labels]
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Self-Organizing Feature Maps
Localized competition & cooperation yield emergent global mapping
o o o o o o o o o o o o o
o o o o o o o o o o o o o
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Capabilities and Limitations of Layered Networks
To approximate a set of functions of the inputs byA layered network with continuous-valued units andSigmoidal activation function…
Cybenko, 1988: … at most two hidden layers are necessary, with arbitrary accuracy attainable by adding more hidden units.
Cybenko, 1989: one hidden layer is enough to approximate any continuous function.
Intuition of proof: decompose function to be approximated into a sum of localized “bumps.” The bumps can be constructed with two hidden layers.
Similar in spirit to Fourier decomposition. Bumps = radial basis functions.
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Optimal Network Architectures
How can we determine the number of hidden units?
- genetic algorithms: evaluate variations of the network, using a metric that combines its performance and its complexity. Then apply various mutations to the network (change number of hidden units) until the best one is found.
- Pruning and weight decay:- apply weight decay (remember reinforcement
learning) during training- eliminate connections with weight below threshold- re-train
- How about eliminating units? For example, eliminate units with total synaptic input weight smaller than threshold.
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Large Network Example
Example of network with many cooperating brain areas:Dominey & Arbib
Issues:
- how to use empirical data to design overall architecture?
- How to implement?- How to test?
PP
SC
vm
sm
ms
FOn
FEF
vm
sm
vs
CD
sm
vs
SNR
vs
ms
qv
TH
qv
LGN
VisCx
wta
Brainstem Saccade Generator
eye movement
RetinaVisInput
delay
FEFvsFEFms
FEFvsFEFms
swit
chPPctr
delay
FOn
Filling in the Schemas: Neural Network Models Based on Monkey NeurophysiologyPeter Dominey & Michael Arbib: Cerebral Cortex, 2:153-175
Develop hypotheses on Neural Networks that yield an equivalent functionality: mapping schemas (functions) to the cooperative cooperation of sets of brain regions (structures)
FEF
CAUDATESC
SNr
SG
VisCx
PP
MD
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Low-Level Processing
Remember: Vision as a change in representation.
At the low-level, such change can be done by fairly streamlined mathematical transforms:
- Fourier transform- Wavelet transform
these transforms yield a simpler but more organized image of the input.
Additional organization is obtained through multiscale representations.
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Laplacian Edge Detection
Edges are defined as zero-crossings of the second derivative (Laplacian if more than one-dimensional) of the signal.
This is very sensitive to image noise; thus typically we first blur the image to reduce noise. We then use a Laplacian-of-Gaussian filter to extract edges.
Smoothed signal First derivative (gradient)
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Illusory Contours
Some mechanism is responsible for our illusory perception of contours where there are none…
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Long-range Excitation
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Modeling long-range connections
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Depth & Stereo
C
A B
D
L R
OA OA
-qmax
-qmax +qmax
+qmax
qRqL
qLB, qLD
q0 q0
qLA, qLC qRB, qRC
qRA, qRD
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Correspondence problem
Segment & recognize objects in each eye separately first,
then establish correspondence?
No! (at least not only): Julesz’ random-dot stereograms
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Regularization
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Higher Visual Function
Examine components of mid/high-level vision:
AttentionObject recognitionGistAction recognition
Scene understandingMemory & consciousness
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and SummaryItti & Koch, Nat Rev Neurosci, Mar. 2001
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Bre
fczynski &
De
Yo
e, Na
ture
Ne
uroscie
nce 1
999
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Tre
ue &
Mar
tinez
-Tru
jillo
, N
atur
e 19
99
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Challenges of Object Recognition
The binding problem: binding different features (color, orientation, etc) to yield a unitary percept. (see next slide)
Bottom-up vs. top-down processing: how much is assumed top-down vs. extractedfrom the image?
Perception vs. recognition vs. categorization: seeing an object vs. seeing is as something. Matching views of known objects to memory vs. matching a novel object to object categories in memory.
Viewpoint invariance: a major issue is to recognize objects irrespectively of the viewpoint from which we see them.
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Fusiform Face Area in Humans
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Eye Movements
1) Free examination
2) estimate material circumstances of family
3) give ages of the people
4) surmise what family hasbeen doing before arrivalof “unexpected visitor”
5) remember clothes worn bythe people
6) remember position of peopleand objects
7) estimate how long the “unexpectedvisitor” has been away from family
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Several Problems…
with the “progressive visual buffer hypothesis:”
Change blindness:
Attention seems to be required for us to perceive change in images, while these could be easily detected in a visual buffer!
Amount of memory required is huge!
Interpretation of buffer contents by high-level vision is very difficult if buffer contains very detailed representations (Tsotsos, 1990)!
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
The World as an Outside Memory
Kevin O’Regan, early 90s:
why build a detailed internal representation of the world?
too complex…not enough memory…
… and useless?
The world is the memory. Attention and the eyes are a look-up tool!
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
The “Attention Hypothesis”
Rensink, 2000
No “integrative buffer”
Early processing extracts information up to “proto-object” complexity in massively parallel manner
Attention is necessary to bind the different proto-objects into complete objects, as well as to bind object and location
Once attention leaves an object, the binding “dissolves.” Not a problem, it can be formed again whenever needed, by shifting attention back to the object.
Only a rather sketchy “virtual representation” is kept in memory, and attention/eye movements are used to gather details as needed
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Gist of a Scene
Biederman, 1981:
from very brief exposure to a scene (120ms or less), we can already extract a lot of information about its global structure, its category (indoors, outdoors, etc) and some of its components.
“riding the first spike:” 120ms is the time it takes the first spike to travel from the retina to IT!
Thorpe, van Rullen:
very fast classification (down to 27ms exposure, no mask), e.g., for tasks such as “was there an animal in the scene?”
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
One lesson…
From 50+ years of research…
Solving vision in general is impossible!
But solving purposive vision can be done. Example: vision for action.
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Grip Selectivity in a Single AIP Cell
A cell that is selective for side opposition (Sakata)
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
FARS (Fagg-Arbib-Rizzolatti-Sakata) Model Overview
AIP
F5
dorsal/ventral streams
Task Constraints (F6)
Working Memory (46)
Instruction Stimuli (F2)
Task Constraints (F6)Working Memory (46?)Instruction Stimuli (F2)
AIPDorsalStream:Affordances
IT
VentralStream:Recognition
Ways to grab this “thing”
“It’s a mug”PFC
AIP extracts the set of affordances for an attended object.These affordances highlight the features of the object relevant to physical interaction with it.
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary
Visual
Cortex
ParietalCortex
InferotemporalCortex
How (dorsal)
What (ventral)
reach programming
grasp programming
AT and DF: "How" versus "What"
“What” versus “How”: AT: Goodale and Milner: object parameters for grasp (How) but not for saying or pantomimingDF: Jeannerod et al.: saying and pantomiming (What) but no “How” except for familiar objects with specific sizes.
Lesson: Even schemas that seem to be normally under conscious control can in fact proceed without our being conscious of their activity.
Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary