1Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

Computational Architectures in Biological Vision, USC, Spring 2001

Lecture 12. Visual Attention

Reading Assignments:

None

2Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

3Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

4Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

Several Forms of AttentionAttention and eye movements:

- overt attention (with eye movements)- covert attention (without eye movements)

Bottom-up and top-down control:- bottom-up control

based on image featuresvery fast (up to 20 shifts/s)involuntary / automatic

- top-down controlmay target inconspicuous locations in visual sceneslower (5 shifts/s or fewer; like eye movements)volitional

Control and modulation:- direct attention towards specific visual locations- attention modulates early visual processing at attended location

5Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

What is attention then?

Attention is often described as an information processing bottleneck.

Controls access to higher levels of processing, short-term memory and consciousness.

Hence, the strategy nature has developed to cope with information overload is to break down the problem of analyzing a visual scene:

from a massively parallel approach

to a rapid sequence of circumscribed recognitions.

6Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

7Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

8Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

9Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

10Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

11Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

12Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

13Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

14Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

15Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

16Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

First Computational Model

Koch & Ullman, Hum. Neurobiol., 1895

Introduce conceptof a single topo-graphic saliencymap.

Most salientlocation selectedby a winner-take-allnetwork.

17Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

18Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

Shifter CircuitsAnderson & van Essen, PNAS, 1987

Information dynamically routed throughcortical hierarchy. Yields rotation- andscale-independent representation.

19Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

Shifter Circuits (cont.)

Olshausen et al., J Neurosci, 1993

Implemented shifter circuits and demonstrated proof of concept.

Control neurons in the pulvinar send the (attention-based) control signals that will determine the �passing� region of the circuit, through a modulation of intracortical connection weights.

Perform recognitionusing associativememory at toplevel.

20Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

only attended item reaches output layer

21Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

Selective Tuning ModelTsotsos Tsotsos et al., Artificial Intelligence, 1995et al., Artificial Intelligence, 1995

-- attention modulates neurons to earliest levels; wherever thereattention modulates neurons to earliest levels; wherever there is a is a manymany--toto--one mappingone mapping

-- signal interference controlled by surround inhibition signal interference controlled by surround inhibition throughout processing network throughout processing network

-- task knowledge biases computations throughout task knowledge biases computations throughout processing networkprocessing network

-- attentionalattentional control is local, distributed and internalcontrol is local, distributed and internal

-- competition is based on WTA competition is based on WTA (different form than previous models)(different form than previous models)

-- pyramid representation with reciprocal pyramid representation with reciprocal convergence and divergence convergence and divergence

neuron �sees� thisneuron �sees� thisreceptive fieldreceptive field

subject �attends�subject �attends�to single itemto single item

22Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

attentional spotlight

effective receptive field of selected unit in unattended case

layers of input abstraction hierarchy

inhibitory attentional beam

"pass" zone

"inhibit" zone

The basic idea (BBS 1990)

23Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

Selective Tuning Model

processingpyramid

inhibited pathways

passpathways

unit of interestat top

input

Caputo & Guerra 1998Caputo & Guerra 1998Bahcall & Kowler 1999Bahcall & Kowler 1999Vanduffel, Tootell, Orban 2000Vanduffel, Tootell, Orban 2000Smith et al. 2000Smith et al. 2000

KastnerKastner, De Weerd, , De Weerd, Desimone, Ungerleider, Desimone, Ungerleider, 19981998

24Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

25Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

Guided SearchWolfe, Psychonomic Bull. & Rev., 1994

How can we combine information from several modalities? Use top-down (task-dependent) weighting.

26Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

27Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

28Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

29Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

30Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

31Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

32Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

33Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

34Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

35Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

36Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

37Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

38Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

39Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

40Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

41Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

42Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

43Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

Imag

e C

ompr

essi

on

44Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

Evaluation of Advertising

45Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

46Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

47Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

Brefczynski & DeYoe, N

ature Neuroscience 1999

48Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

49Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

Treu

e &

Mar

tinez

-Tru

jillo,

Nat

ure

1999

50Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

51Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

Attentional Modulation in Humans

Gandhi et al, 1999

52Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

53Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

54Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

55Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

56Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

57Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

58Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

59Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

60Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

61Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

62Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

Attentional ModulationHernandez et al.

Picture naming by bi-lingualpersons.

Increased attention/concentrationdue to increased difficultywhen non-native tongue?

Broca: speech generationSupramarginal: articulation &phonology processing

Cingulate: emotion, memory,vigilance, attention?

63Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

64Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention

65Laurent Itti: CS599 � Computational Architectures in Biological Vision, USC 2001. Lecture 12: Visual Attention