Chapter VI. From the highest echelons of visual processing ...klab.tch.harvard.edu/academia/classes/Neuro230/... · 3 processing to cognition 4 5 Inferior temporal cortex (ITC) is

BiologicalandComputerVision GabrielKreiman©ChapterVI 2019

1

1Chapter VI. From the highest echelons of visual 2

processing to cognition 3 4Inferior temporal cortex (ITC) is the highest echelon within the visual 5

stream concerned with processing visual shape information. The Felleman and 6Van Essen diagram (Chapter I, Figure I-5) places the hippocampus at the top. 7While visual responses can be elicited in the hippocampus, people with bilateral 8lesions to the hippocampus can still see very well. A famous example is the patient 9known as H.M. This patient had no known visual deficit, but gave rise to the whole 10field of memory studies based on his inability to form new memories. The 11hippocampus is not a visual area and instead receives inputs from all other sensory 12modalities. 13

14The history of how inferior temporal cortex became accepted and 15

described as a visual area is rather fascinating (Gross, 1994), and follows the 16refinements in the ability to make more precise lesions and controlled behavioral 17experiments. In stark contrast to the hippocampus, bilateral lesions to ITC are 18associated with impairment in visual object recognition in macaque monkeys 19(Dean, 1976; Weiskrantz and Saunders, 1984; Afraz et al., 2015), and with several 20object agnosias in humans (Damasio, 1990; Humphreys and Riddoch, 1993; Forde 21and Humphreys, 1999) (Chapter IV) 22 236.1. A well-connected area 24 25 Inferior temporal cortex (ITC) consists of Brodmann’s cytoarchitectonic 26areas 20 and 21. It is a vast expanse of cortex that is usually subdivided into a 27posterior area (PIT), a central area (CIT) and an anterior area (AIT) (Felleman and 28Van Essen, 1991; Logothetis and Sheinberg, 1996; Tanaka, 1996). Biologists are 29fond of confusing people by using different names for the same thing, a 30phenomenon that can be partly explained by independent investigators working on 31related topics in parallel and coming up with new nomenclature to describe their 32findings. ITC is also referred to in the literature as areas TEO and TE. The degree 33of functional specialization among different parts of ITC remains poorly understood 34and it is extremely likely that we will have to subdivide ITC into many different 35subareas beyond the coarse division into posterior, central, and anterior, based on 36connectivity, neurophysiological and computational properties. 37 38 Like most other parts of cortex, the connectivity patterns of ITC are wide 39and complex (Markov et al., 2014). When we describe computational models of 40vision (Chapters VII-VIII), it is quite clear that most models represent a major 41simplification of the actual connectivity diagram. ITC receives feed-forward 42topographically organized inputs from areas V2, V3 and V4 along the ventral visual 43cortex. It also receives (fewer) inputs from areas V3A, MT, and MST, highlighting 44the interconnections between the dorsal and ventral streams. ITC projects back to 45V2, V3, and V4. There are also interhemispheric connections between ITC in the 46


2

right and left hemispheres through the main set of fibers connecting the two 47hemispheres, the corpus callosum. 48 49 ITC also has wide projections to and from non-visual regions including 50areas that provide critical inputs to the medial temporal lobe memory system such 51as the perirhinal cortex, parahippocampal gyrus, and entorhinal cortex, areas 52involved in processing emotions such as the amygdala, and areas in pre-frontal 53cortex that are relevant for decision making, planning, and working memory. Thus, 54from an anatomical standpoint, ITC is ideally situated to interpret visual inputs in 55the context of current goals and previous history, and convey this information to 56make behavioral decisions and create episodic memories. 57 586.2. ITC neurons show shape selectivity 59 60 A heroic school of investigators has been probing the responses of ITC 61neurons in monkeys over the last five decades. Most, if not all, ITC neurons show 62visually evoked responses, responding vigorously to color, orientation, texture, 63direction of movement, and shape. Posterior portions of ITC show a coarse 64retinotopic organization, and an almost complete representation of the 65contralateral visual field. The receptive field sizes are approximately 1.5 – 4 66degrees, and on average, they are larger than those found in V4 neurons. 67 68 As we move to more anterior locations along the ITC, there is weaker and 69weaker retinotopic organization. The receptive field sizes in more anterior parts of 70ITC are often large but there is a wide range of estimations in the literature ranging 71from some neurons with ~2 degrees receptive fields (DiCarlo and Maunsell, 2004) 72to descriptions of neurons with receptive fields that span several tens of degrees 73(Rolls, 1991; Tanaka, 1993). Most receptive fields in anterior ITC include the foveal 74region. 75 76 Investigators have described strong responses in ITC neurons elicited by 77all sorts of different stimuli. For example, several investigators have shown that 78ITC neurons can be driven by the presentation of faces, hands and body parts 79(Gross et al., 1969; Perrett et al., 1982; Rolls, 1984; Desimone, 1991; Young and 80Yamane, 1992). ITC neurons can also be driven to fire selectively by presenting 81different types of objects like fruits, cars, or toys (Hung et al., 2005a). 82 83 Example responses from 3 ITC neurons in response to 5 pictures are 84shown in Figure VI.1. In this figure, there are 10 repetitions of each picture and 85the neuronal capriciousness is evident in the heterogeneous pattern of responses 86from one trial to the next, as noted already for other neurons in Chapter V. Despite 87this trial-to-trial variability, there are some consistent features. All three neurons 88show an increased firing rate that commences approximately 100 ms after stimulus 89onset (approximately near the end of the white horizontal line denoting the stimulus 90presentation). This should not be interepreted as response triggered by the 91stimulus offset; if the stimulus duration were much longer, the neurons would still 92start to fire at around 100 ms after stimulus onset. These 100 ms reflect the latency 93


3

for all the computations that transpire thorughout the ventral visual cortex before 94these ITC neurons. The neurons are picky in their stimulus preferences. The “blue” 95neuron showed an increased response to the first two pictures (toy, food) than to 96the last two (synthetic rendering of a cat, car). In contrast, the red neuron showed 97an increased response to the third and fourth pictures (mokey face, cat). 98 99 Investigators have effectively tested a wide range of visual stimuli to study 100the responses of ITC neurons. For example, some studies have used parametric 101shape descriptors of abstract shapes (Schwartz et al., 1983; Miyashita and Chang, 1021988; Richmond et al., 1990). Logothetis and colleagues trained monkeys to 103recognize paperclips forming different 3D shapes and subsequently found neurons 104that were selective for paperclip 3D configurations (Logothetis and Pauls, 1995). 105 106 This wide range of response preferences might seem puzzling at first. 107Perhaps one would like to conjecture that an area that plays an important role in 108object recognition would have neurons that respond specifically to objects in the 109real world. There could be banana neurons (i.e. a neuron that responds if and only 110if investigators show a picture of a banana to the monkey), peanuts neurons, car 111neurons, chair neurons, face neurons, paperclip neurons, hand neurons, eye 112neurons, spaghetti with meatball neurons. Indeed, if we ignore momentarily the “if 113and only if” part, it is possible to find neurons activated selectively by all of these 114types of images. As illustrated by the examples in Figure VI.1, the responses are 115not all-or-none. ITC neurons do not seem to be activated only upon presentation 116of one specific type of object in the real world with baseline level responses to 117everything else. Rather, ITC neurons show graded activations with stronger 118responses to some stimuli compared to others. 119 120

Figure VI.1. ITC neurons are picky. Example responses from 3 neurons in inferior temporal cortex (labeld “Site 1”, “Site 2”, “Site 3”) to 5 different gray scale objects. Each dot represents a spike, each row represents a separate repetition (10 repetitions per object) and the horizontal white line denotes the onset and offset of the image (100 ms presentation time). Data from (Hung et al., 2005a).


4

It is unclear that ITC neurons show any special treatment to naturally 121occurring objects like chairs or faces. Instead, ITC neurons may represent a 122sufficiently reach dictionary of complex features. These features can be used to 123represent any number of naturally occurring objects in an analogous way to 124forming words by combining different letters, or sentences from words. Those 125features can be found in fractal patterns, in paperclips, in faces, and chairs. We 126will come back to a more quantitative description of these properties and the 127responses of ITC neurons when we describe current computational models of 128visual processing in Chapters VII and VIII. 129 130 As discussed in the case of neurons in earlier visual areas, there is a clear 131topography in the ITC response map. By advancing the electrode in a trajectory 132that is approximately tangential to cortex, investigators describe neurons that have 133similar tuning (Fujita et al., 1992; Gawne and Richmond, 1993; Tanaka, 1993; 134Kobatake and Tanaka, 1994). This level of organization can be represented by 135“columns” of neurons with similar preferences. Moving horizontally, neighboring 136neurons in ITC also show similar, but not identical preferences. 137 1386.3. Selectivity in human ventral visual cortex 139 140 Less is known about the internal machinery that processes visual 141information in the human brain. The main source of information to about the inner 142workings of human ventral visual cortex comes from invasive neurophysiological 143recordings in epilepsy patients. A fraction of patients with epilepsy can be treated 144pharmacologically. In cases of focal epilepsies that do not respond to current drug 145treatment, an important approach has been to surgically remove the epileptogenic 146

Figure VI.2. Human ITC also shows shape selectivity. Example electrode describing the physiological responses to 25 different exemplar objects belonging to 5 different categories. A. Responses to each of 25 different exemplars (each color denotes a different category of images; each trace represents the response to a different exemplar). B. Raster plot showing every single trial in the responses to the 5 face exemplars. Each row is a repetition, the dashed lines separate the exemplars, the color shows voltage (see scale bar on right). C. Electrode location.


5

focus. In most cases, this surgical procedure requires first carefully mapping brain 147activity to discern where seizures are coming from and also to ensure that the brain 148excisions will not interfere with future cognitive function. For this purpose, 149neurosurgeons typically implant electrodes inside the human brain. Because 150current non-invasive techniques are too coarse to map the origin of seizures, the 151neurosurgeons typically implant many tens of electrodes in different brain areas 152with the hope of pinpointing the seizure onset. The patients stay in the hospital for 153about one week or so, providing investigators with a rare and unique opportunity 154to scrutinize human brain function at very high spatiotemporal resolution and with 155very high signal-to-noise ratio compared to anything that can be done from outside 156the brain. 157 158 An example of visually selective responses in the human ITC is shown in 159Figure 6.2. The human intracranial field potential signals show many of the 160hallmarks of the macaque ITC. Signals along the human ventral visual cortex also 161show circumscribed receptive fields, which increase in size from the fovea to the 162periphery, and from one area to the next. Field potentials along the human ventral 163visual cortex are also selective and graded (Figure 6.2A). The field potential 164signals also show trial-to-trial variability, yet the visually evoked responses can be 165readily appreciated in single trials (Figure 6.2B). There have been many more 166neurophysiological studies scrutinizing responses in monkeys compared to 167humans and nobody has yet demonstrated selective responses in human ventral 168visual cortex to fractal patterns or paperclips as shown in monkey studies, but 169absence of evidence does not imply evidence of absence. Latencies in the human 170brain seem to be slightly longer than those in macaques, perhaps because of the 171larger brain size, perhaps because there might be more computational steps 172before information reaches human ITC. Within the scarce and preliminary 173neurophysiological evidence available today, and to a reasonable extent, many of 174the properties of the macaque ITC are recapitulated in the human ITC. 175 176 It should be noted that it is not entirely clear how to compare brain areas 177and functional responses between humans and monkeys (or any other pair of 178species separated by long evolutionary timespans). First of all, we should be 179cautious about comparing spikes in monkeys to field potential signals in humans. 180It turns out that field potential signals in the monkey ITC also show similar 181selectivity patterns to spikes. The coarser field potential responses are somewhat 182less picky than spikes in terms of their ability to distinguish different stimuli, 183perhaps due to the averaging over many neurons. A more challenging 184consideration involves establishing rigorous homologies between species. It 185seems pretty obvious that the eyes in monkeys are homologous to the human 186eyes. And, although the neuroanatomical connections in humans remain unclear, 187it is quite tempting to assume that the monkey primary visual cortex may be 188homologous to the human primary visual cortex. Beyond primary visual cortex, 189homologies become murkier. Regardless of whether we can establish a unique 190evolutionary sound one-to-one map between specific structures in different 191species, it is clear that human ventral visual cortex shows rapid and selective 192


6

responses to complex shapes that are qualitatively similar to those observed in 193monkeys. 194 1956.4. What do ITC neurons really want? 196

197 ITC neurons seem to respond to a wide variety of different shapes that 198investigators have used to probe their stimulus preferences. Recording time is 199limited and investigators need to make choices about which stimuli to use in an 200experiment. Typically, investigators choose stimuli based on a combination of 201inspiration from previous studies (if a certain type of stimulus worked before to 202drive neurons in a given area, it should work now too) or intuitions based on the 203prevalence of natural stimulus statistics (it seems logical to assume that neurons 204may represent the types of inputs that the animal experiences on a daily basis), or 205arguments about the presumed evolutionary importance of certain classes of 206stimuli. Additionally, important advances about neuronal tuning properties have 207been based on semi-serendipitous discoveries. 208 209 These types of experiments carry the potential biases injected by the 210investigators in selecting the stimuli: obviously, we can only find tuning for those 211stimuli that we probe. Even the title of this section has a strong anthropomorphic 212spin. Neurons do not really “want” anything. The question is meant to allude to 213what types of visual stimuli will maximally activate a given neuron (in the sense of 214

Figure VI.3. Letting neurons reveal their tuning preferences. An approach to investigate neuronal tuning in an unbiased manner. A generative neural network is used to create images by inverting a model of visual recognition (Chapter VII). The synthetic images are presented while recording neuronal activity. The neuronal responses are used as a fitness index to guide a genetic algorithm to select a new generation of improved images.


7

triggering more spikes). As emphasized earlier, the key difficulty to elucidate the 215response preferences of neurons involves the curse of dimensionality: given 216limited recording time, we cannot present all possible stimuli. A promising line of 217research to elucidate the feature preferences in inferior temporal cortex involves 218changing the stimuli in real-time dictated by the neuron’s preferences (Kobatake 219and Tanaka, 1994; Yamane et al., 2008). 220 221 More recent work suggests that we may need to rethink the neural code 222for features in ITC (and perhaps earlier visual areas as well). Following up on the 223ideas developed by Yamane et al (2008) to let the neuron itself reveal what it likes 224rather than impose a strong bias in the stimulus selection, Xiao and colleagues 225developed a computational algorithm that is capable of generating images guided 226by neuronal firing rates (Ponce et al 2019, Figure VI.3). They use a genetic 227algorithm using the neuron’s firing rate as the fitness function. In a given 228generation, the investigators probe the responses to a set of images. Images that 229trigger high firing rates are kept, and the rest are modified and recombined by the 230generative algorithm. In Chapter 8, we will introduce deep hierarchical models of 231vision that start with pixels and yield a high-level feature representation of the 232image. The generative algorithm deployed by Xiao and colleagues is essentially 233an inverted version of those computational models, starting with high-level features 234and ending up with the generation of pixels in an image. 235 236 By running this generative computational algorithm while recording the 237activity of a neuron in ITC, they discovered images that elicited higher firing rates 238than any natural image that had been used before to test the responses of the 239neurons. These images contain naturalistic combinations of textures and broad 240strokes, which have been described by investigators as impressionist renderings 241of abstract art like a Kandinsky. The fundamental novel concept here is that 242neurons may be optimally activated by combinations of complex features that 243cannot be easily described in words. In contrast to the language-based 244anthropomorphic descriptions of neuronal feature preferences in ITC (“this neuron 245likes faces”, “this neuron likes chairs”, “this neuron likes convex curved shapes”), 246the new line of work suggests that neurons might be optimally activated by 247complex shapes that defy a language-based definition. A rich basis set of neurons 248tuned to such complex features is capable of allowing the organism to discriminate 249real world objects, but the basis set does not have to be based on real-world 250objects. 251 2526.5. ITC neurons show tolerance to object transformations 253 254 As emphasized in Chapters I and III, a key property of visual recognition 255is the capacity to recognize objects in spite of the transformations of the images at 256the pixel level (Figure III-6). It is therefore interesting to ask whether the visual 257selectivity described in the previous two sections is maintained across image 258transformations. For example, would the first neuron in Figure 6.1 respond 259


8

selectively to those two first objects if they are shown at a different scale, at a 260different position with respect to fixation, a different color, etc.? 261 262 ITC neurons show a significant degree of tolerance to certain object 263transformations. For example, ITC neurons have larger receptive fields and 264therefore show more tolerance to object position changes compared to neurons in 265earlier parts of ventral visual cortex (Ito et al., 1995; Logothetis and Pauls, 1995; 266Hung et al., 2005c). ITC neurons also show similar responses in spite of large 267changes in the retinal size of the stimuli (Ito et al., 1995; Logothetis and Pauls, 2681995; Hung et al., 2005c). Tolerance does not necessarily imply that the firing rate 269in response to a given picture should be identical across different transformations. 270Even if the absolute firing rates are affected by the stimulus size, the rank order 271preferences among different objects, and therefore the relative stimulus 272preferences, are maintained (Ito et al., 1995). ITC neurons also show a certain 273degree of tolerance to depth rotation (Logothetis and Sheinberg, 1996) and even 274to the particular cue used to define their preferred shapes, such as whether the 275shape is defined by luminance, motion, or texture changes (Sary et al., 1993). 276 277 An extreme example of tolerance to object transformations was provided 278by recordings of single neuron responses from the medial temporal lobe in human 279epilepsy patients. Recording from the hippocampus, entorhinal cortex, amygdala 280and parahippocampal gyrus, investigators have found neurons that show 281responses to multiple objects within a semantically-defined object category 282(Kreiman et al., 2000b). They have also shown that some neurons show a 283remarkable degree of selectivity to individual persons or landmarks. For example, 284one neuron showed a selective response to images where the ex-president Bill 285Clinton was present, another neuron preferred pictures of the famous actress 286Jennifer Anniston. Remarkably, the images that elicited a response in these 287neurons were quite distinct from each other in terms of their pixel content ranging 288from a black/white drawings to color photographs with different poses and views 289(Quian Quiroga et al., 2005). Such extreme combination of selectivity and 290tolerance has not been described in ITC areas but rather in areas of the medial 291temporal lobe which receive visual inputs but are not strictly visual areas. In fact, 292damage to medial temporal lobe structures does not seem be associated with any 293obvious visual or other perceptual deficits but rather with memory problems. 294Therefore, it is likely that this combination of selectivity and tolerance reflects a 295read out of activity from a population of ITC neurons to transform sensory inputs 296into episodic memories. 297 2986.6. Neurons can complete patterns 299

300 During natural vision, objects are often only partially visible due to poor 301illumination or because there are other objects in front of them (Chapter 3.5). In 302early visual areas with small receptive fields, occlusion may cover the entire part 303of the visual field that a given neuron is interested in. In contrast, in higher visual 304areas with larger receptive fields, occlusion may only partially restrict the amount 305


9

of inputs to a given neuron. The degree of tolerance to object transformations 306described in the previous section suggests that neurons might potentially also 307tolerate inputs that only contain some of the preferred features. 308

309 Indeed, ITC shows a large degree of robustness to occlusion. The neural 310responses in ITC are able to complete patterns and maintain their selectivity even 311under conditions where more than half of the preferred object features are invisible 312(Tang et al 2014, 2018). Both at the behavioral level (Chapter 3.5), as well as at 313the neurophysiological level, pattern completion requires additional computation 314time: the visually selective neural latencies elicited by partially visible objects are 315about 50 milliseconds longer than those triggered by fully visible objects. These 316observations suggest the need for additional processing to make inferences from 317partial information (we will come back to this point in Chapter 8). 318

3196.7. IT takes a village 320 321 The observation that individual neurons can show a high degree of 322selectivity and tolerance to image transformations should not be taken to imply that 323there is a one-to-one map between a single neuron and recognition of a specific 324object. Sometimes, the idea of a one-to-one map between neurons and specific 325objects is erroneously referred to as the “grandmother cell” theory. A one-to-one 326system would be extremely fragile. In most cases, read-out neurons depend on 327inputs from thousands of other neurons and cannot be directly and reliably driven 328by a single input. Additionally, losing that one neuron might lead to an inability to 329recognize that particular object. As noted before, nearby neurons in visual cortex 330tend to show similar feature tuning properties; even if we cannot currently monitor 331the activity of every neuron in a local area, finding a neuron with a certain tuning 332function is quite likely to imply the existence of a large number of other nearby 333neurons with similar tuning properties. In fact, the idea of a “grandmother cell”, as 334coined by Jerry Letvin in 1969, referred to a whole population of cells with identical 335selectivity and tolerance properties (and in the original description, he referred to 336a “mother cell” rather than a “grandmother cell”). Understood as in the original 337definition, the idea of a grandmother cell, that is, a population of probably nearby 338neurons that show selectivity and tolerance to stimulus properties is a rather 339adequate description of neuronal tuning throughout the visual cortex. Retinal 340ganglion cells are grandmother cells for changes in illumination at sparse and 341specific locations in the visual field, primary visual cortex neurons are grandmother 342cells for oriented lines, and ITC neurons are grandmother cells for complex shape 343features (Kreiman, 2017). 344 345 While each neuron shows a preference for some shapes over others, the 346amount of information conveyed by individual neurons about overall shape is 347limited (Rolls, 1991). Additionally, there seems to be a significant amount of “noise” 348in the neuronal responses in any given trial. The term noise is used here in a rather 349vague way; we are referring to the trial-to-trial variability in the spike times and 350spike counts as noted in Figure 6.1. Whether this is real noise or part of the signal, 351


10

and the origin of 352this variability 353remains a topic of 354debate in the field. 355A post-synaptic 356neuron receiving 357inputs from a 358capricious pre-359synaptic neuron 360that emits a 361different response 362to presumably 363identical inputs 364needs to be able 365to decipher what 366is out there in the 367world in every 368trial. For 369

suprathreshold 370stimuli, perception 371is quite robust: 372you can look at 373the shape of the 374letter “A” a 375thousand times 376and it will always 377look like an “A”. 378 379 Can animals 380use the neuronal 381representation of 382a population of 383capricious ITC 384neurons to 385

discriminate 386among objects in 387single trials? It 388should be noted 389that the emphasis 390on single trials is 391critical here. The 392brain cannot 393average over 394trials, as many 395investigators do 396when they depict 397

Figure VI.4. Decoding population responses. Basic steps involved in training and testing a classifier. A: An illustration of an experiment in which an image of a cat and an image of a fish were shown in random order to a subject while simultaneous recordings were made from five neurons/channels. The grayscale level denotes the activity of each neuron/channel. B: Data points and the corresponding labels are randomly selected to be in either the training set or in the test set. C: The training data points and the training labels are passed to an untrained classifier that ‘learns’ which neural activity is useful at predicting which image was shown – thus becoming a ‘trained’ classifier. D: The test data are passed to the trained classifier which produces predictions of which labels correspond to each unlabled test data point. These predicted labels are then compared to the real test labels (i.e., the real labels that were presented when the test data were recorded) and the percent of correct predictions is calculated to give the total classification accuracy. From Meyers and Kreiman 2011.


11

their recordings (you do not need to look at the letter “A” above 10 times to be able 398to interpret). However, the brain is not constrained to making inferences from the 399activity of a single neuron. Any given neuron in cortex receives input from 400approximately 10,000 other neurons. Such a population could show interesting 401properties that ameliorate or eliminate the challenges associated with interpreting 402the output of a single neuron. Hung et al addressed this question by recording 403activity (sequentially) from hundreds of neurons in ITC and using machine learning 404classifiers to decode the activity of a pseudo-population of neurons in single trials 405(Hung et al., 2005b). Pseudopopulation refers to the notion that these neurons 406were not simultaneously recorded. 407 408 The machine learning decoding approach aims to learn a map between 409the activity patterns of a population of neurons in response to a set of images and 410the labels of objects in those images (Figure VI.4). For example, imagine that we 411present pictures of cats or pictures of dogs. Let jxi represent the response of neuron 412i to image j. For example, x could represent the total number of spikes emitted by 413the neuron in a given window. Due to the latency in ITC responses (Figure VI.1), 414we can consider a window between 100 and 300 ms after stimulus onset. The 415population response of N neurons is jx = [jx1, …, jxN]. If we imagine that all of these 416inputs might project to a given post-synaptic neuron, we can write the total 417aggregated input to the post-synaptic neuron as the weighted sum of all these 418inputs: w1 jx1 + … + wN jxN. We can think of those weights as a measure of the 419synaptic strength, the impact that a given input will have on a post-synaptic neuron. 420We can now build a detector that can read out from the population activity whether 421an image shown in a given trial contained a cat or a dog. We will set a threshold 422on the total combined inputs, g(w . x) for short, where g indicates a non-linear 423function like a sigmoid, w and x are the vectors defined above of dimension N, and 424the “.” represents a dot product. We can define that if g>threshold, the image 425contains a cat and if g<threshold, then the image contains a dog. Machine learning 426algorithms offer several astute ways of choosing those weights w to minimize the 427number of classification errors that the algorithm makes. This approach can be 428extended to many categories, not just binary classification. 429 430 Using this approach, Hung et al found that a relatively small group of ITC 431neurons (N ~200) could support object categorization quite accurately (up to ~90% 432accuracy in a task consisting of 8 possible categories). Furthermore, the pseudo-433population response could extrapolate across changes in object scale and 434position. In other words, it is possible to fit the w values using the responses x1 to 435images at a certain scale, and then subsequently use the responses x2 to images 436at a different scale. Thus, even if each neuron conveys only noisy information 437about shape differences, populations of neurons can be quite powerful in 438discriminating among visual objects in individual trials. 439

4406.8. ITC neurons are more concerned with shape than semantics 441 442


12

In the previous section, we considered whether it is possible to 443discriminate which object category was presented to the monkey by reading out 444neural activity from ITC. Instead of decoding the object category, it is also possible 445to ask which specific exemplar was presented to the monkey and a population of 446ITC neurons thrive at this question as well. Quantitatively comparing exemplar 447identification and categorization performance is a bit tricky because these two 448tasks are not equated in terms of difficulty. First, in the experiment discussed in 449the previous section, there were 8 categories and close to 80 exemplars. 450Therefore, even by chance, it is easier to get the object category right. This can be 451easily solved by randomly subsampling and picking only 8 exemplars. Yet, this 452does not quite address a more challenging problem in this type of comparison: it 453is easier to distinguish a picture of a face from a picture of a house than to 454distinguish between two houses. 455 456 Do ITC neurons carry any type of categorical information or is shape the 457only relevant variable of interest? One way to dissociate semantic categorical 458information from pure shape information is to consider objects that are physically 459similar but semantically distinct and vice versa. For example, a lemon is similar to 460a tennis ball in terms of its color, size and approximate shape. But a lemon is 461semantically closer to a watermelon or a tree and a tennis ball is semantically 462closer to a tennis racquet or a tennis court. There is no evidence to date that ITC 463neurons can link a tennis ball to a tennis court, or a lemon to a tree: ITC neuronal 464responses would be closer for the physically similar images. 465 466 An elegant series of experiments to tackle this question was conducted by 467Earl Miller’s group. They created synthetic images of cats and dogs and morphed 468between them in such a way that they could continuously change shape similarity 469without affecting categorical ownership or change category ownership with small 470changes in similarity. They found that ITC neuronal responses correlated with 471shape similarity rather than with categorical ownership. They also recorded 472responses from neurons in pre-frontal cortex, which is one of the targets of ITC 473neurons. In contrast with the ITC neurons, the responses of those pre-frontal 474cortex neurons did reflect categorical boundaries. 475 476 Another intriguing case where neuronal responses seemed to be 477dissociated from pure shape information is the case of those neurons recorded 478from the human medial temporal lobe discussed earlier (Section 6.5). Those 479neurons do seem to carry semantic information that transcends physical shape 480similarity, and those neurons receive either direct or indirect information from 481anterior ITC but they are not part of the ITC. 482 483 As repeatedly stated, absence of evidence should not be interpreted as 484evidence of absence. It is conceivable that there may be semantic information that 485can be dissociated from pure shape information in ITC but there is no clear 486evidence for this yet. 487 488


13

6.9. Neuronal responses adapt 489 490 The responses of ITC neurons, as those in earlier parts of visual cortex, 491are transient. If a stimulus is shown for a long time, say, many seconds, the 492neuronal responses still only last a few hundred milliseconds. Neurons throughout 493visual cortex are particularly sensitive to change and neuronal responses 494dynamically depend on the temporal context. Temporal context can dramatically 495alter visual experience (Chapter 3.8), as in the illusory perception of upward 496motion after fixating on a waterfall. 497 498 Adaptation is an evolutionary conserved property of visual processing as 499well as other sensory systems. One important function of adaptation is probably to 500save energy by reducing the number of spikes triggered by an unchanging 501stimulus. Yet, adaptation is also evident at much longer time scales than the 502presentation of a single stimulus. For example, exposure to an adapter stimulus 503leads to a reduction in the neural response to subsequent presentations of the 504same or similar stimuli, a phenomenon known as repetition suppression. This 505repetition need not necessarily be adjacent in time. Suppression is also evident 506even when there are other intervening stimuli, though the strength decreases with 507the time interval in between repetitions. At least partly, the biophysical mechanisms 508underlying such suppression may be due to intrinsic changes in a neuron through 509transient modulation of its membrane conductance. 510 511 Adaptation occurs throughout the visual system. The consequences of 512adaptation are stronger in higher areas like ITC compared to earlier neurons, 513perhaps due to the cumulative effects through a hierarchical cascade of neurons, 514each showing increasingly larger effects of adaptation that impact the next stage. 515Another effect that could contribute to the increased adaptation in higher stages is 516that earlier areas are more sensitive to small eye movements, hence reducing the 517similarity in the inputs for prolonged stimulus durations or repetitions of the same 518stimulus. 519 520

6.10. Representing visual information in the absence of a visual stimulus 521 522 Perceptually, prolonged exposure to a stimulus often leads to a temporarily 523reduced sensitivity for its features. The lingering effects after removal of the 524stimulus are called aftereffects, which have been described for a wide range of low 525to high-level visual stimulus properties and they are considered to be related to be 526a consequence of adaptation. In addition to aftereffects, exposure to a stimulus 527leaves a memory trace that allows subjects to remember what they have just seen. 528 529 One classical experiment used to study memory effects at short time 530scales is delayed match to sample tasks. Subjects are presented with an image; 531the image disappears and there is a delay of several seconds. Information stored 532on the order of a few seconds is typically referred to as working memory. After this 533delay, a second image is shown and subjects have to indicate whether the second 534


14

stimulus matches the first one (either because it is identical, or because it is a 535scaled or rotated version of the same object, or because they match in color, or 536any other property). Typically, the delay period consists of a blank screen. For 537subjects to be able to execute this task, neurons somewhere in the brain need to 538be able to maintain information about the preceding stimulus, even during the 539blank screen. It turns out that, although the responses of neurons in ITC are 540drastically reduced in the absence of visual stimulation, their responses do not go 541all the way back to baseline. Instead, ITC neurons maintain a small activation 542above baseline during the delay. Furthermore, this delay activity is stimulus 543selective: a neuron will maintain higher delay activity if its response to the 544preceding stimulus was higher. 545 546 Perhaps due to introspection, some investigators have interpreted those 547neuronal responses during the delay period in the absence of visual stimulation as 548an example of visual imagination. They argue that the subjects are imagining the 549sample stimulus to hold it in memory during the delay. To the extent that this is the 550case, it would seem that ITC neurons may show a selective response that matches 551the animal’s internally generated percept irrespective of sensory inputs. It is difficult 552to elicit volitional visual imagery in animals. In humans, several investigators have 553measured neuronal correlates of volitional imagery, but those responses have 554been investigated in the medial temporal lobe rather than in visual cortex (Kreiman 555et al., 2000a). Taking these ideas a step further, another situation where visual 556percepts can be generated in the absence of concomitant visual inputs is during 557dreams. Humans often report vivid visual imagery during dreams. Whether visual 558cortex is involved in the representation of those visual percepts remains to be 559determined. We will come back to this discussion in Chapter X. 560 561

6.11. The role of experience in shaping neuronal tuning preferences 562 563

We have described properties of the responses of ITC neurons as if they 564were static and immutable but this is far from the case. Neuronal tuning 565preferences are malleable and depend strongly on the diet of visual experience 566that the animal is subject to. The perennial debate between nature and nurture is 567prominent in the study of visual response tuning functions. The basic architecture 568of the visual system is largely dictated by genetic factors. Animals are born with 569basic visual structures including the eyes, the LGN, and the different cortical areas. 570While there are differences between species, the six-layered canonical 571connectivity within and between cortical areas seems to be either already present 572or formed shortly after birth. 573 574 In contrast, tuning properties are a consequence of experience. Several 575experiments going back to the seminal work of Hubel and Wiesel have shown that 576neuronal preferences in primary visual cortex are shaped by visual inputs. For 577example, monocular deprivation (i.e. covering inputs from one eye), leads to an 578expansion of neuronal preferences for the active eye at the detriment of neurons 579responding to inputs from the deprived eye. Dark rearing leads to impaired 580


15

orientation tuning throughout primary visual cortex. Furthermore, experiments 581where cats are reared in environments where they are predominantly exposed to 582vertical lines rather than horizontal lines lead to a preponderance of V1 neurons 583preferring vertical oriented bars rather than horizontal ones. 584 585 Given that even the initial steps in cortex depend on visual experience, it 586is perhaps less surprising that subsequent stages can also be modified by 587changing the statistics of the visual inputs. As mentioned earlier, neurons in 588macaque ITC can respond selectively to shapes like paper clips after the monkey 589is exposed to those images, it is clear that monkeys are not born with neurons that 590are tuned to the arbitrary shapes of paper clips. Furthermore, monkeys can be 591trained to recognize symbols, like numbers or letters; after training, ITC neurons 592can also respond selectively to those novel shapes, and again such tuning is not 593present at birth. The presumed ethological relevance of certain natural stimuli like 594faces has led some investigators to suggest that tuning for those shapes could be 595innate. However, careful experiments have refuted this hypothesis. If an animal is 596reared in an environment without any exposure to faces, then investigators do not 597find neurons tuned to faces. In sum, genetics provides the underlying architecture 598and the plasticity rules and the environment statistics lead to learning the tuning 599functions of neurons throughout visual cortex. 600 601

Neuronal tuning throughout visual cortex can be changed not only during 602development, but also in adults. The paperclip and numeric symbol experiments 603alluded to above were conducted in adult monkeys exposed to those novel images 604over periods of several months. 605

606Neuronal tuning can even be changed much more rapidly. For example, it 607

is likely that if we learn to recognize characters in a new language, or if we learn 608to recognize a new person, we would find changes in neuronal tuning in ITC. 609Indeed, elegant experiments in monkeys have shown that it is possible to alter the 610tuning properties of ITC neurons over the course of a recording session lasting 611less than an hour. 612 613

6.12. The bridge between vision and cognition 614 615 The studies discussed here constitute a non-exhaustive list of examples 616of the type of responses that investigators describe in the highest parts of inferior 617temporal cortex. While the field has acquired a considerable number of such 618examples, there is an urgent need to put together these empirical observations 619into a coherent theory of visual recognition, which will be the focus of the next 620chapters. 621 622 The methodology described in Section VI.4 provides initial steps towards 623unbiased ways of interrogating neuronal tuning functions in visual cortex. It is 624critical to develop more quantitative and systematic approaches to examine feature 625preferences in extrastriate visual cortex (this also applies to other sensory 626


16

modalities). Eventually, we should be able to describe a neuron’s preferences in 627quantitative terms, starting from pixels. What types of shapes would a neuron 628respond to? This quantitative formulation should allow us to make predictions and 629extrapolations to novel shapes. It is not sufficient to show stimulus A and A” and 630then interpolate to predict the responses to A’. If we could really characterize the 631responses of the neuron, we should be able to predict the responses to a different 632shape B. Similarly, as emphasized multiple times, feature preferences are 633intricately linked to tolerance to object transformations. Therefore, we should be 634able to predict the neuronal response to different types of transformations of the 635objects. Much more work is needed to understand the computations and 636transformations along ventral visual cortex. How do we go from oriented bars to 637complex shapes such as faces? A big step would be to take a single neuron in, 638say, ITC, be able to examine the properties and responses of its afferent V4 units 639to characterize the transformations from V4 to ITC. 640 641 This formulation presupposes that a large fraction of the ITC response is 642governed by its V4 inputs. However, we should keep in mind the complex 643connectivity in cortex and the fact that the ITC unit receives multiple other inputs 644as well (recurrent connections, bypass inputs from earlier visual areas, 645backprojections from the medial temporal lobe and pre-frontal cortex, connections 646from the dorsal visual pathway, etc.). There is clearly plenty of virgin territory for 647the courageous investigators who dare explore the vast land of extrastriate ventral 648visual cortex and the computations involved in processing shapes. Another area 649of active research that is still in its infancy and will require serious scrutiny in the 650near future is to further our understanding of how high-level visual information 651interfaces with the rest of cognition. 652

6536.13. Summary 654

655• Inferior temporal cortex (ITC) sits at the pinnacle of the visual cortical 656

hierarchy, receiving strong inputs from both ventral and dorsal cortical areas 657and projecting widely to areas involved in episodic memory formation, 658decision making, and cognitive control 659 660

• Monkey and human ITC neural responses are selective for a wide range of 661shapes including abstract patterns and natural objects like chairs or faces 662 663

• ITC neurons represent a large overcomplete dictionary of features, are 664more concerned with shape rather than semantics, and show invariance to 665image transformations 666 667

• ITC neurons can complete patterns from partially visible stimuli 668• Linear classifiers can be used to decode object information from the activity 669

of neural populations in ITC in single trials 670 671


17

• Neural responses continue representing selective visual information even 672in the absence of a visual stimulus 673

674• Neuronal tuning properties are the result of experience with the statistics of 675

the visual world 676 677 6786.1. References 679 680AfrazA,BoydenES,DiCarloJJ(2015)Optogeneticandpharmacologicalsuppression681

of spatial clusters of face neurons reveal their causal role in face gender682discrimination.ProceedingsoftheNationalAcademyofSciencesoftheUnited683StatesofAmerica112:6730-6735.684

Damasio A (1990) Category-related recognition defects as a clue to the neural685substratesofknowledge.TRENDSINNEUROSCIENCES13:95-98.686

Dean P (1976) Effects of inferotemporal lesions on the behavior of monkeys.687PsychologicalBulletin83:41-71.688

DesimoneR(1991)Face-selectivecellsinthetemporalcortexofmonkeys.Journalof689CognitiveNeuroscience3:1-8.690

DiCarlo JJ, Maunsell JHR (2004) Anterior Inferotemporal Neurons of Monkeys691Engaged in Object Recognition Can be Highly Sensitive to Object Retinal692Position.JournalofNeurophysiology89:3264-3278.693

FellemanDJ,VanEssenDC(1991)Distributedhierarchicalprocessingintheprimate694cerebralcortex.CerebralCortex1:1-47.695

FordeE,HumphreysG(1999)Category-specificrecognitionimpairments:areview696ofimportantcasestudiesandinfluentialtheories.Aphasiology13:169-193.697

Fujita I,TanakaK, ItoM,ChengK(1992)Columns forvisual featuresofobjects in698monkeyinferotemporalcortex.Nature360:343-346.699

Gawne TJ, Richmond BJ (1993) How independent are the messages carried by700adjacentinferiortemporalcorticalneurons?JournalofNeuroscience13:2758-7012771.702

Gross C, Bender D, Rocha-Miranda C (1969) Visual receptive fields of neurons in703inferotemporalcortexofthemonkey.Science166:1303-1306.704

GrossCG(1994)Howinferiortemporalcortexbecameavisualarea.CerebralCortex7055:455-469.706

Humphreys G, Riddoch M (1993) Object agnosias. BAILLIERES CLINICAL707NEUROLOGY2:339-359.708

HungC,KreimanG,PoggioT,DicarloJ(2005a)Ultra-fastobjectrecognitionfromfew709spikes.In.Boston:MIT.710

HungC,KreimanG,Quian-QuirogaR,KraskovA,PoggioT,DiCarloJ(2005b)Using711'read-out' of object identity to understand object coding in the macaque712inferiortemporalcortex.In:Cosyne.SaltLakeCity.713

HungCP,KreimanG,PoggioT,DiCarlo JJ (2005c)FastRead-outofObject Identity714fromMacaqueInferiorTemporalCortex.Science310:863-866.715

ItoM,TamuraH,FujitaI,TanakaK(1995)Sizeandpositioninvarianceofneuronal716responsesinmonkeyinferotemporalcortex.JNeurophysiol73:218-226.717


18

KobatakeE,TanakaK(1994)Neuronalselectivitiestocomplexobjectfeaturesinthe718ventralvisualpathwayofthemacaquecerebralcortex.JNeurophysiol71:856-719867.720

Kreiman G (2017) A null model for cortical representations with grandmothers721galore.Language,CognitionandNeuroscience32:274-285.722

KreimanG, Koch C, Fried I (2000a) Imagery neurons in the human brain. Nature723408:357-361.724

Kreiman G, Koch C, Fried I (2000b) Category-specific visual responses of single725neuronsinthehumanmedialtemporallobe.NatureNeuroscience3:946-953.726

LogothetisNK,PaulsJ(1995)Psychophysicalandphysiologicalevidenceforviewer-727centeredobjectrepresentationsintheprimate.CerebralCortex3:270-288.728

Logothetis NK, Sheinberg DL (1996) Visual object recognition. Annual Review of729Neuroscience19:577-621.730

MarkovNTetal.(2014)AWeightedandDirectedInterarealConnectivityMatrixfor731MacaqueCerebralCortex.CerebralCortex24:17-36.732

MiyashitaY,ChangHS(1988)Neuronalcorrelateofpictorialshort-termmemoryin733theprimatetemporalcortex.Nature331:68-71.734

PerrettD,RollsE,CaanW(1982)Visualneuronesresponsivetofacesinthemonkey735temporalcortex.ExperimentalBrainResearch47:329-342.736

Quian Quiroga R, Reddy L, Kreiman G, Koch C, Fried I (2005) Invariant visual737representationbysingleneuronsinthehumanbrain.Nature435:1102-1107.738

RichmondBJ,OpticanLM,SpitzerH(1990)Temporalencodingoftwo-dimensional739patternsbysingleunitsinprimateprimaryvisualcortex.I.Stimulus-response740relations.JournalofNeurophysiology64:351-369.741

Rolls E (1991) Neural organization of higher visual functions. Current Opinion in742Neurobiology1:274-278.743

RollsET(1984)Neuronsinthecortexofthetemporallobeandintheamygdalaofthe744monkeywithresponsesselectiveforfaces.HumanNeurobiology3:209-222.745

SaryG,VogelsR,OrbanGA(1993)Cue-invariantshapeselectivityofmacaqueinferior746temporalneurons.Science260:995-997.747

SchwartzE,DesimoneR,AlbrightT,GrossC(1983)Shape-recognitionandinferior748temporalneurons.PNAS80:5776-5778.749

TanakaK(1993)Neuronalmechanismofobjectrecognition.Science262:685-688.750Tanaka K (1996) Inferotemporal cortex and object vision. Annual Review of751

Neuroscience19:109-139.752Weiskrantz L, Saunders R (1984) Impairments of visual object transforms in753

monkeys.Brain:ajournalofneurology107:1033-1072.754YamaneY, CarlsonET,BowmanKC,WangZ, ConnorCE (2008)Aneural code for755

three-dimensional object shape in macaque inferotemporal cortex. Nature756Neuroscience11:1352-1360.757

Young MP, Yamane S (1992) Sparse population coding of faces in the inferior758temporalcortex.Science256:1327-1331.759

760 761NOTES: 762Describe Will’s work – Or move this to after computational models? 763


19

Describe work on visual search à move to computational models? 764Attention 765Task modulation 766Conclusion: need models!!! à next chapter 767 768 769Topography 770Yet, this does not mean a lack of topography. On the contrary, nearby neurons 771share similar properties: for example, two nearby neurons are much more likely to 772respond in a similar fashion to a set of stimuli than neurons that are farther apart 773(Tanaka, 1996). 774 775 776 777

Documents

Chapter VI. From the highest echelons of visual processing ...klab.tch.harvard.edu/academia/classes/Neuro230/... · 3 processing to cognition 4 5 Inferior temporal cortex (ITC) is