The Extraction of 3D Shape in the Visual System of Human and Nonhuman Primates

NE34CH16-Orban ARI 13 May 2011 10:21

The Extraction of 3D Shape inthe Visual System of Humanand Nonhuman PrimatesGuy A. OrbanLaboratorium voor Neuro-en Psychofysiologie, KU Leuven Medical School, Leuven,Belgium; email: [email protected]

Annu. Rev. Neurosci. 2011. 34:361–88

First published online as a Review in Advance onMarch 29, 2011

The Annual Review of Neuroscience is online atneuro.annualreviews.org

This article’s doi:10.1146/annurev-neuro-061010-113819

Copyright c© 2011 by Annual Reviews.All rights reserved

0147-006X/11/0721-0361$20.00

Keywords

cerebral cortex, functional imaging, action potential, depth, motion

Abstract

Depth structure, the third dimension of object shape, is extracted fromdisparity, motion, texture, and shading in the optic array. Gradient-selective neurons play a key role in this process. Such neurons occur inCIP, AIP, TEs, and F5 (for first- or second-order disparity gradients), inMT/V5, in FST (for speed gradients), and in CIP and TEs (for texturegradients). Most of these regions are activated during magnetic reso-nance scanning in alert monkeys by comparing 3D conditions with the2D controls for the different cues. Similarities in activation patterns ofmonkeys and humans tested with identical paradigms suggest that likegradient-selective neurons are found in corresponding human corticalareas. This view gains credence as the homologies between such areasbecome more evident. Furthermore, 3D shape-processing networks aresimilar in the two species, with the exception of the greater involvementof human posterior parietal cortex in the extraction of 3D shape frommotion. Thus we can begin to understand how depth structure is ex-tracted from motion, disparity, and texture in the primate brain, but theextraction of depth structure from shading and that of wire-like objectsrequires further scrutiny.

361

Ann

u. R

ev. N

euro

sci.

2011

.34:

361-

388.

Dow

nloa

ded

from

ww

w.a

nnua

lrev

iew

s.or

gby

Uni

vers

ity o

f A

berd

een

on 0

9/20

/13.

For

per

sona

l use

onl

y.


Gradient: a derivativeof a quantitativeproperty, such asspeed, in the retinalimage(s) along an axisin the image

Depth structure:shape in the thirddimension

Contents

INTRODUCTION . . . . . . . . . . . . . . . . . . 3623D Shape and Depth . . . . . . . . . . . . . . . 362Depth Structure of Surfaces and

Depth Orders. . . . . . . . . . . . . . . . . . . 363PERCEPTION OF 3D SHAPE . . . . . . 365EXTRACTION AND PROCESSING

OF DEPTH STRUCTURE INTHE MONKEY MODEL . . . . . . . . . 365A 3D Shape-Processing Network

in the Monkey . . . . . . . . . . . . . . . . . . 365Experimental Strategy. . . . . . . . . . . . . . 368Disparity-Gradient Selective

Neurons in the IntraparietalSulcus . . . . . . . . . . . . . . . . . . . . . . . . . . 369

Speed-Gradient Selective Neuronsin the Posterior SuperiorTemporal Sulcus . . . . . . . . . . . . . . . . 372

Extraction of Depth Structurein the Ventral Stream . . . . . . . . . . . 374

Overview of Monkey Studies . . . . . . . 375HUMAN FUNCTIONAL IMAGING

STUDIES OF 3D SHAPEPROCESSING. . . . . . . . . . . . . . . . . . . . 376Parallel Human and Monkey

fMRI Studies . . . . . . . . . . . . . . . . . . . 376Other Human fMRI Studies . . . . . . . . 3783D Shape Processing in Human

and Monkey . . . . . . . . . . . . . . . . . . . . 379TWO TYPES OF 3D SHAPE

EXTRACTION . . . . . . . . . . . . . . . . . . . 379BEYOND THE EXTRACTION

OF 3D SHAPE . . . . . . . . . . . . . . . . . . . . 379

INTRODUCTION

Although we live in a three-dimensional (3D)world, surprisingly little is known about theprocessing of 3D shape in the primate brain.We move within a 3D environment, weinteract with 3D objects, including animalsand conspecifics, and the processing of visual3D information can be of vital importance, ifone is, say, walking along a cliff or fighting offa tiger. My interest in 3D shape arose from

interactions, during the mid-1990s, with twoeminent scientists, Jan Koenderink and OlivierFaugeras, in the framework of interdisciplinaryEU projects (the Insight series). In this article,I review the progress made these 15 yearsin understanding the first steps of 3D shapeprocessing. Results show that the brain extractsrepresentations from the visual array that mapdirectly onto a real-world object property, here3D shape, as predicted by Gibson (1950). Thequantities extracted from the optic array areprimarily the gradients of visual parametersthat covary with changes in depth along 3Dsurfaces, most notably disparity and speedgradients for the stereo and motion cues,respectively. The timeliness of this review isfurther underscored by the recent develop-ments in the movie and television industry,introducing 3D movies such as Avatar.

3D Shape and Depth

Three-dimensional shape is defined by theboundary edges and surfaces of a 3D object. Be-cause projection onto the retina loses explicitdepth information that must be recovered bysubsequent cerebral processing, it is custom-ary to subdivide visual 3D shape into a fronto-parallel component, the 2D shape, and a depthcomponent, the depth structure (Durand et al.2007). Because the processing of 2D shape isubiquitous throughout the dorsal and ventralvisual streams (see Kourtzi & Connor 2011,this issue), this article addresses the extractionof depth structure from the visual array. Depthstructure is nothing more than the variation ofdepth along the surface or edge of an object.Hence, many of the depth cues also provide in-formation about depth structure.

The most sensitive depth cue is stereopsis,which exploits the small differences in relativehorizontal positions of retinal images in thetwo eyes, a consequence of the slightly differentviewpoints of the two eyes. Monocular depthcues include motion, texture, and shading.The motion cue may be compounded becauseretinal speed depends on distance from theeye, and direction of motion on the retina

362 Orban

Ann

u. R

ev. N

euro

sci.

2011

.34:

361-

388.

Dow

nloa

ded

from

ww

w.a

nnua

lrev

iew

s.or

gby

Uni

vers

ity o

f A

berd

een

on 0

9/20

/13.

For

per

sona

l use

onl

y.


3D SFD: 3D shapefrom disparity

3D SFM: 3D shapefrom motion

3D SFT: 3D shapefrom texture

3D SFS: 3D shapefrom shading

reverses at the fixation point (Roy et al. 1992).Other depth cues, such as blur or occlusion,provide less information about depth structure.This review concentrates on the four cuespertaining to the depth structure of surfaces,stereopsis, motion, texture, and shading,which we refer to as 3D shape from disparity(3D SFD; see Supplemental Figures 1and 2. Follow the Supplemental Materiallink from the Annual Reviews home page athttp://www.annualreviews.org), from mo-tion (3D SFM; Supplemental Videos 1–4),from texture (3D SFT; Supplemental Figure3), and from shading (3D SFS; SupplementalFigure 3). Notice that the first two cues alsoapply to thin elongated 3D objects (e.g., awire) or edges of a surface, supplemented byperspective as a third cue. Closure constitutesa possible fourth cue for edge-like structures,which may be processed in early extrastriateareas (S. Sunaert, J. Todd, and G. Orban,unpublished observations).

Depth Structure of Surfacesand Depth Orders

The depth structure of a surface is basically de-fined by the variation of depth along the surfaceand hence is intimately related to nonzero depthorders (Figure 1a–c). Zero-order depth is sim-ply range or distance relative to the observeror the fixation point. It includes both a quali-tative aspect, whether one surface is in front ofanother, and a quantitative aspect, i.e., the dis-tance between two surfaces, i.e., depth intervals(Anzai & DeAngelis 2010). First-order depthis the derivative of depth along an axis in thefront-parallel plane. It reflects 3D orientation,i.e., orientation in depth, specified by two pa-rameters: tilt and slant, respectively indicatingthe direction and the degree by which the planarsurface deviates from the fronto-parallel plane.Second-order depth, or depth curvature, is thederivative of 3D orientation along an axis of thefronto-parallel plane. This derivative specifies asurface curved in depth, with the orientation,sign, and degree of curvature as parameters.Such a singly curved surface is but one member

of a larger family of second-order surfaces de-fined by curvature in two orthogonal directions,the so-called quadratic surfaces. Koenderink(1990) proposed shape index and curvednessas the two parameters of quadratic or second-order surfaces. The shape index ranges from −1to +1, with values of −0.5 or 0.5 correspondingto ridges, values between −0.5 and 0.5 to sad-dles, and values smaller than −0.5 or larger than0.5 to dimples or bumps (Figure 1d). Thesesurfaces need orientation as an additional pa-rameter to be fully described because ridges andsaddles can have different orientations in thefronto-parallel plane. Higher-order depth cor-responds to variations in the curvature in depth.

First-order depth is ambiguous with respectto depth structure insofar as it may signal the 3Dorientation of a bounding surface of a 3D ob-ject, such as a pyramid, or simply of a planar ob-ject, such as a disc. Second-order depth is nec-essarily related to depth structure. Moreover,for stereo, it is invariant with distance from theeye, enhancing its attractiveness as a descriptorof depth structure. The importance of second-order depth for 3D shape was not well appre-ciated by Marr (1982), who considered mainly3D surface orientation and its discontinuities.It is worth noting that discontinuities in eitherzero- or first-order depth correspond to edgesof objects, not to surfaces.

Depth orders specify only local depth struc-ture of surfaces. Hence this information needsto be combined with size information to de-fine the depth structure of an object. Small-to-medium-sized surfaces (up to 10◦–15◦) mostlikely belong to 3D objects; larger surfaces de-scribe the 3D layout of the environment. Atthe other end of the size scale, 3D textureis defined by nonzero-depth orders on a verysmall spatial scale. Notice that in first approx-imation the extraction of depth structure as-sumes that the 3D object or the 3D surfaceis rigid. So far, very little work has addresseddeforming surfaces. Notice also that specifyingthe layout of 3D surfaces within the 2D shapedefined by the occluding edges does not ex-haust the volumetric perception of 3D objects,which may also include the processing of the 3D

www.annualreviews.org • 3D Shape in the Visual System 363

Supplemental Material

Ann

u. R

ev. N

euro

sci.

2011

.34:

361-

388.

Dow

nloa

ded

from

ww

w.a

nnua

lrev

iew

s.or

gby

Uni

vers

ity o

f A

berd

een

on 0

9/20

/13.

For

per

sona

l use

onl

y.

http://www.annualreviews.org/doi/suppl/10.1146/annurev-neuro-061010-113819



e

a

c

b

0

20

40

60

80

100

Anticorrelation Correlation

Monkey H

Corr

ect r

espo

nses

(%)

0

20

40

60

80

100Monkey Cd

–1.0

–0.5

0

+0.5C

+1.0 S

Figure 1Stimulus definitions for 3D shape and behavioral evidence in the monkey. (a–c): orders of depth: zero- (a),first- (b), and second- (c) order surfaces seen from the left, red dot: fixation point, on the right in panels b andc, zero-order and first-order approximations. (d ) Shape index (S, polar angle) and curvedness (C, radialdimension) of quadratic surfaces as defined by Koenderink (1990). (e) Behavior:% correct for discriminationof convex and concave disparity-defined surfaces in correlated and anticorrelated stereograms ( Janssen et al.2000b).

364 Orban

Ann

u. R

ev. N

euro

sci.

2011

.34:

361-

388.

Dow

nloa

ded

from

ww

w.a

nnua

lrev

iew

s.or

gby

Uni

vers

ity o

f A

berd

een

on 0

9/20

/13.

For

per

sona

l use

onl

y.


orientation and its derivative at their occludingedges.

Stereopsis is the most sensitive cue for depthstructure, and it also applies to all depth ordersand recovers the sign of depth curvature. How-ever, its distance from the subject over which itoperates is restricted, a limitation far less rele-vant for motion. Motion is almost as sensitive asstereopsis but provides ambiguous informationabout the sign of curvature. Texture providesexcellent first-order depth information aboutsurfaces, which shading does not do. Shading,on the other hand, provides clear second-orderdepth information, although its sign dependsstrongly on the position of the light source(Hanazawa & Komatsu 2001).

PERCEPTION OF 3D SHAPE

Numerous psychophysical studies have shownthat humans perceive 3D shape well (forreview, see Todd 2004) using any of the fourcues (Norman et al. 2004). Several tasks havebeen used to study 3D shape perception, butone that stands out is a 3D shape adjustmenttask (Georgieva et al. 2009, Koenderinket al. 2001). Subjects view a randomly de-formed sphere (a potato; see SupplementalFigure 3) onto which a horizontal line is su-perimposed. The subject’s task is to manipulatea comparison horizontal line until its profilematches as precisely as possible the perceivedrelief along the horizontal line on the potato.Line height generally correlates very closelywith the actual relief, but the slope of a linefitted to the data is typically less than one, indi-cating that subjects underestimate the amountof relief (Todd 2004). Three-dimensionalshape from motion and disparity develops asearly as two and four months after birth, respec-tively (Arterberry & Yonas 2000, Yonas et al.1987). Sensitivity for pictorial cues develops afew months later (Tsuruhara et al. 2009).

Because we use the macaque as an animalmodel, it is important to verify that this speciesperceives 3D shape. This is difficult to estab-lish, but several psychophysical tests indicatethat monkeys achieve thresholds for detecting

or discriminating 3D shapes similar to thoseof humans. Janssen et al. (2003) showed thatmonkeys can discriminate convex and concavesingly curved disparity-defined surfaces, datathat have been confirmed in subsequent ex-periments using singly (Figure 2 and Supple-mental Figure 4) or doubly curved (Verhoefet al. 2010) surfaces. Moreover, anticorrelatedstereograms produced no 3D shape perception(Figure 1e) ( Janssen et al. 2003), as in humans(Cumming et al. 1998). Monkeys also perceive3D structure from motion and detect thecoherence of a 3D rotating cylinder even morequickly than do humans (Siegel & Andersen1990). Zhang & Schiller (2008) confirmedthese findings using another task, in which themonkey had to indicate the point protrudingfurthest from the background. Tsutsui et al.(2002) demonstrated that monkeys can cross-match the 3D orientations of texture-definedand disparity-defined surfaces and can gener-alize to new orientations and texture patterns.Figure 2c and Supplemental Figure 4 confirmthat monkeys generalize from disparity-definedto texture-defined singly curved surfaces, whendiscriminating convex from concave surfaces,even though performance levels were relativelylow. The responsiveness of monkeys to thetexture cue has also been demonstrated in in-fant monkeys (Gunderson et al. 1993). Finally,Zhang et al. (2007), using an oddity task, haveprovided evidence that monkeys can also per-ceive 3D shape from shading, although perfor-mance was poorer in monkeys than in humans.Thus the available evidence suggests macaquemonkeys process 3D shape, particularly frommotion or disparity, and combine cues (Schilleret al. 2011) in ways similar to humans.

EXTRACTION AND PROCESSINGOF DEPTH STRUCTURE IN THEMONKEY MODEL

A 3D Shape-Processing Networkin the Monkey

I tentatively propose the following network(Figure 3) for processing 3D shape in the



Ann

u. R

ev. N

euro

sci.

2011

.34:

361-

388.

Dow

nloa

ded

from

ww

w.a

nnua

lrev

iew

s.or

gby

Uni

vers

ity o

f A

berd

een

on 0

9/20

/13.

For

per

sona

l use

onl

y.



a

c

Flat Concave Convex

b

–10

10

30

50

70

90

110Disparity + texture (human; n = 8)

Corr

ect r

espo

nses

(%)

Disparity + texture (M2)

0

4

8

12

16

20

24

d

0 10 20 4030 50 60 70 80

Disparity only (n = 56)

Neu

ral r

espo

nses

(spi

kes/

s)

Stereo coherence (%)

e

0 10 20 4030 50 60 70 80

0 10 20

In agreement

In conflict

4030 50 60 70 80

Stereo coherence (%)0 10 20 4030 50 60 70 80

Disparity + texture (n = 56)

Figure 2Mixing disparity and texture for second-order depth in behavior and TEs (stereo part of TE) neurons.(a) Stimuli: example of flat, single-curved concave and convex texture-defined surfaces; (b, c) % correct indiscrimination of convex and concave surfaces defined by texture and disparity in agreement or conflict fordifferent % disparity coherences (0% disparity coherence indicates only texture cue, red ) for eight humansubjects (b) and one monkey subject (M2, c); (d, e) average response of 56 TEs neurons, recorded in monkeysM1 and M2 (see here and in Supplemental Figure 4. Follow the Supplemental Material link from theAnnual Reviews home page at http://www.annualreviews.org), to disparity-defined surfaces for preferred(blue) and nonpreferred ( purple) curvature as a function of stereo coherence for surfaces defined only bydisparity (d ) and surfaces defined by both disparity and texture (e) in agreement (solid lines) and in conflict(dashed lines). Vertical bars indicate standard errors. Red and dark yellow symbols in panel e indicateresponses to texture-only preferred and nonpreferred curvature, respectively; arrow in d points to thecoherence threshold for discrimination (see Supplemental Figure 4), arrows in c and e point to texture cueonly conditions. Unpublished data from Y Liu, R Vogels, and GA Orban.

366 Orban

Ann

u. R

ev. N

euro

sci.

2011

.34:

361-

388.

Dow

nloa

ded

from

ww

w.a

nnua

lrev

iew

s.or

gby

Uni

vers

ity o

f A

berd

een

on 0

9/20

/13.

For

per

sona

l use

onl

y.



V2vV3v

V4v

V4dV3d

V2d

V3A

V1

V1

PITd

FST

MSTvMT/V5

V4t

MTp

Horizontal meridian

Eccentricity ridgeVertical meridians

STS

IOS

LuS

IPS

LaS

OTS

PG

MSTd

STPm

LSTSTPa

TEO TE

PIPLIP

CIP

AIP

PFG

PF

Opt

V6

MIP

TEs

MSTd

D

A

a

b

V1

V6

V3A

MT/V5

V2

FST

V4

CIP

PIP

AIP F5a

TE

STPa

TEO?

* * * *

* **

*

?

TEs

3 mm

Figure 3Monkey 3D shape areas: location and proposed network. (a) Flat map showing the visual system of themonkey, including indications of retinotopic areas ( yellow) with borders defined by meridians, as indicated,and other cortical regions (orange). Abbreviations: IPS, intraparietal sulcus; LaS, lateral sulcus; LuS, lunatesulcus; OTS, occipito-temporal sulcus; STS, superior temporal sulcus. (b) Network of visual areas showingmain connections. Blue-shaded boxes indicate areas processing 3D shape; question marks indicate a putativearea near TEO extracting disparity gradients and a part of STPa (anterior part of superior temporalpolysensory region) that may process speed gradients. Dashed arrows indicate weak or uncertain functionalconnections. Whereas texture and disparity gradients seem to be extracted by the same neurons in the caudalintraparietal area (CIP), segregated populations may extract depth structure from disparity, texture, andshading in or near TEO.


Ann

u. R

ev. N

euro

sci.

2011

.34:

361-

388.

Dow

nloa

ded

from

ww

w.a

nnua

lrev

iew

s.or

gby

Uni

vers

ity o

f A

berd

een

on 0

9/20

/13.

For

per

sona

l use

onl

y.


CIP: caudalintraparietal area

AIP: anteriorintraparietal area

MT/V5: middletemporal area or V5

FST: the floor of thetemporal sulcus area

IPS: intraparietalsulcus

STS: superiortemporal sulcus

TEO and TE:two neighboringarchitectonic fields,which togetherconstitute the infero-temporal cortex (IT)of monkeys; they likelycontain multiplecortical areas

ITG: inferiortemporal gyrus

macaque brain, hypothesizing that depth struc-ture is extracted at three points in the visualsystem. First, depth structure is extracted fromdisparity and texture in caudal intraparietalarea (CIP) (Sakata et al. 2005), an area dealingwith the depth structure of both objects and theenvironment, hence the wide range of surfacesizes it handles. The CIP may not use shadingas a cue because slanted surfaces are importantfor the layout of the environment and shadingprovides no first-order depth information.Depth-structure information related to objectsis further relayed to the anterior intraparietalarea (AIP), then to F5a, to which the AIPprojects (Borra et al. 2008). In this process,quantitative information gains importance forguiding prehension, explaining the dominanceof disparity, the more sensitive cue, overtexture. Second, depth structure is extractedfrom motion in two steps involving gradient-selective neurons in the middle temporal area(MT)/V5 and the floor of the temporal sulcusarea (FST) (Mysore et al. 2010b). The motioninformation, which is relatively precise, isprojected to the AIP in the intraparietal sulcus(IPS), to the anterior part of the superior tem-poral polysensory region (STPa) in the superiortemporal sulcus (STS), and probably to partsof the TE. Finally, depth structure is extractedfrom texture and shading in ventral and dorsalTEO. This extraction may not be based on agradient mechanism, as in the CIP or MT/V5-FST, but on an analysis of the 2D orientationdistribution, represented in V4. Indeed Flem-ing et al. (2004) have shown that the locationsand sizes of those distributions’ peaks provideinformation about the minimum slant in textureand the minimum curvature in shading. Thisalso may help explain why shading is apparentlyprocessed only in the ventral pathway. TheFST and TEO are very close to one another,so perhaps some yet-unidentified region in thisintermediate stage of the ventral stream (see ?in Figure 3) extracts depth structure from theremaining cue, stereo, using a gradient-basedmechanism. This region located near TEOwould then send signals into the lower bank ofthe STS and particularly to TEs, where depth

structure from stereo is represented. It mayalso correspond to the caudal inferior temporalgyrus (ITG) region in humans, where all threestatic cues are processed (see below).

Experimental Strategy

How does one address a problem as complexas the extraction of depth structure from theoptic array? Single-cell studies and functionalimaging are quite complementary because theyoperate at different spatio-temporal scales.Indeed, single-cell recordings are well suitedto demonstrate the analytical power of corticalneurons by documenting their selectivitiesfor stimulus parameters or features. However,this technique samples neuronal activity in avery local and often biased manner. Functionalmagnetic resonance imaging (fMRI) is gen-erally used to localize perceptual or cognitivefunctions across the entire brain or large por-tions of it by testing for significant differencesbetween experimental and control conditions,either in clusters of single voxels or in patternsof activity over many voxels. fMRI is verysensitive and capable of detecting small activitydifferences between conditions, but it does notreflect neuronal preferences in the sense ofparameter selectivity. Indeed, selective neuronswithin an area will prefer all parameter values,and pooling over many neurons yields an MRsignal largely independent of parameter value.By using these techniques in tandem in themonkey, where both techniques are now well-established (Logothetis et al. 1999, Vanduffelet al. 2001), one takes maximum advantage ofthese complementarities. We are ultimatelyinterested in the human brain, in which directsingle-cell recordings are difficult to obtain(Fried et al. 1997). Functional imaging is usedroutinely in humans. Its findings are difficultto relate to single-cell recordings in monkeys,however, because it amounts to solving anequation with two unknowns: technique andspecies. By adding functional imaging in themonkey, we reduce the problem to two equa-tions, each with a single unknown (Figure 4):technique, comparing single cells and fMRI

368 Orban

Ann

u. R

ev. N

euro

sci.

2011

.34:

361-

388.

Dow

nloa

ded

from

ww

w.a

nnua

lrev

iew

s.or

gby

Uni

vers

ity o

f A

berd

een

on 0

9/20

/13.

For

per

sona

l use

onl

y.


in the monkey, and species, comparing fMRIin awake humans and monkeys (see sidebar,Parallel Functional Imaging). Thus fMRIin the alert monkey not only complementssingle-cell studies, but also serves to establishthe link with the human work.

Disparity-Gradient Selective Neuronsin the Intraparietal Sulcus

Neurons selective for first-order disparity gra-dients were first described in the CIP by Shikataet al. (1996), but proof of higher-order selectiv-ity had to wait until Taira et al. (2000) showedthat selectivity was maintained despite changesin fixation distance. These neurons were sub-sequently observed in a delayed match to sam-ple (DMS) task with 2-s intervals between testand sample. Muscimol injections in the CIP im-paired performance on this task in 50% of thecases (Tsutsui et al. 2001). It is unclear whetherthis finding indicates that stereoscopic analysisin the CIP is critical for surface orientation per-ception or that the CIP is critical for bridgingthe interval between the test and the sample. In-deed, in human imaging (Cornette et al. 2002),ventral or parietal regions were activated in asame-different task for short (300 ms) or longintervals (6 s), respectively. In support of therole of the CIP in short-term memory, a sub-sequent study (Tsutsui et al. 2003) showed thatCIP neurons remain selective for 3D surfaceorientation during the 2-s delay between sam-ple and test stimuli.

CIP neurons are also selective for first-ordertexture gradients (Tsutsui et al. 2002), withmatched preferences for texture and disparity(Figure 5a). In this study, investigators usedlarge textured surfaces (30◦ diameter), muchlarger than stimuli used initially (Taira et al.2000) to investigate disparity selectivity (6.3◦

squares). A similar convergence between first-order selectivities for texture- and disparity-defined surfaces was observed by Liu et al.(2004) in TEs. The small stimuli (5◦ by 5◦),having various 2D outlines used to accommo-date TEs neurons’ selectivity for 2D shape, still

Monkey single cells

HomologiesNeural basis fMRI

Scouting

Monkey fMRI Human fMRI

Figure 4Experimental strategy: the triadic approach integrating monkey and humanstudies through fMRI in the alert monkey. Note that the link betweensingle-cell activity and monkey fMRI refers to the relationship betweenmagnetic resonance paradigms and neuronal selectivity.

portrayed large 3D surfaces because these werestrongly slanted. Thus, the texture stimuli usedin the CIP and TEs suggest that neurons areselective for texture element size gradients, inwhich the dimension that varies is orthogo-nal to the gradient (Todd & Thaler 2010). In

PARALLEL FUNCTIONAL IMAGING

Even with the same stimulus parameters and paradigms in mon-key and human functional magnetic resonance imaging (fMRI), anumber of differences, other than the species difference, remain.

1. In our studies, macaque fMRI, unlike human fMRI, usesa contrast agent (MION or equivalent) and is more sensi-tive and specific. Signal signs are opposite and hemodynamicresponse functions are different, but these are minor issues.

2. Scanning parameters differ, especially voxel size. Typi-cally human and monkey imaging have used 3-mm and 2-mm isotropic voxels, respectively. Sizes have recently beenreduced to 2-mm and 1-mm isotropic voxels, respectively,resulting in sampling ratios close to the tenfold ratio betweenhuman and monkey cortical surfaces.

3. The number of subjects is smaller (2–4) in monkey versushuman studies (12–20), owing to the lesser variability be-tween macaques. Hence, human fMRI more frequently usesrandom rather than fixed effects with more smoothing.

4. Monkey and human subjects are both highly motivated butby different means: control of water supply and monetaryrewards, respectively.

5. To control for possible differences in attentiveness duringpassive viewing, subjects perform a high-acuity task requir-ing them to pay acute attention to a central stimulus. Thiscontrol has invariably confirmed the results obtained withpassive viewing.


Ann

u. R

ev. N

euro

sci.

2011

.34:

361-

388.

Dow

nloa

ded

from

ww

w.a

nnua

lrev

iew

s.or

gby

Uni

vers

ity o

f A

berd

een

on 0

9/20

/13.

For

per

sona

l use

onl

y.


180° 0°

1 s

0

50/s

Line

– T

P

FP

Dot

– T

PRD

S

Near

Concave

Concave

Convex

Convex

Far

0

30

15

Spik

es/s

a

d

e

Ave

rage

resp

onse

(spi

kes/

s)

10

30

50 b

0 090 m ob45 135bump

Ridge orientation (º) Relative depthSaddle

10

30

50 c

StereoLefteye

Righteye

Firingrate

Stimulus duration

0

90/s

Firingrate

MS272_F

MS272_F

Preferred

Nonpreferred

370 Orban

Ann

u. R

ev. N

euro

sci.

2011

.34:

361-

388.

Dow

nloa

ded

from

ww

w.a

nnua

lrev

iew

s.or

gby

Uni

vers

ity o

f A

berd

een

on 0

9/20

/13.

For

per

sona

l use

onl

y.


RL: random lines

LIP: lateralintraparietal area

PIP: posteriorintraparietal area

contrast, gradients used in the 3D shape percep-tion of textured potatoes are supposedly basedon gradients of the texture element dimen-sion parallel to the gradient direction (Todd& Thaler 2010). Finally, CIP neurons are alsoselective for perspective gradients created bydeforming the stimulus outline (Tsutsui et al.2001), and the optimal tilt for perspectivematches that for disparity. Thus the selectivityof CIP neurons for first-order gradients is wellestablished. Katsuyama et al. (2010) recently re-ported that CIP neurons in one animal were se-lective for second-order stereo surfaces definedby the shape index.

The Sakata group has proposed thatdisparity-gradient selectivity arises in the CIPand is then transmitted to the AIP (Nakamuraet al. 2001, Sakata et al. 2005, Tsutsui et al.2005). Second-order disparity selectivity forvertically curved surfaces has indeed beenobserved in the AIP (Srivastava et al. 2009).Compared with their TEs counterparts, AIPsecond-order disparity-selective neuronsrespond much earlier, their selectivity for vari-ations in disparity is coarser, and it emphasizesthe metric changes more than sign reversalsin curvature. Neurons selective for first-orderdisparity gradients have also been reported inthe AIP (Srivastava et al. 2009). The short re-sponse latencies of gradient-selective neuronsin the AIP, much shorter than their ventralcounterparts, certainly support the schemeproposed by the Sakata group. Second-order

disparity-selective neurons have also beenobserved in ventral premotor cortex area F5a(Theys et al. 2009), which receives direct inputfrom the AIP (Borra et al. 2008).

The involvement of two cortical regions, theCIP and the AIP, located in the lateral bankof the IPS, in the extraction and processingof depth structure from stereo has been am-ply confirmed by fMRI studies. Two types ofparadigms have been used. One, inspired bythe perceptual function, contrasts experimen-tal conditions in which stimuli appeared as 3D,with control conditions in which only 2D shapewas perceived. Other paradigms start with theproperties of known gradient-selective neuronsand attempt to devise a contrast that selectivelyactivates regions housing such neurons.

The extraction of 3D shape from disparitywas investigated with a random line (RL) per-ceptual paradigm (Supplemental Figure 2) inwhich three conditions were compared: RLsslanted in depth depicting a 3D shape (3Dshape), RLs in multiple fronto-parallel planes(zero-order depth), and RLs in the fixationplane (no depth). Although regions outside theIPS, e.g., in the caudal part of the STS, poste-rior to MT/V5, were activated in the contrast3D shape minus zero-order depth, we concen-trate here on the three activation sites in theIPS: one in the CIP, another in the anteriorlateral intraparietal area (LIP) and the AIP, anda third one in the posterior intraparietal area(PIP) and possibly the medial intraparietal area

←−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−Figure 5Gradient-selective neurons: a first-order disparity- and texture-gradient selective caudal intraparietal area(CIP) neuron (a); a second-order speed gradient selective floor of the temporal sulcus (FST) area neuron(b,c) and a second-order disparity-gradient selective TEs neuron (d,e). (a) Responses of a CIP neuron to threeorientations of dot texture (dot-TP), line texture (line-TP), and random dot stereogram (RDS) surfaces; (b,c)responses of an FST neuron to speed-defined quadratic surfaces for preferred (solid dark yellow line) andnonpreferred (dotted orange line) mean speed and two positions in the RF (black circles in insets showing the RFmap); (d ) responses of a TEs neuron to singly curved disparity-defined concave and convex surfaces forbinocular and monocular presentations; (e) response for the same neuron to concave and convex surfaces fordifferent positions in depth of the stimuli. In a, horizontal lines indicate stimulus presentation, numbersindicate tilt, FP stands for fronto-parallel, dashed lines and arrows in the RDS indicate the surface orientationand the surface normal, and vertical calibration bar indicates firing rate; in d vertical line indicates 90 spikes/s;in a, d, e horizontal lines indicate stimulus duration. Data from Tsutsui et al. (2002), Mysore et al. (2010a),and Janssen et al. (2000a).



Ann

u. R

ev. N

euro

sci.

2011

.34:

361-

388.

Dow

nloa

ded

from

ww

w.a

nnua

lrev

iew

s.or

gby

Uni

vers

ity o

f A

berd

een

on 0

9/20

/13.

For

per

sona

l use

onl

y.



(MIP) (Durand et al. 2007). Activity in bothregions in the lateral bank, and to a lesser de-gree in those in the medial bank, was reducedby a scrambling of object images (Kourtzi &Kanwisher 2000), indicating that these regionsalso process 2D shape.

The second MR paradigm was inspiredby the properties of TEs neurons (neuronalparadigm), which are disparity selective, re-sponding better to binocular than monocularstimuli ( Janssen et al. 2000a), and selectivefor second (first)-order disparity gradients,responding better to curved (or slanted) thanflat stereo surfaces ( Janssen et al. 2000b).Therefore, a factorial design was used withstereo (present or absent) and nature of sur-faces (curved or slanted versus flat) as factors,and the interactions between these factorsshould correspond to regions housing second(first)-order gradient-selective neurons. Thestimuli, identical to those of the single-cellstudies (Supplemental Figure 1), were pre-sented in the stereo conditions at differentpositions in depth to emulate the positionin depth invariance of the higher-order TEsneurons. Finally, the distributions of dis-parities were matched as well as possible inthe curved (slanted) and flat conditions. Forcurved surfaces, interaction between the stereoand surface-type factors, indicating strongerresponse, relative to monocular controls, tocurved than to flat stereo surfaces, reachedsignificance in the anterior IPS (in the AIP andanterior LIP) and F5a (Durand et al. 2007, Jolyet al. 2009). For slanted surfaces, the interac-tions occurred mainly in the AIP, consistentwith the observation (Srivastava et al. 2009)that first-order disparity-selective neuronswere more numerous in the AIP than in theTEs. The absence of interactions in the CIPwas ascribed to the small stimulus size bettersuited to the TEs than to CIP neurons. Insubsequent studies using a more sensitive MRtechnique (3T instead of 1.5T and several coilsin parallel), interactions were also observed inthe CIP (Van Dromme et al. 2010).

The importance of stimulus size for drivingCIP was demonstrated in one animal in which

linear texture gradients were tested (O. Joly, J.Todd, W. Vanduffel, and G. Orban, unpub-lished observations), using a factorial design,with surface type (slanted versus front-parallel)and retinal size (small versus large) as factors.The interaction of size and type yielded an acti-vation site in the CIP, consistent with the largestimuli used by Tsutsui et al. (2002). The in-volvement of CIP in 3D SFT was demonstratedby Nelissen et al. (2009), using large, second-order textured stimuli (see below). So far, littleevidence for processing of depth structure fromshading has been obtained in the IPS, althoughthe AIP was activated in one animal tested byNelissen et al. (2009).

Speed-Gradient Selective Neuronsin the Posterior SuperiorTemporal Sulcus

Selectivity for first-order speed gradients wasdemonstrated in MT/V5 neurons by Xiao et al.(1997a). The speed invariance of the MT/V5gradient selectivity was demonstrated in alertanimals by Nguyenkim & DeAngelis (2004) andMysore et al. (2010b). These authors comparedMT/V5 and the FST in the same animals andobserved more first-order selective neurons inFST than in MT/V5. Nguyenkim & DeAn-gelis (2003) reported MT/V5 neurons selec-tive for surface orientation defined by dispar-ity, but such selectivity was relatively modest,compared with that in the CIP or TEs. In asubsequent study in which surface orientationsdefined by speed, disparity, or texture gradientswere compared, MT/V5 neurons were more se-lective for surface orientation defined by speedthan by disparity gradients, texture being ratherineffective (Nguyenkim & DeAngelis 2004). Anumber of studies have examined the responsesof MT/V5 neurons while monkeys made judg-ments about the (ambiguous) direction of mo-tion of a rotating transparent cylinder (Bradleyet al. 1998, Dodd et al. 2001). This task revealsmechanisms related to the monkey’s perceptionof order in depth, i.e., zero-order depth, but re-quires no judgment about 3D shape (Born &Bradley 2005).

372 Orban

Ann

u. R

ev. N

euro

sci.

2011

.34:

361-

388.

Dow

nloa

ded

from

ww

w.a

nnua

lrev

iew

s.or

gby

Uni

vers

ity o

f A

berd

een

on 0

9/20

/13.

For

per

sona

l use

onl

y.


Second-order selectivity based on motionwas investigated in the FST using a representa-tive set of quadratic surfaces, including a bump,ridges, and saddles, corresponding to differentvalues of the shape index (SupplementalVideos 1–2) and several orientations of theseridges and saddles (Mysore et al. 2010b). Sixtypercent of the responsive FST neurons wereselective for second-order SFM. Figure 5b il-lustrates a neuron selective for a saddle. This se-lectivity was invariant for changes in speed andposition (Figure 5b,c), demonstrating thehigher-order nature of the selectivity. Addi-tional invariance was observed for changesin stimulus size, monocular or binocularpresentation, and the nature of the motion cue(speed gradient, direction reversal, or both).This second-order selectivity was alreadypresent in 45% of the MT/V5 neurons, whichrepresent a processing stage earlier than FSTneurons (Ungerleider & Desimone 1986).Compared with their MT/V5 counterparts,FST neurons were more selective, and selectivein a more invariant manner, for a wider rangeof surfaces and throughout the stimulationinterval (Mysore et al. 2010b). Hence, theseauthors proposed that the extraction of speedgradients involves two stages: At the level ofMT/V5, a surround-based mechanism extractslinear gradients because (a) the antagonisticsurrounds are asymmetric and restricted to asingle suppressive zone in 50% of the surroundneurons (Xiao et al. 1997b) and (b) the speedsensitivity of this surround zone is relative(Xiao et al. 1997a), in agreement with com-putational models (Buracas & Albright 1996).A similar mechanism but one with a double-symmetric suppressive zone in the surround(Xiao et al. 1995) may extract second-ordergradients. This idea is consistent with the weakposition invariance of the selectivity, with therelationship between preferred orientationsfor first- and second-order selectivity and withthe restriction of selectivity to tilted planesand ridges in MT/V5 (Mysore et al. 2010b).The convergence of several MT/V5 neuronshaving offset centers, and possibly selective fororthogonal ridges, combined with inputs from

V4 selective for kinetic contours created byopposing directions of motion (Mysore et al.2006), could explain the full range of propertiesof FST speed-gradient selective neurons.

MT/V5 and FST send the speed gradientsignals further along the STS, most notably inMSTd. Duffy & Wurtz (1997) reported thatMSTd neurons were sensitive to the inversionof the speed gradient present in expansion pat-terns. MSTd neurons selective for first-orderspeed gradients superimposed onto optic flowcomponents were reported by Sugihara et al.(2002), indicating that motion patterns otherthan translation can carry speed gradients.It remains to be seen whether selectivity for3D SFM, reported for STP neurons testedwith coherent and incoherent rotating spheres(Andersen & Siegel 2005), survives moresophisticated testing with a wider range ofspeeds, positions, and 3D shapes.

Several fMRI experiments have confirmedthe role of MT/V5 and the FST in 3D SFM.In one experiment, two conditions with nineconnected RLs undergoing either rotationaround a vertical axis or translation at uniformspeed were contrasted in a perceptual paradigm(Supplemental Videos 3 and 4). Indeed, thefirst display evokes the perception of a 3D wire-frame figure rotating in depth, and the second,that of a flat figure moving in the fronto-parallelplane. This contrast yielded a significant acti-vation of MT/V5 and the FST (Vanduffel et al.2002a). The activation was marginally higherin the FST than in MT/V5 (Nelissen et al.2006, figure 10c), which is consistent with thelarger proportion of gradient-selective neuronsin FST. Mysore et al. (2010b) compared FSTneuronal responses to rotating and translatinglines identical to those used in these MR stud-ies. FST neurons responded significantly morestrongly to rotating lines than to translatinglines, and the percent difference was similarto that of MR signals in the FST documentedby Nelissen et al. (2006). Three-dimensionalshape and 2D motion control conditions werealso compared by Sereno et al. (2002), whoobserved MR activation of MT/V5 and theFST, consistent with the neuronal properties in


Ann

u. R

ev. N

euro

sci.

2011

.34:

361-

388.

Dow

nloa

ded

from

ww

w.a

nnua

lrev

iew

s.or

gby

Uni

vers

ity o

f A

berd

een

on 0

9/20

/13.

For

per

sona

l use

onl

y.


these areas, but also saw a widespread activationin anterior STS and in the posterior two-thirdsof the IPS not observed by Vanduffel et al.(2002a). There are many differences betweenthe two studies: Sereno et al. (2002) usedtextured surfaces, not randomly connectedlines, an anaesthetized preparation rather thanawake monkeys, and lower statistical thresh-olds. Investigators recently confirmed theinvolvement of MT/V5 and the FST in a thirdMR study contrasting second- and zero-orderspeed distributions (Mysore et al. 2010a) usingthe surface stimuli of Mysore et al. (2010b).

The latter fMRI study also provided evi-dence for further processing in the lower bankof the STS and the convexity of the TE. Theinitial study (Vanduffel et al. 2002a) reportedthat activation by 3D SFM did not reach sig-nificance in the IPS, but a later study using asomewhat different group of monkeys revealedweak but significant activations in the AIP andthe PIP (Durand et al. 2007). One animal witha clear SFM activation in the AIP showed aweak activation in the stereo experiments, butit is too early to draw conclusions from thisobservation.

Extraction of Depth Structurein the Ventral Stream

Second-order disparity-gradient selectivity hasbeen documented in the ventral stream byJanssen et al. (1999), who tested several second-order vertical configurations of disparity varia-tion (some corresponding to a horizontal ridge;see Supplemental Figure 1), which are moreeasily produced (avoiding spurious texture den-sity changes). They verified, however, that neu-rons can be selective for vertical or horizontalridges or both ( Janssen et al. 2001). This dispar-ity selectivity was initially discovered in a smallregion in the rostral part of the lower bank ofthe STS, labeled TEs ( Janssen et al. 2000a).Higher-order selectivity was demonstrated byshowing invariance of selectivity for changes inposition in depth (Figure 5d,e) ( Janssen et al.1999, 2000a). TEs neurons (see Orban et al.2006b for review) give a fine description of the

disparity variations, emphasize the change insign from convex to concave, and are not se-lective for anticorrelated stereograms ( Janssenet al. 2000b, 2003). Neurons selective for first-order disparity gradients have also been re-ported in the TEs ( Janssen et al. 2000b, Liuet al. 2004). It remains unclear whether the TEsand the region behind it, where Yamane et al.(2008) reported selectivity for combinations ofdepth curvatures (see Kourtzi & Connor 2011,this volume), constitute a single area or two. AIPgradient-selective neurons have much shorterlatencies than do their TEs counterparts; hence,we cannot deny that TEs neurons inherit theirselectivity via AIP input (Borra et al. 2008). Itis more likely that disparity gradients are ex-tracted earlier in the ventral pathway, but itis unclear where. Little selectivity for dispar-ity gradients has been observed in V4 for sur-face stimuli (Hegde & Van Essen 2005). Butthe TEO, which has not yet been explored withsingle-unit recordings, remains a possibility, asdoes the region described by Yamane et al.(2008).

Selectivity for second-order vertical varia-tions in texture has been tested in the TEs (Liuet al. 2002). Because the 3D texture stimuli alsohave a 2D interpretation, the standard strat-egy has been to test the convergence of dispar-ity and texture signals onto single neurons thathave similar preferences for the two cues. Hor-izontal ridges were portrayed by a combinationof disparity and texture (Figure 2), either inagreement or in conflict, and the strength of thedisparity was manipulated by reducing stereocoherence. When stereo was strong, selectivitydepended entirely on this cue. At 20% stereocoherence—where the monkey can still use thestereo cue to distinguish convex from concave(Supplemental Figure 4)—selectivity was sig-nificantly lower when cues conflicted than whenthey agreed. At zero coherence, i.e., with onlythe texture cue, the selectivity for curvaturesign was weak but matched that for disparity(Figure 2e). Thus the disparity cue wasstronger than the texture cue for second-ordergradients in the TEs. However, convergencebetween first-order selectivities for texture- and

374 Orban


Ann

u. R

ev. N

euro

sci.

2011

.34:

361-

388.

Dow

nloa

ded

from

ww

w.a

nnua

lrev

iew

s.or

gby

Uni

vers

ity o

f A

berd

een

on 0

9/20

/13.

For

per

sona

l use

onl

y.



disparity-defined surfaces was observed in theTEs by Liu et al. (2004).

The selectivity for shading has been investi-gated using randomly deformed spheres (pota-toes) in V4 (Arcizet et al. 2009) and in theinfero-temporal (IT) cortex (Vangeneugdenet al. 2006). Selectivity for 3D shape was testedfor four directions of illumination and com-pared with that for 2D shape controls. Thesestudies were relatively disappointing becauseselectivity depended on illumination direction,and the small response differences between 3Dand 2D shape barely reached significance in theTEO and were not significant in V4 or the TE.

In contrast with the results obtained in theIPS and the STS, the fMRI results did notmatch the single-cell data particularly wellin the ventral stream. For the pictorial cues,Nelissen et al. (2009) used the perceptualstrategy, again comparing conditions in which3D shape, or only 2D shape, was perceived.Because a single control condition could notcontrol for all low-level 2D aspects presentin the 3D shape conditions, they followed thehuman experiment (Georgieva et al. 2008),using a battery of control conditions thatcollectively included all low-level properties ofthe 3D displays and required activation in theconjunction of subtractions comparing the 3Dcondition to each of the control conditions.The 3D SFT activation in the ventral streamwas restricted to the ventral TEO (Nelissenet al. 2009). No activation of the TEs wasobserved, reflecting either the low numberof first-order selective neurons in the TEs ortechnical limitations, because the signals in theSTS were relatively weak in the 1.5T. Theseresults only partially replicate those of Sereno

et al. (2002), who reported widespread activa-tion in the occipito-temporal cortex, includingin MT/V5, but not in the TEO. Activationby 3D SFS was also restricted (Nelissen et al.2009) and limited to the dorsal TEO and theadjacent portion of V4. These very restrictedMR activations may explain the largely negativesingle-cell results obtained so far for shading.

Interactions between the stereo and surfacetype factors, indicating stronger activation, rel-ative to monocular controls, for curved ratherthan flat surfaces, yielded activation sites in thelower STS bank in a region near the TEs ( Jolyet al. 2009). Such interactions were not ob-served for slanted surfaces. A subsequent studyat 3T (Van Dromme et al. 2010) has revealedmore extensive activation of the lower bank ofthe STS, yet posterior to the TEs, as well as anactivation near the TEO.

Overview of Monkey Studies

Gradient-selective neurons form a family ofhigher-order neurons that occur in at leastseven predominantly middle- or high-levelvisual areas: MT/V5, FST, MSTd, CIP, AIP,TEs, and F5a (Table 1). In most of theseareas, zero-, first-, and second-order selectiveneurons co-occur. For example, MT/V5neurons are selective for depth specified bymotion parallax (Nadler et al. 2008, 2009),which can be considered zero-order selectivityfor motion, more precisely depth intervals,complementing the selectivity for qualitativeorder in depth documented with the rotatingcylinder (Bradley et al. 1998).

In general, a good match can be foundbetween the MR activation sites obtained

Table 1 Areas housing gradient selective neuronsa

Order/cue Motion Stereo TextureFirst order MT/V5, MSTd, FST CIP, TEs, AIP, MT/V5? CIP, TEsSecond order MT/V5, FST, STP? TEs, AIP, F5a, CIP? TEs?

aAbbreviations: AIP, anterior intraparietal area; CIP, caudal intraparietal area; FST, floor of the temporal sulcus area;MSTd, dorsal part of the medial temporal sulcus area; MT/V5, middle temporal area or V5; STP, superior temporalpolysensory region; TEs, stereo part of TE.


Ann

u. R

ev. N

euro

sci.

2011

.34:

361-

388.

Dow

nloa

ded

from

ww

w.a

nnua

lrev

iew

s.or

gby

Uni

vers

ity o

f A

berd

een

on 0

9/20

/13.

For

per

sona

l use

onl

y.


PIPD T

TEsD

Ant LIPD (M) T

V2/V3D M T

V4M S

MT/V5M

TEOdS

TEOvT

V1

AIPD M (T)

FSTM

vPreMD

CIPD T

IPSIPS

LuSLuS

STSSTS

CeSCeS

ArcSArcSIPS

LuS

STS

CeS

ArcS

Figure 6Overview of monkey functional magnetic resonance imaging (fMRI): activation sites for the four cuesindicated by capital letters: D, disparity (red text); M, motion ( green text); T, texture (blue text); and S, shading(dark yellow text). Abbreviations: ArcS, arcuate sulcus; CeS, central sulcus; IPS, intraparietal sulcus;LuS, lunate sulcus; STS, superior temporal sulcus. Note that the texture and motion cue activations wereweak in PIP.

Cortical area:a functional corticalentity defined by fourcriteria: archi- andmyelo-architectonics,anatomicalconnections,topographicorganization, andfunctional properties

(Figure 6) and the cortical areas that housegradient-selective neurons (Table 1), especiallyif one notes that the activations in early cortex(V2 and V3) occur at the edges of the stimulusrepresentation and may reflect depth discon-tinuities or attentional effects. Such effectsoccur in V1–V3 of both monkeys and humans,typically at the edge of stimuli (Kolster et al.2010, Saygin & Sereno 2008). Beyond V2–V3,fMRI produces, if anything, more widespreadactivation than expected from the locations ofgradient-selective neurons, e.g., in V4 for 3DSFM and V3 for 3D SFD. The V3 activationcould either be an edge effect or represent sen-sitivity for relative disparity, i.e., a nonspecificeffect owing to a lower-order confound. Morerecent experiments in the 3T using steps inzero-order disparity (Figure 1a) support thelatter view (Van Dromme et al. 2010). Theinterpretation of the dorsal V4 activation is

unclear because no relevant single-cell data areavailable. It may represent a region unknown tohouse gradient-selective neurons, as may be thecase for the PIP or the anterior LIP in 3D SFMor 3D SFD, respectively. Further studies com-bining fMRI and single-cell recording in thesame animal are needed to clarify these issues.

HUMAN FUNCTIONAL IMAGINGSTUDIES OF 3D SHAPEPROCESSING

Parallel Human and MonkeyfMRI Studies

The ramifications of parallel human andmonkey fMRI studies extend beyond the com-parison of functional neuroanatomy into thehomologies between cortical areas in the twospecies, which remain largely uncharted. In

376 Orban

Ann

u. R

ev. N

euro

sci.

2011

.34:

361-

388.

Dow

nloa

ded

from

ww

w.a

nnua

lrev

iew

s.or

gby

Uni

vers

ity o

f A

berd

een

on 0

9/20

/13.

For

per

sona

l use

onl

y.


V3A+D M T (S)

V2/V3D M

V1

phAIPD

cITG3D

vPreMD

DIPSMD M T

POIPSD M T

DIPSAD M (T)

hMT/V5+M

cITG2T

cITG1S

VIPS(V7)D M T

IPSIPS

STSSTS

CeSCeS

ITSITS

IPS

STS

CeS

ITS

Figure 7Overview of human functional magnetic resonance imaging (fMRI): activation sites for the four cuesindicated by capital letters: D, disparity (red text); M, motion ( green text); T, texture (blue text); and S, shading(dark yellow text). Abbreviations: see list in Figure 6; ITS, inferior temporal sulcus; X, lack of magneticresonance signals. VIPS may correspond to several retinotopic areas including V7.

Activation pattern:distribution ofdifferential MRactivity betweenexperimental andcontrol conditionsover the brain orcortical surface

Homology: cross-species similarities of acortical area because itderives from that of acommon ancestor,here of primates

addition, combining these two approaches withsingle-cell recordings (Figure 4) allows one toextrapolate from single-cell results to humansby devising MR paradigms that activate themonkey areas in which a given selectivityoccurs, porting those paradigms to humanimaging, and obtaining activations in homol-ogous areas. This method circumvents certainproblems plaguing human fMRI when address-ing neuronal selectivity because the underlyingassumptions were either questionable, as forrepetition suppression (Sawamura et al. 2006),or inapplicable to certain areas, as with mul-tivoxel analysis of orientation selectivity in V4(Vanduffel et al. 2002b). Hence, this sectionstarts with the human fMRI experiments usingexactly the same stimuli and paradigms asthose used in the monkey (Durand et al. 2009;Georgieva et al. 2008, 2009; Orban et al.1999; Vanduffel et al. 2002a). These studiesincluded many control experiments [up to

eight additional experiments in Georgievaet al. (2008)], and the reader is referred to theoriginal publications for descriptions of thesecontrols (see also Figure 8).

A comparison of Figures 6 and 7, summa-rizing the activation sites obtained with the four3D shape cues in the two species, reveals severalstriking similarities. The agreement betweenmonkey and human activation patterns is evenmore remarkable if one considers homologies.There is clear support for the homology of theDIPSM (dorsal IPS medial region) and theDIPSA (dorsal IPS anterior region) withthe monkey anterior LIP and posterior AIP,respectively (Durand et al. 2009), and of thephAIP (putative human homolog of AIP) withthe anterior AIP (Georgieva et al. 2009). Thefollowing homologies are only putative: theVIPS (ventral IPS region) and the POIPS(parieto-occipito IPS region) with the CIP andthe PIP, respectively (Durand et al. 2009), and


Ann

u. R

ev. N

euro

sci.

2011

.34:

361-

388.

Dow

nloa

ded

from

ww

w.a

nnua

lrev

iew

s.or

gby

Uni

vers

ity o

f A

berd

een

on 0

9/20

/13.

For

per

sona

l use

onl

y.


hMT/V5+: thehuman middletemporal/V5 complexcorresponding tomonkey MT/V5 and anumber of its satellites;for human MT/V5,see Kolster et al.(2010)

the caudal ITG with the TEO (Nelissen et al.2009). Finally, firm evidence now indicates thathMT/V5+ includes homologues of MT/V5and the FST (Kolster et al. 2009, 2010). Thefirst similarity between the two species, shownin Figures 6 and 7, is the absence of activationin the striate cortex. Second, the extractionof 3D shape involves both the dorsal and theventral (Ungerleider & Mishkin 1982) visualpathways. More precisely, three cues, disparity,motion, and texture, are processed in the twopathways, whereas shading is processed pre-dominantly in the ventral stream. Third, in thedorsal pathway, activation is observed at eventhe most anterior level, the AIP in the monkeyand the phAIP in humans, whereas this wasnot the case in the ventral stream. Ventral acti-vation sites occur at the middle level, with theexception of the site identified as the TEs, thehuman counterpart of which is not yet known.Caution is needed regarding this differencebetween dorsal and ventral streams becausesusceptibility artifacts may limit visualizationin the temporal regions of human cortex (X inFigure 7) (Georgieva et al. 2009). Fourth, inthe parietal cortex, multiple cues are processedwithin the same areas. On the other hand, in theoccipito-temporal cortex, including in MT/V5,cues are processed largely in segregated areas:in MT/V5 and the FST for motion and in thedorsal and ventral TEO and correspondinghuman sites in the caudal ITG for shading andtexture. The main species difference lies in theactivation of the human V3A complex, but it islikely that its stereo activation reflects relativedepth rather than depth structure (Prestonet al. 2008, Tsao et al. 2003b).

Other Human fMRI Studies

The human fMRI results, summarized inFigure 7, are in general agreement with thoseobtained in other laboratories with differentstimuli. Several studies (Beer et al. 2009,Klaver et al. 2008, Martinez-Trujillo et al.2005, Murray et al. 2003, Paradis et al. 2000,Yamamoto et al. 2008) have confirmed our3D SFM observations, with the exception that

surface stimuli activate more ventral areasthan do RLs (Beer et al. 2009, Kriegeskorteet al. 2003, Orban et al. 2006a). Beer et al.(2009) also observed a 3D SFM activation inthe human STS. The timing of the activationin the various regions has been revealed bymagnetoencephalography (MEG) experiments(Iwaki et al. 2007, Jiang et al. 2008). To controlfor motion in 3D space present in the rotatingRL stimuli, we performed an additional ex-periment whereby subjects attended to eitherthe 3D shape or the 3D motion, thereby con-firming the involvement of hMT/V5+ in theprocessing of 3D shape (Peuskens et al. 2004).

The 3D SFD experiments agree with therecent studies of Preston et al. (2008, 2009).Georgieva et al. (2009) have correlated MR ac-tivity with the strength of the 3D shape percept,itself derived from the regression providedby the 3D shape adjustment task. Significantcorrelation was restricted to the V3A complexand the adjacent area V7/VIPS. Similar regionswere implicated in 3D shape estimation experi-ments by Preston et al. (2009) using after-effectsproduced in ambiguous stimuli by binocularand monocular depth cues. Preston et al. (2008)investigated only far and near distinctions butfound that dorsal parietal areas represented thedisparity structure metrically while the lateralocciptal complex (LOC) did so categorically, afinding reminiscent of the difference betweenhigher-order disparity-selective neurons inthe AIP and the TEs (Srivastava et al. 2009).Other studies in which subjects made judg-ments about the stereo-defined 3D shape haveemphasized the role of hMT/V5+ and theLOC in such tasks (Chandrasekaran et al.2007, Welchman et al. 2005), but these regionswere defined by localizer tests and includedmultiple cortical areas (Kolster et al. 2010),making difficult any links with the voxel-basedapproach of Georgieva et al. (2009). Oneof the studies (Chandrasekaran et al. 2007)observed the parietal activations described byGeorgieva et al. (2009) but found that theiractivity decreased with accuracy of judgments.

Several studies (Gerardin et al. 2010;Humphrey et al. 1997; Konen & Kastner 2008;

378 Orban

Ann

u. R

ev. N

euro

sci.

2011

.34:

361-

388.

Dow

nloa

ded

from

ww

w.a

nnua

lrev

iew

s.or

gby

Uni

vers

ity o

f A

berd

een

on 0

9/20

/13.

For

per

sona

l use

onl

y.


Kourtzi et al. 2003; Moore & Engel 2001;Shikata et al. 2001, 2008; Taira et al. 2001) haveattempted to locate regions involved in extract-ing 3D shape from texture and shading but havereported activation sites only partially match-ing our results. The two main inconsistenciesconcern the involvement of the parietal cortexin 3D SFS and the parts of the LOC involvedin this process. The discrepancies probably re-flect the many differences between these experi-ments and ours: differences in stimuli or controlconditions or in task, exploration restricted toposterior visual regions or to regions defined bylocalizer tests rather than whole brain, or theuse of repetitive suppression, which can yieldquestionable results (Sawamura et al. 2006).

3D Shape Processing in Humanand Monkey

The similarities of activation patterns inhumans and monkeys tested with identicalparadigms (Figures 6 and 7), and the homolo-gies outlined above, suggest that the 3D shapenetwork in humans is very similar to that inmonkeys. The major exception is related tothe capacity of human, but not monkey, V3Ato process motion information (Tootell et al.1997, Vanduffel et al. 2001). This capacity en-dows the VIPS, the putative homolog of theCIP, with the additional ability to extract depthstructure from motion (Orban et al. 2006a), ex-plaining the stronger activation of human com-pared with monkey parietal cortex by 3D SFM(Vanduffel et al. 2002a).

Similarities in activation patterns for 3Dshape processing in the two species also sug-gest that the human areas activated by 3Dshape house gradient-selective neurons similarto those recorded in monkeys. As noted above,the MR paradigms may err on the nonspecificside and thus yield false positives. Therefore,one should derive such conclusions only for ar-eas whose homology is at least putative. For ex-ample, the DIPSA likely houses higher-orderdisparity-selective neurons. Because the DIPSAmay not be a single cortical area, there is roomfor alternative interpretations. On the other

hand, considerable evidence suggests that re-cently identified human MT/V5 (Kolster et al.2010) and monkey MT/V5 (Kolster et al. 2009)are homologous. Hence, this human corti-cal area likely houses speed-gradient selectiveneurons (Figure 8).

TWO TYPES OF 3D SHAPEEXTRACTION

Even 3D surfaces have boundaries, and Janssenet al. (2001) have shown that TEs neurons canindependently process the depth structures ofthe boundaries and of the surface itself. Thisdissociation is reminiscent of the distinctionbetween axis and plate neurons in the CIP(Sakata et al. 2005). Computationally, the ex-traction of depth curvature of edges and sur-faces requires different operations (Li & Zucker2006a,b), which suggests the involvement ofdifferent areas. Far less is known about the ar-eas involved in processing the depth structure ofboundaries. It has recently been observed thatboundary stimuli are more efficient at drivinghigher-order disparity-selective neurons in theAIP (Srivastava et al. 2008), whereas in the TEs,boundaries and surfaces were equally effective( Janssen et al. 2001). Furthermore, investiga-tors have reported that V4 neurons are selectivefor the 3D orientations of line stimuli (Hinkle& Connor 2002), which is not the case for sur-face stimuli (Hegde & Van Essen 2005). Also,line stimuli are more effective for 3D SFM inparietal cortex, and conversely, surface stimuliare more effective in occipito-temporal regions(Orban et al. 2006a, figure 2). Hence, furtherwork is needed to clarify the networks extract-ing and processing these two types of 3D shapeinformation.

BEYOND THE EXTRACTIONOF 3D SHAPE

So far I have considered the extraction ofdepth structure from the four cues separately,although single-cell studies using monocularstatic cues have combined disparity with textureto circumvent ambiguity present in pictorial-cue stimuli. The MR activation patterns


Ann

u. R

ev. N

euro

sci.

2011

.34:

361-

388.

Dow

nloa

ded

from

ww

w.a

nnua

lrev

iew

s.or

gby

Uni

vers

ity o

f A

berd

een

on 0

9/20

/13.

For

per

sona

l use

onl

y.


b

c

Monkey

Human

* **

**

*

*

*

* * * *

**

*

*

*

Monkey

% M

R si

gnal

cha

nge

2

1

0

MT/V5

Attention

*

Fixation

*

* Significant differences

% M

R si

gnal

cha

nge

LOS

OTS

TOS

AOS

IOS

LuS

STS

–

–

+ + +

+ –

– –

– +

–

–

+

+

++

–

–

––

++

+

+–

+–

–

+

– –

V1

V1 V2 V3

V2 V3

hV4

V3AV3B

LO1

VO1

phPITd

pFST

pMSTv

MT/V5MT/V5MT/V5

LO2

pV4t

phPITv

V2vV3v V4v

V4dV3dV2d

V3A

V1

V1

MSTv

V4t

MTp

–+

+

++

++

––

–

+– –+

++ –

–

–

–PITd

FST

0

1

2MT/V5

Human

Horizontal meridian

Eccentricity ridgeVertical meridians

D

A3 mm

D

A1 cm

00

10

20

90 180Tilt (º)

Ave

rage

resp

onse

(spi

kes/

s)

Monkey

270

a iv141_F

*

RotatingTranslating

MT/V5MT/V5MT/V5

380 Orban

Ann

u. R

ev. N

euro

sci.

2011

.34:

361-

388.

Dow

nloa

ded

from

ww

w.a

nnua

lrev

iew

s.or

gby

Uni

vers

ity o

f A

berd

een

on 0

9/20

/13.

For

per

sona

l use

onl

y.


(Figures 6 and 7) suggest that the different cuesare processed in the same parietal areas, whichdoes not necessarily guarantee convergence atthe single-cell level. Up to now, neuronal con-vergence has been observed only in the CIP(Tsutsui et al. 2002), although it likely occursin the AIP also. Cortical areas in which multi-ple depth structure cues converge consistentlyat the neuronal level are likely to be final rep-resentations where depth structure can be usedfor behavioral purposes. The representation inthe CIP probably lays out the 3D structure ofthe environment, part of the metric descriptionof space, necessary for visual control of action inspace. Consider a winding mountain path alonga cliff. The representation in the AIP lays outthe 3D shape of objects for visual control ofgrasping (Sakata et al. 2005). Grasping a discrequires a different grip than does grasping anorange.

Cue convergence has also been documentedin the ventral stream. First-order gradients de-fined by disparity and texture converge con-sistently onto TEs neurons (Liu et al. 2004).Yamane et al. (2008) reported that disparity,shading, and texture converged onto neuronslocated mainly in the lower bank of the STS,stretching broadly speaking from the TEO intothe TEs. Because responses were sought usingcomplex curved surfaces defined by all threecues, the demonstration that selectivity for sur-face fragments was retained for isolated cuesleaves open the question of cue consistency in

this region. I propose that cues converge notonly in the lower bank of the STS, as docu-mented by Yamane et al. (2008), but also inoverlapping and neighboring regions involvedin the extraction of the 3D shape of the static(Tsao et al. 2003a) and the moving human body(Perrett et al. 1985, Singer and Sheinberg 2010)and face, as discussed by Mysore et al. (2010b).Indeed, perceptual judgments about actions im-prove in 3D conditions ( Jackson & Blake 2010).

Haptics also provides information about the3D shape of objects. In humans, 3D shape fromtactile information (Amedi et al. 2002) activatesregions close to the caudal ITG site activated by3D shape-from-disparity and monocular cues(Georgieva et al. 2009). Thus, somatosensoryinformation and visual information may con-verge relatively early in the ventral stream. De-velopmentally, haptic 3D shape may be impor-tant for calibrating visual 3D shape, which inturn may guide the development of 2D shapeselectivity. This developmental sequence mayreconcile cases of developmental prosopagnosia(Laeng & Caviness 2001) with the predomi-nantly 2D selectivity of face-selective neurons(Freiwald et al. 2009). At the level of V4, orien-tation information provided by both vision andtouch converges (Maunsell et al. 1991). Hence,similar analyses of orientation distributions inboth modalities may give rise to a multimodaldepth structure representation in the neighbor-hood of the TEO. Thus neurons selective forfirst- and second-order depth in or near the

←−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−Figure 8Integration of human and monkey 3D shape studies: the example of area MT/V5. (a) First-order speed-gradient selective neuron recorded in monkey MT/V5; (b) flat map showing the location of MT/V5 in themonkey; (c) flat map with location of MT/V5 in the human. Insets in b and c plot the magnetic resonance(MR) signal change (relative to static control) for rotating (dark blue) and translating (light blue) random lines(RL) in three monkeys (both when fixating and when attention was distracted by a high-acuity bar task) andsix humans (when fixating); vertical bars indicate standard errors (SE). Asterisks indicate significantdifferences between conditions. Considerable evidence indicates that human and monkey MT/V5 arehomologous (Kolster et al. 2010). Both show similar MR response differences to rotating and translating RL.In the monkey, this difference corresponds with the presence of speed-gradient selective neurons, hencehuman MT/V5 in all likelihood also houses speed-gradient selective neurons. Data from Vanduffel et al.(2002a), Kolster et al. (2009, 2010) and Mysore et al. (2010b). Note that the high-acuity bar task (b) was oneof the typical control experiments in the imaging studies.


Ann

u. R

ev. N

euro

sci.

2011

.34:

361-

388.

Dow

nloa

ded

from

ww

w.a

nnua

lrev

iew

s.or

gby

Uni

vers

ity o

f A

berd

een

on 0

9/20

/13.

For

per

sona

l use

onl

y.


middle ventral pathway may provide the build-ing blocks for describing the depth structures ofobjects or object parts, just as orientation- andcurvature-selective neurons in V4 (Patsupathy& Connor 2002) do for such descriptions in 2D.The same may hold true for building visual rep-resentations of actions ( Jackson & Blake 2010)and gestures, at least those based on tempo-ral derivatives of shape (Giese & Poggio 2003,Singer & Sheinberg 2010). The importance ofthe 3D descriptions of actions and objects maydecrease with development to the point thatdepth structure may not be required for adultsto recognize objects (Pizlo et al. 2010), althoughfurther studies using meaningful images of ob-jects and actions to test 3D selectivity are war-ranted. At present, regions such as the TEs orthat described by Yamane et al. (2008) seem toplay a supplemental role in object representa-tion in IT during adulthood, very much as colordoes (Matsumora et al. 2008). The representa-tions can be called on when details about 3Dshape are needed to better identify visual ob-jects or to better describe their physiological oremotional state.

Finally, most of the experiments reviewedhere were performed in passively fixatingsubjects, although subjects in some humanfMRI experiments performed a task. As notedby Shikata et al. (2008), using a task implies thattask differences unrelated to stimulus depthstructure may determine the MR activationpattern. This ambiguity may also apply to thestudies of Tsutsui et al. (2001, 2002) in whichmonkeys performed a DMS task (see above).Given task-dependent processing in sensorysystems (Colombo et al. 1990, Orban & Vogels1998), all studies using tasks will face similarproblems, except where the task is unrelatedto the stimuli studied (Figure 8b). Hence, it isdifficult to interpret results obtained in subjectsperforming 3D shape tasks without preliminaryinformation provided by passive experiments.The present review thus paves the way forexperiments using tasks (Tsutsui et al. 2001,Verhoef et al. 2010) to demonstrate causal linksbetween cortical areas and behavior. Even inthese experiments, results will depend on theproperties of the stimuli (depth structure) andthe task components.

FUTURE ISSUES

1. Further work is needed on the mechanisms extracting depth structure from texture andshading, as well as the processes for extracting depth structure from boundaries andsurfaces.

2. Once these mechanisms have been fully identified, their corresponding wiring diagramswill have to be worked out.

3. How does 3D shape information derived from different visual cues converge? To whatextent is this process generalized to other senses such as the somato-sensory system?

4. How is 3D shape information used in different tasks, whether related to guidance ofmotor behavior or for perceptual purposes?

5. How is the human 3D shape network affected by brain damage from trauma or disease?How does this compare to experimental lesions or inactivation studies?

DISCLOSURE STATEMENT

The author is not aware of any affiliations, memberships, funding, or financial holdings that mightbe perceived as affecting the objectivity of this review.

382 Orban

Ann

u. R

ev. N

euro

sci.

2011

.34:

361-

388.

Dow

nloa

ded

from

ww

w.a

nnua

lrev

iew

s.or

gby

Uni

vers

ity o

f A

berd

een

on 0

9/20

/13.

For

per

sona

l use

onl

y.


ACKNOWLEDGMENTS

The author thanks J. Todd for the excellent collaboration spanning 15 years and for comments onthe manuscript. The author is indebted to S. Mysore for help with the figures and references andto H. Kolster and O. Joly for designing figures. He thanks S. Raiguel, P. Janssen, W. Vanduffel,and R. Vogels for comments on earlier versions of the manuscript. The author has enjoyed themultiple collaborations underlying the work reported in this article with his co-PIs, P. Janssen, W.Vanduffel, and R. Vogels, and with all past and present members and associates of the Laboratoriumvoor Neuro-en Pyschofysiologie. Supported by Geconcerteerde Onderzoeks Acties (GOA), Fondsvoor Wetenschappelijk Onderzoek (FWO), Inter-University Attraction poles (IAP), ExcellentieFinanciering (EF), and European Union (EU) projects from Information Society Technologies(IST) Future and Emerging Technologies (FET).

LITERATURE CITED

Amedi A, Jacobson G, Hendler T, Malach R, Zohary E. 2002. Convergence of visual and tactile shapeprocessing in the human lateral occipital complex. Cereb. Cortex 12(11):1202–12

Anderson KC, Siegel RM. 2005. Three-dimensional structure-from-motion selectivity in the anterior superiortemporal polysensory area, STPa, of the behaving monkey. Cereb. Cortex 15(9):1299–307

Anzai A, DeAngelis GC. 2010. Neural computations underlying depth perception. Curr. Opin. Neurobiol.20:367–75

Arcizet F, Jouffrais C, Girard P. 2009. Coding of shape from shading in area V4 of the macaque monkey. BMCNeurosci. 10:140

Arterberry ME, Yonas A. 2000. Perception of three-dimensional shape specified by optic flow by 8-week oldinfants. Percept. Psychophys. 62(3):550–56

Beer AL, Watanabe T, Ni R, Sasaki Y, Andersen GJ. 2009. 3D surface perception from motion involves atemporal-parietal network. Eur. J. Neurosci. 30(4):703–13

Born RT, Bradley DC. 2005. Structure and function of visual area MT. Annu. Rev. Neurosci. 28:157–89Borra E, Belmalih A, Calzavara R, Gerbella M, Murata A, et al. 2008. Cortical connections of the macaque

anterior intraparietal (AIP) area. Cereb. Cortex 18(5):1094–111Bradley DC, Chang GC, Andersen RA. 1998. Encoding of three-dimensional structure-from-motion by

primate MT neurons. Nature 392:714–17Buracas GT, Albright TD. 1996. Contribution of area MT to perception of three-dimensional shape: a

computational study. Vis. Res. 36(6):869–87Chandrasekaran C, Canon V, Dahmen JC, Kourtzi Z, Welchman AE. 2007. Neural correlates of disparity-

defined shape discrimination in the human brain. J. Neurophysiol. 97(2):1553–65Colombo M, D’Amato MR, Rodman HR, Gross CG. 1990. Auditory association cortex lesions impair auditory

short-term memory in monkeys. Science 247(4940):336–38Cornette L, Dupont P, Orban GA. 2002. The neural substrate of orientation short-term memory and resistance

to distractor items. Eur. J. Neurosci. 15(1):165–75Cumming BG, Shapiro SE, Parker AJ. 1998. Disparity detection in anticorrelated stereograms. Perception

27(11):1367–77DeYoe EA, Carman GJ, Bandettini P, Glickman S, Wieser J, et al. 1996. Mapping striate and extrastriate

visual areas in human cerebral cortex. Proc. Natl. Acad. Sci. USA 93(6):2382–86Dodd JV, Krug K, Cumming BG, Parker AJ. 2001. Perceptually bistable three-dimensional figures evoke high

choice probabilities in cortical area MT. J. Neurosci. 21(13):4809–21Duffy CJ, Wurtz RH. 1997. Medial superior temporal area neurons respond to speed patterns in optic flow.

J. Neurosci. 17(8):2839–51Durand JB, Nelissen K, Joly O, Wardak C, Todd JT, et al. 2007. Anterior regions of monkey parietal cortex

process visual 3D shape. Neuron 55(3):493–505Durand JB, Peeters R, Norman JF, Todd JT, Orban GA. 2009. Parietal regions processing visual 3D shape

extracted from disparity. Neuroimage 46(4):1114–26


Ann

u. R

ev. N

euro

sci.

2011

.34:

361-

388.

Dow

nloa

ded

from

ww

w.a

nnua

lrev

iew

s.or

gby

Uni

vers

ity o

f A

berd

een

on 0

9/20

/13.

For

per

sona

l use

onl

y.


Fleming RW, Torralba A, Adelson EH. 2004. Specular reflections and the perception of shape. J. Vis. 4(9):798–820

Freiwald WA, Tsao DY, Livingstone MS. 2009. A face feature space in the macaque temporal lobe. Nat.Neurosci. 12(9):1187–96

Fried I, MacDonald KA, Wilson CL. 1997. Single neuron activity in human hippocampus and amygdaladuring recognition of faces and objects. Neuron 18:753–65

Georgieva S, Peeters R, Kolster H, Todd JT, Orban GA. 2009. The processing of three-dimensional shapefrom disparity in the human brain. J. Neurosci. 29(3):727–42

Georgieva S, Todd JT, Peeters R, Orban GA. 2008. The extraction of 3D shape from texture and shading inthe human brain. Cereb. Cortex 18(10):2416–38

Gerardin P, Kourtzi Z, Mamassian P. 2010. Prior knowledge of illumination for 3D perception in the humanbrain. Proc. Natl. Acad. Sci. USA 107(37):16309–14

Gibson JJ. 1950. The Perception of the Visual World. Boston: Houghton MifflinGiese MA, Poggio T. 2003. Neural mechanisms for the recognition of biological movements. Nat. Rev.

Neurosci. 4(3):179–92Gunderson VM, Yonas A, Sargent PL, Grant-Webster KS. 1993. Infant macaque monkeys respond to pictorial

depth. Psychol. Sci. 4:93–98Hanazawa A, Komatsu H. 2001. Influence of the direction of elemental luminance gradients on the responses

of V4 cells to textured surfaces. J. Neurosci. 21(12):4490–97Hegde J, Van Essen DC. 2005. Role of primate visual area V4 in the processing of 3-D shape characteristics

defined by disparity. J. Neurophysiol. 94(4):2856–66Hinkle DA, Connor CE. 2002. Three-dimensional orientation tuning in macaque area V4. Nat. Neurosci.

5(7):665–70Humphrey GK, Goodale MA, Bowen CV, Gati JS, Vilis T, et al. 1997. Differences in perceived shape from

shading correlate with activity in early visual areas. Curr. Biol. 7:144–47Iwaki S, Bonmassar G, Belliveau JW. 2007. Neuromagnetic brain responses during 3D object perception from

2D optic flow. Int. Congr. Ser. 1300:543–46Jackson S, Blake R. 2010. Neural integration of information specifying human structure from form, motion

and depth. J. Neurosci. 30(3):838–48Janssen P, Vogels R, Liu Y, Orban GA. 2001. Macaque inferior temporal neurons are selective for three-

dimensional boundaries and surfaces. J. Neurosci. 21(23):9419–29Janssen P, Vogels R, Liu Y, Orban GA. 2003. At least at the level of inferior temporal cortex, the stereo

correspondence problem is solved. Neuron 37(4):693–701Janssen P, Vogels R, Orban GA. 1999. Macaque inferior temporal neurons are selective for disparity-defined

three-dimensional shapes. Proc. Natl. Acad. Sci. USA 96(14):8217–22Janssen P, Vogels R, Orban GA. 2000a. Selectivity for 3D shape that reveals distinct areas within macaque

inferior temporal cortex. Science 288(5473):2054–56Janssen P, Vogels R, Orban GA. 2000b. Three-dimensional shape coding in inferior temporal cortex. Neuron

27(2):385–97Jiang Y, Boehler CN, Nonnig N, Duzel E, Hopf JM, et al. 2008. Binding 3-D object perception in the human

visual cortex. J. Cogn. Neurosci. 20(4):553–62Joly O, Vanduffel W, Orban GA. 2009. The monkey ventral premotor cortex processes 3D shape from

disparity. Neuroimage 47(1):262–72Katsuyama N, Yamashita A, Sawada K, Naganuma T, Sakata H, Taira M. 2010. Functional and histological

properties of caudal intraparietal area of macaque monkey. Neuroscience 167(1):1–10Klaver P, Lichtensteiger J, Bucher K, Dietrich T, Loenneker T, Martin E. 2008. Dorsal stream development

in motion and structure-from-motion. Neuroimage 39:1815–23Koenderink JJ. 1990. Solid Shape. Cambridge, MA: MIT PressKoenderink JJ, van Doorn AJ, Kappers AM, Todd JT. 2001. Ambiguity and the ‘mental eye’ in pictorial relief.

Perception 30(4):431–48Kolster H, Mandeville JB, Arsenault JT, Ekstrom LB, Wald LL, Vanduffel W. 2009. Visual field map clusters

in macaque extrastriate cortex. J. Neurosci. 29(21):7031–39

384 Orban

Ann

u. R

ev. N

euro

sci.

2011

.34:

361-

388.

Dow

nloa

ded

from

ww

w.a

nnua

lrev

iew

s.or

gby

Uni

vers

ity o

f A

berd

een

on 0

9/20

/13.

For

per

sona

l use

onl

y.


Kolster H, Peeters R, Orban GA. 2010. The retinotopic organization of the human middle temporal areaMT/V5 and its cortical neighbors. J. Neurosci. 30(29):9801–20

Konen CS, Kastner S. 2008. Two hierarchically organized neural systems for object information in humanvisual cortex. Nat. Neurosci. 11(2):224–31

Kourtzi Z, Connor CE. 2011. Neural representations for object perception: structure, category, and adaptivecoding. Annu. Rev. Neurosci. 34:45–67

Kourtzi Z, Erb M, Grodd W, Bulthoff HH. 2003. Representation of the perceived 3-D object shape in thehuman lateral occipital complex. Cereb. Cortex 13(9):911–20

Kourtzi Z, Kanwisher N. 2000. Cortical regions involved in perceiving object shape. J. Neurosci. 20(9):3310–18Kriegeskorte N, Sorger B, Naumer M, Schwarzbach J, van den BE, et al. 2003. Human cortical object recog-

nition from a visual motion flowfield. J. Neurosci. 23(4):1451–63Laeng B, Caviness VS. 2001. Prosopagnosia as a deficit in encoding curved surface. J. Cogn. Neurosci. 13(5):556–

76Li G, Zucker SW. 2006a. Contextual inference in contour-based stereo correspondence. Int. J. Comput. Vis.

69:59–75Li G, Zucker SW. 2006b. Differential geometric consistency extends stereo to curved surfaces. Proc. Eur. Conf.

Comp. Vis. (ECCV’06), ed. A Leonardis, H Bischof, A Pinz, pp. 44–57. New York: Springer VerlagLiu Y, Vogels R, Orban GA. 2002. The effect of texture depth cue on the disparity selectivity of macaque

inferior temporal neurons. Soc. Neurosci. Abstr. 56.13Liu Y, Vogels R, Orban GA. 2004. Convergence of depth from texture and depth from disparity in macaque

inferior temporal cortex. J. Neurosci. 24(15):3795–800Logothetis NK, Guggenberger H, Peled S, Pauls J. 1999. Functional imaging of the monkey brain. Nat.

Neurosci. 2:555–62Marr D. 1982. Vision. San Francisco: FreemanMartinez-Trujillo JC, Tsotsos JK, Simine E, Pomplun M, Wildes R, et al. 2005. Selectivity for speed gradients

in human area MT/V5. NeuroReport 16(5):435–38Matsumora T, Koida K, Komatsu H. 2008. Relationship between color discrimination and neural responses

in the inferior temporal cortex of the monkey. J. Neurophysiol. 100(6):3361–74Maunsell JH, Sclar G, Nealey TA, DePriest DD. 1991. Extraretinal representations in area V4 in the macaque

monkey. Vis. Neurosci. 7(6):561–73Moore C, Engel SA. 2001. Neural response to perception of volume in the lateral occipital complex. Neuron

29(1):277–86Murray SO, Olshausen BA, Woods DL. 2003. Processing shape, motion and three-dimensional shape-from-

motion in the human cortex. Cereb. Cortex 13(5):508–16Mysore SG, Vogels R, Kolster H, Vanduffel W, Orban GA. 2010a. Cortical network of 3D-structure from

motion (3D-SFM) in the macaque: a functional imaging study. Soc. Neurosci. Abstr. 776.9Mysore SG, Vogels R, Raiguel SE, Orban GA. 2006. Processing of kinetic boundaries in macaque V4.

J. Neurophysiol. 95(3):1864–80Mysore SG, Vogels R, Raiguel SE, Todd JT, Orban GA. 2010b. The selectivity of FST neurons for 3D

structure from motion. J. Neurosci. 30(46):15491–508Nadler JW, Angelaki DE, DeAngelis GC. 2008. A neural representation of depth from motion parallax in

macaque visual cortex. Nature 452(7187):642–45Nadler JW, Nawrot M, Angelaki DE, DeAngelis GC. 2009. MT neurons combine visual motion with smooth

eye movement signal to code depth-sign from motion parallax. Neuron 63:523–32Nakamura H, Kuroda T, Wakita M, Kusunoki M, Kato A, et al. 2001. From three-dimensional space vision to

prehensile hand movements: the lateral intraparietal area links the area V3A and the anterior intraparietalarea in macaques. J. Neurosci. 21(20):8174–87

Nelissen K, Joly O, Durand JB, Todd JT, Vanduffel W, Orban GA. 2009. The extraction of depth structurefrom shading and texture in the macaque brain. PLoS One 4(12):e8306

Nelissen K, Vanduffel W, Orban GA. 2006. Charting the lower superior temporal region, a new motion-sensitive region in monkey superior temporal sulcus. J. Neurosci. 26(22):5929–47

Nguyenkim JD, DeAngelis GC. 2003. Disparity-based coding of three-dimensional surface orientation bymacaque middle temporal neurons. J. Neurosci. 23(18):7117–28


Ann

u. R

ev. N

euro

sci.

2011

.34:

361-

388.

Dow

nloa

ded

from

ww

w.a

nnua

lrev

iew

s.or

gby

Uni

vers

ity o

f A

berd

een

on 0

9/20

/13.

For

per

sona

l use

onl

y.


Nguyenkim JD, DeAngelis GC. 2004. Macaque MT neurons are selective for 3D surface orientation definedby multiple cues. Soc. Neurosci. Abstr. 368.12

Norman JF, Todd JT, Orban GA. 2004. Perception of three-dimensional shape from specular highlights,deformations of shading, and other types of visual information. Psychol. Sci. 15(8):565–70

Orban GA, Claeys K, Nelissen K, Smans R, Sunaert S, et al. 2006a. Mapping the parietal cortex of humanand non-human primates. Neuropsychologia 44(13):2647–67

Orban GA, Janssen P, Vogels R. 2006b. Extracting 3D structure from disparity. Trends Neurosci. 29(8):466–73Orban GA, Sunaert S, Todd JT, Van HP, Marchal G. 1999. Human cortical regions involved in extracting

depth from motion. Neuron 24(4):929–40Orban GA, Van Essen DC, Vanduffel W. 2004. Comparative mapping of higher visual areas in monkeys and

humans. Trends Cogn. Sci. 8(7):315–24Orban GA, Vogels R. 1998. The neuronal machinery involved in successive orientation discrimination. Prog.

Neurobiol. 55(2):117–47Paradis AL, Cornilleau-Peres V, Droulez J, van-de-Moortele PF, Lobel E, et al. 2000. Visual perception of

motion and 3-D structure from motion: an fMRI study. Cereb. Cortex 10:772–83Patsupathy A, Connor CE. 2002. Population coding of shape in area V4. Nat. Neurosci. 5(12):1332–38Perrett DI, Smith PA, Mistlin AJ, Chitty AJ, Head AS, et al. 1985. Visual analysis of body movements by

neurones in the temporal cortex of the macaque monkey: a preliminary report. Behav. Brain Res. 16(2–3):153–70

Peuskens H, Claeys KG, Todd JT, Norman JF, Van HP, Orban GA. 2004. Attention to 3-D shape, 3-Dmotion, and texture in 3-D structure from motion displays. J. Cogn. Neurosci. 16(4):665–82

Pizlo Z, Sawada T, Li Y, Kropatsch WG, Steinman RM. 2010. New approach to the perception of 3D shapebased on veridicality, complexity, symmetry and volume. Vis. Res. 50:1–11

Preston TJ, Kourtzi Z, Welchman AE. 2009. Adaptive estimation of three-dimensional structure in the humanbrain. J. Neurosci. 29(6):1688–98

Preston TJ, Li S, Kourtzi Z, Welchman AE. 2008. Multivoxel pattern selectivity for perceptually relevantbinocular disparities in the human brain. J. Neurosci. 28(44):11315–27

Roy JP, Komatsu H, Wurtz RH. 1992. Disparity sensitivity of neurons in monkey extrastriate area MST.J. Neurosci. 12(7):2478–92

Sakata H, Tsutsui K, Taira M. 2005. Toward an understanding of the neural processing for 3D shape percep-tion. Neuropsychologia 43(2):151–61

Sawamura H, Orban GA, Vogels R. 2006. Selectivity of neuronal adaptation does not match response selec-tivity: a single-cell study of the FMRI adaptation paradigm. Neuron 49(2):307–18

Saygin AP, Sereno MI. 2008. Retinotopy and attention in human occipital, temporal, parietal, and frontalcortex. Cereb. Cortex 18(9):2158–68

Schiller PH, Slocum WM, Jao B, Weiner VS. 2011. The integration of disparity, shading and motion parallaxcues for depth perception in humans and monkeys. Brain Res. 1377:67–77

Sereno ME, Trinath T, Augath M, Logothetis NK. 2002. Three-dimensional shape representation in monkeycortex. Neuron 33:635–52

Shikata E, Hamzei F, Glauche V, Knab R, Dettmers C, et al. 2001. Surface orientation discrimination ac-tivates caudal and anterior intraparietal sulcus in humans: an event-related fMRI study. J. Neurophysiol.85(3):1309–14

Shikata E, McNamara A, Sprenger A, Hamzei F, Glauche V, et al. 2008. Localization of human intraparietalareas AIP, CIP, and LIP using surface orientation and saccadic eye movement tasks. Hum. Brain Mapp.29(4):411–21

Shikata E, Tanaka Y, Nakamura H, Taira M, Sakata H. 1996. Selectivity of the parietal visual neurons in 3Dorientation of surface of stereoscopic stimuli. NeuroReport 7(14):2389–94

Siegel RM, Andersen RA. 1990. The perception of structure from visual motion in monkey and man. J. Cogn.Neurosci. 2:306–19

Singer JM, Sheinberg DL. 2010. Temporal cortex neurons encode articulated actions as slow sequences ofintegrated poses. J. Neurosci. 30(8):3133–45

Srivastava S, Orban GA, De Maziere PA, Janssen P. 2009. A distinct representation of three-dimensionalshape in macaque anterior intraparietal area: fast, metric, and coarse. J. Neurosci. 29(34):10613–26

386 Orban

Ann

u. R

ev. N

euro

sci.

2011

.34:

361-

388.

Dow

nloa

ded

from

ww

w.a

nnua

lrev

iew

s.or

gby

Uni

vers

ity o

f A

berd

een

on 0

9/20

/13.

For

per

sona

l use

onl

y.


Srivastava S, Orban GA, Janssen P. 2008. Disparity-defined 3D boundary and surface selectivity in macaquearea AIP. Soc. Neurosci. Abstr. 462.4

Sugihara H, Murakami I, Shenoy KV, Andersen RA, Komatsu H. 2002. Response of MSTd neurons tosimulated 3D orientation of rotating planes. J. Neurophysiol. 87(1):273–85

Taira M, Nose I, Inoue K, Tsutsui K. 2001. Cortical areas related to attention to 3D surface structures basedon shading: an fMRI study. Neuroimage 14:959–66

Taira M, Tsutsui KI, Jiang M, Yara K, Sakata H. 2000. Parietal neurons represent surface orientation fromthe gradient of binocular disparity. J. Neurophysiol. 83(5):3140–46

Theys T, Van Loon J, Goffin J, Janssen P. 2009. Selectivity for disparity-defined three-dimensional shape inmacaque premotor cortex. Soc. Neurosci. Abstr. 852.9

Todd JT. 2004. The visual perception of 3D shape. Trends Cogn. Sci. 8(3):115–21Todd JT, Thaler L. 2010. The perception of 3D shape from texture based on directional width gradients.

J. Vis. 10(5):17:1–13Tootell RB, Mendola JD, Hadjikhani NK, Ledden PJ, Liu AK, et al. 1997. Functional analysis of V3A and

related areas in human visual cortex. J. Neurosci. 17(18):7060–78Tsao DY, Freiwald WA, Knutsen TA, Mandeville JB, Tootell RB. 2003a. Faces and objects in macaque

cerebral cortex. Nat. Neurosci. 6(9):989–95Tsao DY, Vanduffel W, Sasaki Y, Fize D, Knutsen TA, et al. 2003b. Stereopsis activates V3A and caudal

intraparietal areas in macaques and humans. Neuron 39(3):555–68Tsuruhara A, Sawada T, Kanazawa S, Yamaguchi MK, Yonas A. 2009. Infant’s ability to form a common

representation of an object’s shape from different pictorial cues: a transfer-across-cues study. Infant Behav.Dev. 32(4):468–75

Tsutsui K, Jiang M, Sakata H, Taira M. 2003. Short-term memory and perceptual decision for three-dimensional visual features in the caudal intraparietal sulcus (area CIP). J. Neurosci. 23(13):5486–95

Tsutsui K, Jiang M, Yara K, Sakata H, Taira M. 2001. Integration of perspective and disparity cues in surface-orientation-selective neurons of area CIP. J. Neurophysiol. 86(6):2856–67

Tsutsui K, Sakata H, Naganuma T, Taira M. 2002. Neural correlates for perception of 3D surface orientationfrom texture gradient. Science 298(5592):409–12

Tsutsui K, Taira M, Sakata H. 2005. Neural mechanisms of three-dimensional vision. Neurosci. Res. 51(3):221–29

Ungerleider LG, Desimone R. 1986. Cortical connections of visual area MT in the macaque. J. Comp Neurol.248(2):190–222

Ungerleider LG, Mishkin M. 1982. Two cortical visual systems. In Analysis of Visual Behavior, ed. DJ Ingle,MA Goodale, RJW Mansfield, pp. 549–86. Cambridge: MIT Press

Van Dromme IC, Janssen P, Kolster H, Vanduffel W. 2010. Mapping 3D-shape processing in the macaquemonkey. Soc. Neurosci. Abstr. 19.7

Vanduffel W, Fize D, Mandeville JB, Nelissen K, Van HP, et al. 2001. Visual motion processing investigatedusing contrast agent-enhanced fMRI in awake behaving monkeys. Neuron 32(4):565–77

Vanduffel W, Fize D, Peuskens H, Denys K, Sunaert S, et al. 2002a. Extracting 3D from motion: differencesin human and monkey intraparietal cortex. Science 298(5592):413–15

Vanduffel W, Tootell RB, Schoups AA, Orban GA. 2002b. The organization of orientation selectivity through-out macaque visual cortex. Cereb. Cortex 12(6):647–62

Vangeneugden J, Koteles K, Orban GA, Vogels R. 2006. The coding of 3-D shape from shading in macaqueareas TE and TEO. Perception 35(Suppl.):34

Verhoef BE, Vogels R, Janssen P. 2010. Contribution of inferior temporal and posterior parietal activity tothree-dimensional shape perception. Curr. Biol. 20(10):909–13

Welchman AE, Deubelius A, Conrad V, Bulthoff HH, Kourtzi Z. 2005. 3D shape perception from combineddepth cues in human visual cortex. Nat. Neurosci. 8(6):820–27

Xiao DK, Raiguel S, Marcar V, Koenderink J, Orban GA. 1995. Spatial heterogeneity of inhibitory surroundsin the middle temporal visual area. Proc. Natl. Acad. Sci. USA 92(24):11303–6

Xiao DK, Marcar VL, Raiguel SE, Orban GA. 1997a. Selectivity of macaque MT/V5 neurons for surfaceorientation in depth specified by motion. Eur. J. Neurosci. 9(5):956–64


Ann

u. R

ev. N

euro

sci.

2011

.34:

361-

388.

Dow

nloa

ded

from

ww

w.a

nnua

lrev

iew

s.or

gby

Uni

vers

ity o

f A

berd

een

on 0

9/20

/13.

For

per

sona

l use

onl

y.


Xiao DK, Raiguel S, Marcar V, Orban GA. 1997b. The spatial distribution of the antagonistic surround ofMT/V5 neurons. Cereb. Cortex 7(7):662–77

Yamamoto T, Takahashi S, Hankawa T, Urayama S-I, Aso T, et al. 2008. Neural correlates of the stereokineticeffect revealed by functional magnetic resonance imaging. J. Vis. 8(10):1–17

Yamane Y, Carlson ET, Bowman KC, Wang Z, Connor CE. 2008. A neural code for three-dimensional objectshape in macaque inferotemporal cortex. Nat. Neurosci. 11(11):1352–60

Yonas A, Arterberry ME, Granrud CE. 1987. Four-month-old infants’ sensitivity to binocular and kineticinformation for three-dimensional-object shape. Child Dev. 58(4):910–17

Zhang Y, Schiller PH. 2008. The effect of overall stimulus velocity on motion parallax. Vis. Neurosci. 25(1):3–15Zhang Y, Weiner VS, Slocum WM, Schiller PH. 2007. Depth from shading and disparity in humans and

monkeys. Vis. Neurosci. 24(2):207–15

RELATED RESOURCES

Orban GA. 2008. Higher order visual processing in macaque extrastriate cortex. Physiol. Rev.88(1):59–89

Orban GA. 2008. Three-dimensional shape: cortical mechanisms of shape extraction. In TheSenses: A Comprehensive Reference, ed. AI Basbaum, A Kaneko, GM Shepherd, G Westheimer,pp. 245–74. San Diego: Academic

388 Orban

Ann

u. R

ev. N

euro

sci.

2011

.34:

361-

388.

Dow

nloa

ded

from

ww

w.a

nnua

lrev

iew

s.or

gby

Uni

vers

ity o

f A

berd

een

on 0

9/20

/13.

For

per

sona

l use

onl

y.

Documents

The Extraction of 3D Shape in the Visual System of Human and Nonhuman Primates