PI: CONNOR, CHARLES E 9774 42 121 ication · Johns Hopkins University Krieger Mind/Brain Institute 3400 N Charles Street / 338 Krieger Hall Baltimore MD 21218 E-MAIL ADDRESS: [email protected]

Form Approved Through 05/2004

9 7 7 4

_ Department of Health and Human

4 2 ication121

PI: CONNOR, CHARLES ECouncil: 05/2005

1 R01 EY016711-01

Dual:

IPF:4134401

' Do not exceed characteriength restrictions indicated. IRG: CVP Received: 1U/UWUU

1 . TITLE OF' PROJECT (Do not exceed 56 characters, including spa.

Neural coding of complex 3D shape' •

2. RESPONSE TO SPECIFIC REQUEST FOR APPLICATIONS OR PROGRAM ANNOUNCEMENT OR SOLICITATION |3 NO(If "Yes, " state number and title)Number: Title:

3. PRINCIPAL INVESTIGATOR/PROGRAM DIRECTOR

3a. NAME (Last, first, middle)Connor, Charles Edward

3c. POSITION TITLEAssociate Professor

3e. DEPARTMENT, SERVICE, LABORATORY, OR EQUIVALENTKrieger Mind/Brain Institute

3f. MAJOR SUBDIVISIONFaculty of Arts and Sciences

3g. TELEPHONE AND FAX (Area code, number and extension)

TEL: 410-516-7342 FAX: 410-516-8648

4. HUMAN SUBJECTS 4a. Research Exempt D No Q YesRESEARCH if -Yes," Exemption No.

S NO 4b Human Subjects 4c. NIH-defmed Phase IIIr~j yes Assurance No. Clinical Trial

D No D Yes

DYES

New Investigator ^ No I I Yes .

3b. DEGREE(S)

Ph.D.3d. MAILING ADDRESS (Street, city, state, zip code)

Johns Hopkins University

Krieger Mind/Brain Institute3400 N Charles Street / 338 Krieger HallBaltimore MD 21218

E-MAIL ADDRESS:

[email protected]. VERTEBRATE ANIMALS Q No M Yes

Sa. If "Yes,' lACUC 5b. Animal welfare assurance no.approval Date

03/25/2004 A3272-016. DATES OF PROPOSED PERIOD OF 7. COSTS REQUESTED FOR INITIAL 8. COSTS REQUESTED FOR PROPOSED

SUPPORT (month, day, year—MM/DD/YY) BUDGET PERIOD PERIOD OF SUPPORTFrom Through 7a. Direct Costs (S)

07/01/05 06/30/09 $250,0009. APPLICANT ORGANIZATION

Name Johns Hopkins University

Address Krieger Mind/Brain Inst \

3400 N. Charles Street ;338 Krieger HallBaltimore, MD 21218

Institutional Profile File Number (if known) 41 34401

12. ADMINISTRATIVE OFFICIAL TO BE NOTIFIED IF AWARD IS MADE

Name Nancy Kerner

Title Sponsored Projects Officer

Address 34o0 N. Charles Street

225 Mergenthaler HallBaltimore, MD 21 21 8

Tel: 410-516-8841 FAX: 410-516-5063E-Mail: [email protected]. PRINCIPAL INVESTIGATOR/PROGRAM DIRECTOR ASSURANCE: I certify that thestatements herein are true, complete and accurate to the best of my knowledge. I amaware that any false, fictitious, or fraudulent statements or claims may subject me tocriminal, civil, or administrative penalties. I agree to accept responsibility for the scientificconduct of the project and to provide the required progress reports if a grant is awarded asa result of this application.15. APPLICANT ORGANIZATION CERTIFICATION AND ACCEPTANCE: I certify thatthe statements herein are true, complete and accurate to the best of my knowledge, andaccept the obligation to comply with Public Health Services terms and conditions if a grantis awarded as a result of this application. I am aware that any false, fictitious, or fraudulentstatements or claims may subject me to criminal, civil, or administrative penalties.

7b. Total Costs ($) 8a. Direct Costs ($) 8b. Tot

£ 4 0 7 , 5 0 0 $1,000,000 $il

al Costs (S)

,636., 25010. TYPE OF ORGANIZATION

Public: -> D Federal D State Q Local

Private: -»• £3 Private Nonprofit

For-profit: -> I I General I I Small Business

C] Woman-owned O Socially and Economically Disadvantaged

--- ---------- -- ---------- IFICATION NUMBER- ---------- - ------ -

DUNS NO. 00-191-0777

Congressional District Seventh

13. OFFICIAL SIGNING FOR APPLICANT ORGANIZATION

Name Nancy Kerner

Title Sponsored Projects Officer

Address 3400 N. Charles Street

225 Mergenthaler HallBaltimore, MD21218

Tel: 410-516-8841 FAX: 410-516-5369E-Mail: [email protected] OF PI/PD NAMED IN 3a.(In ink. "Per" signature not acceptable.)

(yn /U/i L^A/ /

SIGNATURE OF OFFICIAL NAMED IN 13.(In ink. "Per" signature not acceptable.)

DATE

DATE

PHS 398 (Rev. 05/01) Face Page Form Page 1

Principal Investigator/Program Director (Last, First, Middle): Connor, Charles Edward

DESCRIPTION: State the application's broad, long-term objectives and specific aims, making reference to the health relatedness of the project Describeconcisely the research design and methods for achieving these goals. Avoid summaries of past accomplishments and the use of the first person. This abstractis meant to serve as a succinct and accurate description of the proposed work when separated from the application. If the application is funded, thisdescription, as is, will become public information. Therefore, do not include proprietary/confidential information. DO NOT EXCEED THE SPACEPROVIDED.

Our ability to live within and interact with a world composed of 3D objects depends largely on our spectacularcapacity for visual shape perception. This is what makes vision so critical to our health, happiness, andsurvival. The long-term goal of this project is to understand 3D object perception by discovering the neuralcode for complex 3D shape in the primate ventral visual pathway. After decades in which neurophysiologicalstudies of object representation in the monkey ventral pathway have focused exclusively on 2D shape,recent reports indicate a robust representation of 3D shape, although the nature of that representationremains completely unknown. We will address this issue using the same techniques we have recentlyapplied to produce the first quantitative descriptions of complex 2D shape representation. We will combinedense, parametric exploration of 3D shape space with intensive computational analysis to test hypothesesabout 3D shape coding dimensions, tuning functions, integration mechanisms, and population codingprinciples. The stimuli will be complex, smooth (spline-based), abstract, randomly generated 3D shapes.Successive generations of random shape stimuli will be determined with a genetic algorithm, using neuralresponses as feedback to guide sampling toward the most relevant regions of 3D shape space. Theresulting data will be used to test hypotheses about coding dimensions relating to 2D boundary contours, 3Dsurface patches, and 3D medial axis shape, all described in terms of absolute and relative position, 2D and3D orientation, 2D and 3D curvature, curvature orientation, and curvature derivative. We will test tuningfunctions ranging from simple Gaussians to complex manifolds describing highly specific part shapes. Wewill test a variety of mechanisms for integrating information across object parts, ranging from single-parttuning through multi-part tuning to holistic tuning for overall object shape. The hypotheses surviving fromthese individual cell analyses will then be tested at the population coding level.

PERFORMANCE SITE(S) (organization, city, state)

Zanvyl Krieger Mind/Brain Institute, Johns Hopkins University, Baltimore MD

KEY PERSONNEL. See instructions. Use continuation pages as needed to provide the required information in the format shown below.Start with Principal Investigator. List all other key personnel in alphabetical order, last name firstName Organization Role on Project

Connor, Charles, Ph.D. Johns Hopkins University Principle Investigator------- ----------- -------- --------- Johns Hopkins University Software Engineer----------- ----------- ------- --------- Johns Hopkins University Postdoctoral Fellow

Disclosure Permission Statement Applicable to SBIR/STTR Only. See instructions. Q Yes Q No

PHS 398 (Rev. 05/01) Page 2 Number pages consecutively at the bottom throughout Form Page 2the application. Do not use suffixes such as 2a, 2b.


The name of the principal investigator/program director must be provided at the top of each printed page and each continuation page.

RESEARCH GRANT

TABLE OF CONTENTSPage Numbers

Face Page

Description, Performance Sites, and PersonnelTable of Contents

Detailed Budget for Initial Budget Period (or Modular Budget)Budget for Entire Proposed Period of Support (not applicable with Modular Budget)Budgets Pertaining to Consortium/Contractual Arrangements (not applicable with Modular Budget)Biographical Sketch - Principal Investigator/Program Director {Not to exceed four pages)Other Biographical Sketches (Not to exceed four pages for each - See instructions)Resources

Research Plan.

(Items A-D: not to exceed 25 pages*)

* SBIR/STTR Phase I: Items A-D limited to 15 pages.

Introduction to Revised Application (Not to exceed 3 pages)

Introduction to Supplemental Application (Not to exceed one page)

A. Specific Aims

B. Background and SignificanceC. Preliminary Studies/Progress Report/

Phase I Progress Report (SBIR/STTR Phase II ONLY)

D. Research Design and Methods

E. Human Subjects

Protection of Human Subjects (Required if Item 4 on the Face Page is marked "Yes")

Inclusion of Women (Required if Item 4 on the Face Page is marked "Yes")

Inclusion of Minorities (Required if Item 4 on the Face Page is marked "Yes")

Inclusion of Children (Required if Item 4 on the Face Page is marked "Yes")

Data and Safety Monitoring Plan (Required if Item 4 on the Face Page is marked "Yes" and a Phase I, II, or III clinical

trial is proposed

F. Vertebrate Animals

G. Literature Cited

H. Consortium/Contractual Arrangements

I. Letters of Support (e.g., Consultants)

J. Product Development Plan (SBIR/STTR Phase II and Fast-Track ONLY)

Checklist.

Appendix (Five collated sets. No page numbering necessary for Appendix.)

Appendices NOT PERMITTED for Phase I SBIR/STTR unless specifically solicited.

Number of publications and manuscripts accepted for publication (not to exceed 10)

Other items (list):

1

4-5

6-8

12

13

131415

24

3234

41

Check ifAppendix isIncluded

PHS 398 (Rev. 05/01) Page 3 Form Page 3

Principal Investigator/Program Director (Last, first, middle): Connor. Charles Edward

BUDGET JUSTIFICATION PAGEMODULAR RESEARCH GRANT APPLICATION

Initial Budget Period

250,000

Second Year of Support

250,000

Third Year of Support

250,000

Fourth Year of Support

250,000

Fifth Year of Support

Total Direct Costs Requested for Entire Project Period 1 <\ QQQ QQQ

Personnel

Charles Connor, Ph.D., (Principal Investigator,---- % effort) will be responsible for designing andoverseeing experiments, developing quantitative analyses of neural and behavioral data, andpreparing the results for publication in collaboration with Yukako Yamane and other students. Inaddition, Dr. Connor will perform all surgeries required for these experiments.

-------- - ----------- Ph.D. (Postdoctoral Fellow,------ % effort) will be responsible for experimental designin collaboration with the PI, software development to create 3D stimuli and analyze neural data,performing neurophysiological recording experiments, and preparation of results for publication incollaboration with the PI. --- ---------- - did her Ph.D. thesis work in the laboratory of Manabu Tanifujiat RIKEN Brain Science Institute, studying 2D object representation in primate inferotemporal cortex.

--------- - -------- Ph.D. (Senior Progammer/Analyst,---- % effort) will be responsible for developingsoftware and managing hardware for - -- ------- - display, neural and behavioral data collection, andneural and behavioral data analysis. --- ------ - received a PhD. in Biophysics with an emphasis on3D visual perception and has extensive experience in software and hardware development, makinghim uniquely qualified for this project.

------ - ------ -- (Technician, --- % effort) will be responsible for histological procedures, supplementarybehavioral training, ordering supplie- ----------- - in surgeries and experiments, and generating figuresfor presentations and publications. ---- ------ -- - has been trained in all these procedures as part of her3 years of experience in this laboratory.

---------- - -------- - (Electronics Engineer, - % effort) will be responsible for design, manufacture and------ --- ---- stom electronic equipment needed for neurophysiological and behavioral experiments.--- -------- - supervises the Mind/Brain electronics shop and is responsible for the electronics needs of6 laboratories. He received a B.S. in electrical engineering and has 4 years of experience inelectronics design.

------- -- --------- - (Machinist, - % effort) will be responsible (along with---- ------- - for design,manufacture --- - ------- --- custom equipment needed for neurophysiological and behavioralexperiments. --- ----------- in collaboration with --- ------- , is responsible for the equipment needs of 6laboratories. He has 27 years experience as a machinist.

------- -- ------ - (Machinist,-- % effort) will be responsible (along with---- ----------- ) for design,manufacture and repair of custom equipment needed for neurophysiological and behavioralexperiments. --- ------ - supervises the Mind/Brain machine shop and is responsible for theequipment needs of 6 laboratories. He has 27 years experience as a machinist.PHS 398 (Rev. 05/01) Page.4 Modular Budget Format Page

Principal Investigator/Program Director (Last, first, middle): Connor. Charles Edward

------- - -------- - (Te----------- -- -- ------- ----- - be responsible for supplementary histological procedures incollaboration with ---- --------- ----- -------- - is in charge of the histology facility in the Mind/BrainInstitute. She oversees histology and electrode manufacture for 6 laboratories and has 4 years ofexperience.

------ -------------- - (Electrician,-- % effort) will be responsible (under the supervision of --- --------- - formanufacture and repair of custom electronics equipment needed for behavioral andneurophysiological experiments. He has 29 years of experience in manufacture and support oflaboratory electronics equipment.

Consortium

Fee (SBIR/STTR Only)

PHS 398 (Rev. 05/01) Page 5 Modular Budget Format Page


BIOGRAPHICAL SKETCHProvide the following information for trie key personnel in the order listed on Form Page 2.

Follow this format for each person. DO NOT EXCEED FOUR PAGES.

NAME

Connor, Charles EdwardPOSITION TITLE

Associate Professor

EDUCATION/TRAINING (Begin with baccalaureate or other initial profess/ona/ education, such as nursing, and include postdoctoral training.)

INSTITUTION AND LOCATION

Loyola College of MarylandVanderbilt University School of MedicineJohns Hopkins University School of MedicineJohns Hopkins University School of MedicineWashington University School of Medicine

DEGREE(if applicable)

B.S.M.S.Ph.D.

PostDocPostDoc

YEAR(s)

197819821989

1990-19921992-1996

FIELD OF STUDY

BiologyPharmacologyNeuroscienceNeurophysiologyNeurophysiology

A. Positions and Honors.

2003-present Associate Professor, Johns Hopkins University1996-2002 Assistant Professor, Johns Hopkins University1998-2002 Pew Scholarship in the Biomedical Sciences

B. Selected peer-reviewed publications (in chronological order).

Research PapersBrincat, S.L. & Connor, C.E. (2004) Underlying principles of visual shape selectivity in posterior

------------------ --------- -------- - ----------------- - -- ------------ -------- ------ -- ---------- ----- -------- ---------------- - -------------------- - -- ---------- - ------- - - - --------- ---------- -

---- - -- - ---------------- Pasupathy, A. & Connor, C.E. (2002) Population coding of shape in area V4. Nature Neuroscience 5:

1332-1338.Hinkle, D.A. & Connor, C.E. (2002) Three-dimensional orientation tuning in macaque area V4. Nature

Neuroscience 5: 665-670.Pasupathy, A. & Connor, C.E. (2001) Shape representation in area V4: Position-specific tuning for

boundary conformation. Journal of Neurophysiology 86: 2505-2519.Hinkle, D.A. & Connor, C.E. (2001) Disparity tuning in macaque area V4. NeuroReport 12: 365-369.Pasupathy, A. & Connor, C.E. (1999) Responses to contour features in macaque area V4. Journal of

Neurophysiology 82: 2490-2502.Gallant, J.L., Connor, C.E. & Van Essen, D.C. (1998) Neural activity in areas V1, V2 and V4 during

free viewing of natural scenes compared to controlled viewing. NeuroReport 9: 2153-2158.Connor, C.E., Preddie, D.C., Gallant, J.L. & Van Essen, D.C. (1997) Spatial attention effects in

macaque area V4. Journal of Neuroscience 17: 3201-3214.Connor, C.E., Gallant, J.L., Preddie, D.C. & Van Essen, D.C. (1996) Responses in area V4 depend

on the spatial relationship between stimulus and attention. Journal of Neurophysiology 7 5:1306-1308.

Gallant, J.L., Connor, C.E., Rakshit, S., Lewis, J. & Van Essen, D.C. (1996) Neural responses topolar, hyperbolic, and Cartesian gratings in area V4 of the macaque monkey. Journal ofNeurophysiology 76: 2718-2739.

PHS 398/2590 (Rev. 05/01) Page 6 Biographical Sketch Format Page


Steinmetz, MA, Connor, C.E., Constantinidis, C. & Mclaughlin, J.R. (1994) Covert attentionsuppresses neuronal responses in area 7a of the posterior parietal cortex. Journal ofNeurophysiology72: 1020-1023.

Connor, C.E. & Johnson, K.O. (1992) Neural coding of tactile texture: comparison of spatial andtemporal mechanisms for roughness perception. Journal of Neuroscience 12: 3414-3426.

Connor, C.E., Hsiao, S.S., Phillips, J.R. & Johnson, K.O. (1990) Tactile roughness: neural codes thataccount for psychophysical magnitude estimates. Journal of Neuroscience 10: 3823-3836.

Connor, C.E. & Kuczenski, R. (1986) Evidence that amphetamine and Na+ gradient reversal increasestriatal synaptosomal dopamine synthesis through carrier-mediated efflux of dopamine.Biochemical Pharmacology 35. 3123-3130.

Reviews. Comments and Chapters--------- ------ -------- --------- ------------ ------------ - ---- ------------- ---------- --------- - -- - --------- Connor, C.E. (2003) Active vision and visual activation in area V4. Neuron 40: 1056-1058.--------- ----- -------- -------- - -------------- - ---- - ------- ------------- ---- ---- - -------- --------------------

----------- -- -- ---------- ----- ------ ---- - -------- ------------- - --- -- -- - ----------------- Connor, C.E. (2003) Perceptual Learning. Quarterly Review of Biology 78: 259-260.Connor, C.E. (2002) Reconstructing a 3D world. Science 298: 376-377.Connor, C.E. (2002) Representing whole objects: temporal neurons learn to play their parts. Nature

Neuroscience 5: 1105-1106.Connor, C.E. (2001) Visual perception: sunny side up. Current Biology 11: R776-R778.Connor, C.E. (2001) Shifting receptive fields. Neuron 29: 548-549.Connor, C.E. (2001) Multiple cues for object perception. Trends in Neuroscience 24: 64-65.Connor, C.E. (2000) Visual perception: monkeys see things our way. Current Biology 10: R836-R838.

C. Research Support.

Ongoing Research Support

R01 EY11797 Connor (PI) 04/01/03-03/31/07NIH/NEIRole: PIObject Synthesis in Extrastriate Visual CortexThis is a study of 2D object representation in the primate ventral visual pathway, using large sets ofparameterized shape stimuli to elucidate mechanisms of shape coding at the single cell andpopulation levels in higher stages of monkey ventral pathway visual cortex.

P01 NS38034 Johnson (PI) 07/01/99-06/30/04NIH/NINDSRole: PI for Project 2Three-dimensional tactile and visual form perceptionThis was a program project grant involving four laboratories (two tactile, two visual) aimed atelucidating common mechanisms of 3D form perception in touch and vision. The research in ourlaboratory funded by this grant led to the first demonstration of binocular disparity tuning in area V4and the first demonstration of 3D orientation tuning. This grant is currently in a no-cost extensionperiod from 07/01/04-06/30/05. There is no plan to apply for renewal of this grant.

PHS 398/2590 (Rev. 05/01) Page 7 Continuation Format Page


Completed Research Support

---------- -- -- Connor (PI) 07/01/98-06/30/02------ -- --- -- ------------ - -------- - Object-based Attention in Extrastriate Visual CortexThe major goal of this project was to investigate the neural correlates of object-based attention inextrastriate cortex.

Overlap

P01NS38034. which does have scientific overlap with the proposed project, is currently in a no-costextension period, will end on 06/30/05. There is no plan to apply for renewal of P01NS38034. Thus,P01NS38034 will terminate before the earliest point at which the proposed project funding wouldbegin (07/01/05).

R01EY11797 has no scientific overlap with the proposed project (or with P01NS38034). For anumber of reasons, 2D and 3D object vision require very different scientific approaches, and thushave always represented two relatively independent lines of research in our laboratory. While almostnothing is known about 3D complex shape representation, a great deal is known about 2D shaperepresentation, and the experiments in R01EY11797 build on that background. 2D object shape hasa much simpler, lower-order dimensionality, making it more amenable to dense sampling of broadregions of shape space. 2D object vision is faster and does not depend on complex inferences aboutdepth based on subtle cues like binocular disparity, making it possible to sample shape responsesmuch more rapidly. 3D object vision appears to depend particularly on specific, more restricted partsof ventral pathway cortex, such as the temporal cortex inside the superior temporal sulcus. 3D shapeexperiments, especially those involving stereoscopic vision, entail a number of additional technicalchallenges in generating stimuli and ensuring fused, binocular vision and normal depth perception.Thus, the 2D project is bound to yield a more complete picture in the near future, while the 3D projectwill constitute the first exploration of a new, richer perceptual realm. The students and majorequipment associated with the two projects will remain separate.

PHS 398/2590 (Rev. 05/01} Page 8 Continuation Format Page

3 pages redacted--biosketches of other lab staff


RESOURCESFACILITIES: Specify the facilities to be used for the conduct of the proposed research. Indicate the performance sites and describe capacities,pertinent capabilities, relative proximity, and extent of availability to the project. Under "Other," identify support services such as machine shop,electronics shop, and specify the extent to which they will be available to the project. Use continuation pages if necessary.

Laboratory:

The PI has the sole use of a 750 sq ft laboratory equipped with 2 electrically shielded 8X10 rooms formonkey training and recording and a large general room for the use of experimenters.

Clinical:

Animal:

The Department of Comparative Medicine at Johns Hopkins administers animal facilities capable of housing25 monkeys and surgical facilities, both located within the Mind/Brain Institute. The sterile surgical suitecomprises 3 rooms (prep room, surgical theater, procedure room) and is maintained and equipped by theMind/Brain Institute.

Computer:

The laboratory is equipped with 4 highspeed Linux workstations for visual stimulus generation, datacollection and analysis and 16 Windows PCs for data analysis, figure generation and word processing.

Office:

The PI has a 170 sq ft office. Offices for students are provided by the Mind/Brain Institute.

Other:

The Mind/Brain Institute provides a well-equipped machine shop, electronics, shop, histology laboratory andelectrode manufacturing room. The machine and electronics shops both have staffs of 2, the histology andelectrode facility is staffed by one person.MAJOR EQUIPMENT: List the most important equipment items already available for this project, noting the location and pertinent capabilities of each.

1. 2 high-speed linux workstations for stimulus generation and data collection2. 8 high-speed Windows PCs for data analysis, figure generation and word processing3. ISCAN dual video eyetracker system for monitoring the positions of both eyes during stereoscopicstimulus presentation.4. Custom-built stereoscopic display system with 2 20" high-resolution CRT monitors laser-aligned in aframe for positioning the monkey's eyes so that a binocular display with disparity cues can be presented atthe appropriate vergence angle (adjusted for interpupillary distance) through cold mirrors that reflect themonitor images toward the eyes but transmit images of the eyes to the ISCAN video cameras.5. Neurophysiological recording system including hardware for accessing any part of inferotemporal cortexthrough a variable angle guide tube with stepper-motor microdrive and electronic filtering, windowing, andhigh-speed computerized data collection equipment.

PHS 398 (Rev. 05/01) Page 12 Resources Format Page


a. SPECIFIC AIMSLong-term goal: To understand how complex 3D shape is encoded by neurons in high-level

object regions of visual cortex. Our ability to live and interact in a world composed of 3D objects dependslargely on our spectacular capacity for visual shape perception. This is what makes vision so critical to ourhealth, happiness, and survival. Human shape perception ultimately depends on unknown neural codingmechanisms in high-level ventral pathway visual cortex. Deciphering this neural code is fundamental tounderstanding human visual appreciation of the world and complex pattern perception in general. We plan toaddress this issue by recording neural responses in higher stages of the macaque monkey ventral pathway,specifically the posterior, central, and anterior subdivisions of inferotemporal cortex (PIT, CIT, AIT).

Significance: The proposed experiments would be the first to investigate specific mechanismsof complex 3D shape representation in the ventral pathway. For many decades, neurophysiologicalstudies of complex object representation in monkey IT cortex have focused on 2D shape. Several recentreports indicate that 3D shape is also represented in IT, especially along its dorsal aspect, and in analogousregions of human cortex. These initial findings raise but do not address the basic question of how complex 3Dshape is encoded by neurons in visual cortex. Ours would be the first neurophysiological experiments toaddress fundamental questions about neural coding of complex 3D shape.

Methods: We will use complex, smooth (spline-based), abstract, randomly generated 3D stimuliguided by a genetic algorithm to explore 3D shape space efficiently. We will test hypotheses about 3Dshape coding mechanisms by constructing mathematical models to explain neural response patterns.The genetic algorithm will use neural responses as feedback to guide successive generations of randomstimuli toward the most relevant regions of 3D shape space. This is an efficient way to achieve densesampling in high-response regions of a stimulus space too large to explore with ordinary methods. It alsoavoids the biases introduced by an experimenter-defined stimulus set. Our system for generating smooth,abstract 3D objects permits both broad random variation and precise mathematical control over stimulusshape. Mathematically defined stimuli allow us to construct quantitative models to test hypotheses aboutshape coding mechanisms.

Specific Aims: We propose to answer a number of fundamental, never-before-addressedquestions about how the brain represents 3D objects. Because neurophysiological studies of complexshape representation have focused on 2D stimuli, the basic theoretical questions about 3D shape codingremain to be addressed empirically. In the following, competing hypotheses are stated as questions.

AIM 1: What kind of 3D shape information is encoded by individual IT neurons? In recentstudies of 2D shape coding in areas V4 and IT, we showed that the specific shape information signaled byindividual neurons can be quantified with tuning functions in multiple shape-related dimensions. Here, we willuse related analyses to test hypotheses about three aspects of 3D shape coding:

(a) Coding dimensions. What measurable 3D shape characteristics are signaled by IT responses?What parts, features or entities do they relate to—occlusion boundaries (i.e. external or internal object edges,which typically have 3D conformations), surface patches, medial axes, volumetric parts? What geometriccharacteristics of these entities are explicitly represented—absolute or relative position, orientation, curvature,curvature orientation, curvature derivative, thickness variation?

(b) Tuning function shape. Are IT neurons narrowly selective for specific 3D shape characteristics,producing a sparse, efficient, compact representation of 3D objects? Or are they broadly tuned, with widetolerance for shape and orientation changes, supporting the ability to generalize across different views of thesame object? Is tuning smooth and Gaussian, indicating basis function coding? Or do tuning functions haveabrupt boundaries indicative of categorical object coding?

(c) Integration mechanisms. Are 3D objects represented in terms of their parts, as many theoriesmaintain? If so, how do neurons at the highest levels in IT integrate information across multiple object parts?How many parts? Adjacent or disjoint? Is integration mainly linear, favoring implicit, highly distributed coding?Or is it mainly nonlinear, favoring explicit, sparse coding? Do some IT neurons, especially at higher stages inAIT, encode holistic shape, by integrating information nonlinearly across entire objects?

AIM 2: How is 3D shape encoded at the neural population level? The ultimate representation of a3D object is the activity pattern it evokes across large neural populations. Can the neural representation of anentire object be read out from this activity pattern as a high-dimensional vector averaged over individual neuraltuning (basis) functions? Or is a more complex, optimization decoding method required? Do any decodingmethods specify 3D shape completely enough to reconstruct the original stimulus?PHS 398/2590 (Rev. 05/01) Page 13 Continuation Format Page


b. BACKGROUND AND SIGNIFICANCE

Summary: The proposed study would be the first direct investigation of neural coding mechanisms forcomplex 3D shape. The specific neural coding mechanisms for shape can be studied most directly usingneurophysiology in non-human primates. Past non-human primate neurophysiology has focused on the 2Daspects of complex shape, but the world we live in and the objects we interact with are 3D. Understanding theneural basis of visual shape perception is a fundamental goal of vision science with broad implications forhuman welfare and the general study of higher-level brain function. Quantitative data on neural representationof 3D objects would provide a critical comparison with previous psychophysical, theoretical and computationaldata and models of natural object perception. Understanding the neural code for 3D object shape wouldprovide an essential basis for investigating higher-level cognitive processes that involve generation, storage,retrieval, and/or manipulation of real object information.

Understanding the neural basis of complex 3D shape perception would constitute a critical insight intohuman vision and higher-level brain function in general, providing a basis for comparison withprevious psychophysical, theoretical and computational results, and supporting future studies ofhigher-level cognition involving realistic object information. Shape perception is a fundamental aspect ofhuman vision. It is essential to most normal interactions with other individuals and with the world. Thus,diseases that impair shape perception (apperceptive and associative agnosias1) have a profound effect onquality of life. The human (and primate) capacity for shape perception is unmatched by the most advancedcomputer vision systems. Humans can effortlessly perceive any shape, in a way that generalizes across aninfinity of retinal image changes (e.g. different views of the same face) yet remains sensitive to the subtlestnuances (e.g. in facial expression). Thus, visual shape perception is a paradigmatic case of complexperceptual processing that can shed light on general higher-level brain mechanisms. Real objects are 3D, anda large body of psychophysical, theoretical and computational literature is devoted to 3D object perception2"36.The proposed studies would provide quantitative neural data to complement that literature. Deciphering theneural code for shape would also provide the essential basis for examining memory formation and retrieval,decision-making, and other high-level cognitive processes that depend on object information.

In humans and other primates, object shape is represented in the ventral cortical pathway.Neurophysiological and lesion studies in monkeys have shown that complex shape information is processed inhigher stages of the ventral pathway—V4, PIT/TEO, CIT/TE, AIT/TE37"*0. Imaging research in humans hasidentified homologous regions in human cortex—V4, V8, LOG, pFS, FFA, PPA—and elucidated how differenttypes of object information are distributed across these areas41^9.

Previous work on neural coding of complex shape in the ventral pathway has focused on 2Dperception. Neurophysiology in monkeys remains the most direct method for studying how shape informationis encoded. Past experiments on complex shape coding have focused on 2D images. (In some cases, real 3Dobjects have been used as stimuli, but not to investigate how 3D shape characteristics are encoded.) Theseexperiments have shown that individual neurons in V4 and IT represent information about object parts50"55. V4neurons mainly represent small, simple shape fragments, in terms of their basic geometrical characteristics(orientation, curvature) and position (absolute, relative)51156'57 (Pasupathy & Connor, J. Neurophysiol. 1999,2001, Nat. Neurosci. 2002, APPENDIX). IT neurons integrate information about multiple shape fragments insimilar geometric and position coding dimensions, using both linear and non-linear mechanisms54 (Brincat &Connor, Nat. Neurosci. 2004, APPENDIX). At the population level, the activity pattern in geometric andposition dimensions has multiple peaks corresponding to constituent object parts, and overall object shape canbe decoded from this parts-level representation (Pasupathy & Connor, Nat. Neurosci. 2002, APPENDIX; seePRELIMINARY STUDIES). Parts-based coding seems predominant, especially for abstract or unfamiliarobjects, but more holistic coding may emerge with increasing familiarity and/or behavioral relevance40' ^ 58r 59.

The proposed experiments would be the first to investigate specific neural coding mechanisms forcomplex 3D shape. Despite the fact that the retinal images are 2D, humans and other primates areexquisitely sensitive to depth information, especially in the context of local object surfaces60'61. At the neurallevel, simple local information about depth is encoded in V1 and V262^7. Until recently, it was thought thatPHS 398/2590 (Rev. 05/01) Page 14 Continuation Format Page


higher order depth processing occurred mainly in the dorsal visual pathway68"74. Recent evidence indicates arobust representation of complex 3D shape in the ventral pathway of the monkey75"80 and human 73'81t 82. Atthe neural level, V483'84 and IT85 neurons are sensitive to binocular disparity, V4 neurons are tuned for 3Dorientation (Hinkle & Connor, Nat. Neurosci. 2002, APPENDIX), and IT responses (especially in dorsalsubregions in the bank of the superior temporal sulcus) to flat 2D shapes are altered by cues that make thoseflat surfaces appear to curve outwards or inwards76"79. These results indicate that IT neurons encode 3Dshape, but they do not begin to address how 3D shape is encoded. A few studies have tested generalizationof IT responses across different views of a 3D object86"89, but such experiments do not target shape codingmechanisms. The extensive investigations of 2D shape coding in u5*"55 (Brincat & Connor, Nat. Neurosci.2004, APPENDIX) have no counterpart in the 3D domain. The proposed studies would constitute a firstapproach to this open question.

The proposed analyses will answer many fundamental, never-before-addressed questions about howthe brain represents complex 3D objects. What kind of specific 3D shape information is represented byindividual IT neurons? What 3D shape characteristics of object parts or whole objects are they tuned for? Theproposed analyses will measure the precise 3D shape information signaled by IT neural responses. Are ITneurons narrowly selective for specific 3D shape characteristics, producing a sparse, efficient, compactrepresentation of 3D objects? Or are they broadly tuned, with wide tolerance for shape and orientationchanges, supporting the ability to generalize across different views of the same object? The proposedanalyses will answer these questions with tuning function fits based on dense sampling along 3D shapedimensions. Are 3D objects represented in terms of their parts, as many theories maintain? Or is 3D shaperepresentation holistic, especially at the final stages in anterior temporal cortex? The proposed analyses willdetermine how parts-level information is integrated across an object, and whether that integration reaches thelevel of whole-object shape tuning. Can the neural representation of an entire object be read out from theactivity pattern across the neural population? Does this population activity pattern specify 3D shape socompletely that it can be used to reconstruct the original stimulus? The proposed analyses culminate withestimation of neural population activity patterns and stimulus reconstruction to answer these questions.

c. PRELIMINARY STUDIES

Summary: We have developed and tested the necessary experimental and analytical tools for broadsampling in complex 3D shape space and quantitative analysis of 3D shape coding mechanisms.Specifically, we have(i) Demonstrated that multi-dimensional Gaussian tuning analyses can explain neural coding for complex 2D

shape in IT, thus setting the stage for an analogous 3D study.(ii) Demonstrated Gaussian tuning for simple 3D shape characteristics (binocular disparity, 3D orientation) in

lower order ventral pathway area V4, which argues that complex 3D shape tuning in IT can also becharacterized with Gaussians or other smooth, mathematically definable functions.

(iii) Demonstrated that complex random spline-based shape stimuli are effective for sampling 2D shape spaceand deriving shape tuning functions, suggesting a similar strategy will be effective in 3D shape space.

(iv) Developed a spline-based graphics method for generating complex random 3D shape stimuli.(v) Developed and tested a genetic algorithm for optimal search of 3D shape space with random stimuli.(vi) Built and extensively tested a 3D stimulation system for displaying flat, shaded, textured, stereoscopic and

random dot stereogram (RDS) stimuli.(vii) Collected behavioral data from one monkey on an RDS task that verifies binocular fusion and

stereoscopic depth vision, which is critical for interpreting responses to stereoscopic stimuli.

(i) We have used multi-dimensional Gaussian tuning analyses to explain complex 2D shape coding inIT cortex at the individual cell and population levels. This provides a basis for applying similaranalyses to complex 3D shape coding in IT. We recently published a study showing how 2D shaperesponses of IT neurons reflect integration of object part information across multiple Gaussian tuning regions inposition * orientation * curvature space (Brincat & Connor, Nat. Neurosci. 2004, APPENDIX). Our currentanalyses are directed at understanding how these complex parts-level tuning functions are integrated intoneural population representations of entire objects (Brincat & Connor, Soc. Neurosci. Abstr. 2004, APPENDIX).PHS 398/2590 {Rev. 05/01) Page 15 Continuation Format Page


This analysis is related to our previously published study of population coding of 2D shape in area V4(Pasupathy & Connor, Nat. Neurosci. 2002, APPENDIX). However, the greater complexity of IT shape tuningrequires testing new hypotheses about population-level integration. Fig. 1 shows a population activity estimatebased on weighted tuning function summation in object-centered relative position x orientation x curvaturespace. This activity pattern provides accurate information about local contour orientation and curvature, butless accurate information about relative position of contour fragments. Therefore, we are currently testingother integration mechanisms (vector summation, optimization) and other types of relative position dimensions(including local part-part relationships) to determine which provide the most accurate whole-object shapeinformation and produce the most veridical reconstructions of the original stimuli. In the proposed study, wewill perform analogous tests of population coding hypotheses in 3D shape-related dimensions.

Figure 1. IT population activity plotted inrelative position x orientation x curvature space.This estimate of population activity evoked by a 2Dshape is a sum of neural tuning functions (based on76 IT neurons) weighted by corresponding neuralresponses to the shape (shown as a black outline inside plots; the actual stimulus was solid). Thetuning functions were derived by iterative fitting toneural responses across a large stimulus set(Brincat & Connor, Nat. Neurosci. 2004,APPENDIX). The center plot shows activity peaksin the orientation x curvature domain. For eachpeak in this domain, a corresponding slice throughthe x,y relative position domain is shown in one ofthe side plots. White circles indicate veridicalcontour fragment values, black squares areestimates based on the population activity.

90 ISO/ 270 I 360

Orientation

X Relative Pos

-1

(ii) We have demonstrated smooth, Gaussian-like tuning for simple 3D shape characteristics (binoculardisparity and 3D orientation) in area V4. This supports our proposed strategy of using smooth tuningfunctions to characterize complex 3D shape responses at higher levels in IT cortex. We have previouslyexplored the simplest level of 3D shape tuning—edge orientation—in area V4. We found smooth, Gaussian-like tuning for 3D orientation (Hinkle & Connor, Nat. Neurosci. 2002, APPENDIX; see Fig. 2) comparable topreviously described tuning for 2D edge orientation. We also reported the first evidence for stereoscopic depthposition tuning in area V4, buttressing the general case for 3D representation in the ventral pathway83. Thesefindings provide a basis for the proposed experiments addressing more complex 3D shape tuning at higherventral pathway stages in IT cortex. Specifically, the demonstration of Gaussian-like tuning for these simpler3D shape-related properties argues that similar smooth, mathematically definable tuning for more complex 3Dproperties exists at higher levels. Our recent results on 2D shape tuning in IT (Brincat & Connor, Nat.Neurosci. 2004, APPENDIX) also support this idea. Our proposed experiments and analyses are directed atcharacterizing smooth tuning for complex 3D shape characteristics.


Principal Investigator/Program Director (Last, First Middle): Connor, Charles Edward

Figure 2. Gaussian-like tuning for 3D orientation in area V4. In thisgraph, the 3D orientation domain is plotted as the surface of an invertedbowl (tilted slightly upwards to provide perspective). The averageresponses of an individual V4 neuron to bar stimuli at each orientation areindicated by the color scale, which ranges from blue (0 spikes/sec)through pink (-40 spikes/sec) to yellow (69 spikes/sec). This cell showsGaussian tuning for 3D orientation, with a peak at diagonal barorientations slanting away toward the lower right. 3D orientation tuningwas consistent across changes in absolute depth. Similar depth-invariant,3D orientation tuning was observed for approximately one-third of V4neurons (Hinkle & Connnor, Nat. Neurosci. 2002, APPENDIX).

(iii) We have developed and tested random spline-based stimuli as an unbiased approach tocharacterizing 2D shape tuning. The effectiveness of this method supports our proposal to userandom spline-based stimuli to investigate 3D shape tuning. Random stimuli are optimal for avoiding thebiases associated with arbitrary stimulus choices. They are also useful for avoiding possible long-termlearning effects associated with repeated exposure to a standard stimulus set. Most importantly, they providea way to explore an extremely large stimulus space, which is an especially daunting problem in 3D shaperesearch, where the range of possible stimuli is virtually infinite. Previous experiments at lower levels in thevisual system have employed random pixel patterns to successfully characterize receptive field and second-order kernel structure90'91. However, random pixel patterns cannot be effective for studying complex shaperepresentation at higher levels because these stimuli do not typically contain continuous contours that definerecognizable shapes.

As an alternative, we have developed random spline-based contour stimuli and tested their effectiveness forstudying 2D shape tuning in area V4 (Carlson & Connor, Soc. Neurosci. Abstr. 2004, APPENDIX). The stimuliare constructed by random positioning of control points to define a smooth Bezier-spline contour. Thesestimuli have proven to be effective for characterizing 2D shape tuning in area V4. Fig. 3 shows results for anindividual V4 neuron. The left panel displays the 25 stimuli evoking the strongest average responses (given inspikes/sec above each stimulus icon). Many of these stimuli contain sharp curvature pointed toward the left orlower left. Because these stimuli were presented as a rapid (10Hz) serial sequence, there is a good deal ofnoise in the responses, but this averages out over the total of 5000 stimuli used in the experiment. The rightpanel shows a tuning surface in orientation * curvature space based on this stimulus set and derived byiterative fitting of a Gaussian function to the responses. The fitted Gaussian indicates that the mostexplanatory shape characteristic common across high-response stimuli was curvature (weighted toward highor sharp curvature, but broadly tuned) oriented near 200° (i.e. with the peak pointing toward the left or lowerleft). This tuning function is similar to those we have previously derived using experimenter-defined stimulussets (APPENDIX), but the random stimulus approach confirms the generality of such tuning function fits.Similar results have been obtained for 70 area V4 neurons.


Principal Investigator/Program Director (Last, First, Middle): CoPPOr, CharieS Edward

1:6.20 2: 5.40 3: 5.60 4: 5.20 5: 5.40

0.8

<D 0.6

O 0.4

0.2

100 250 300 350150 200

Orientation (°)

Figure 3. Use of random spline-based stimuli tomeasure 2D shape tuning in area V4. The 25 stimulievokipg stropgest respopses are showp ip the left papel,with the average respopse ip spikes/sec above eachstimulus. (Respopses were relatively low because stimuliwere SPOWP ip a rapid upipterrupted sequepce at 10 Hz.) A

tuning function ip orieptation * curvature space based OP 5000 such respopses is shown ip the right papel.This peurop was tuped for a rapge of curvatures (weighted toward sharp curvature, represepted by the value1.0) oriepted pear 200° (poiptipg toward the left or lower left).

(iv) We have developed a spline-based graphics method for generating complex random object stimulisuitable for broad, unbiased sampling in 30 shape space. Ope of the greatest difficulties in studyingcomplex shape is the virtually infinite stimulus space to be explored. Apy experimenter-defined stimulus set isnecessarily limited and arbitrary. Random stimuli provide a means for broader, less biased exploration ofcomplex shape space. We have developed a method for randomly generatipg smooth, closed 3D objectsusipg pop-upiform ratiopal B-splipes (NURBS). Our aim was to achieve wide variatiop ip local surfacecopfiguratiops that could then be combiped to form a rich set of complex 3D objects. The graphical difficultylay ip achievipg smooth coptipuity betweep all the local surface patches required to copstruct a whole object.NURBS guarantee smoothpess apd coptipuity across a sipgle patch defiped OP a 2D domaip (typically labeledu,v), but a whole object must be stitched together from mapy patches arranged to be smoothly coptipuous attheir joipt edges.

Figure 4. Method for random generation of smooth 3D object stimuli. See main text for details.

The solutiop we adopted (shown in Fig. 4) is based OP a stacked series of circular control point arrayslying ip parallel plapes and closing to a point at either end of the stack. This set of control points defipes asmooth lozepge shape. The basic lozepge topology cap thep be distorted by varyipg the relative positions,orieptations, apd shapes of the copstituept plapar control point arrays. The position apd orieptatiop variatiopsPHS 398/2590 (Rev. 05/01) Page 18 Continuation Format Page


define the central axis of the final object. The shapes of the planar arrays are varied by adding randomly-defined 2D Gaussian distortions to control point radius along the circular arrays, producing local projectionsand indentations in the final object. An example is shown in Fig. 4A; each successive planar array of controlpoints is depicted as a uniquely colored polygon. The NURBS formulation is used to calculate a dense array ofvertices forming a smooth surface around those polygons, shown as a wire mesh in Fig. 4B. The final stimulusis rendered as a smooth surface shaded according to an OpenGL lighting model (Fig. 4C; in this case the lightsource comes from the viewer's direction). Disparity is added to the left and right eye images to producecorresponding stereoscopic depth variation (Figs. 4C and D for uncrossed free fusers; Figs. 4D and E forcrossed fusers).

As can be seen in Fig. 4, this method produces complex 3D objects with a variety of projections, indentations,saddle-shapes, ridges, channels, and flat surfaces (i.e., all 6 pair-wise combinations of positive, negative andflat principal curvatures). As in our previous studies of 2D shape coding, each stimulus simultaneously testsmany shape features, and the different combinations of features in different stimuli make it possible to analyzewhich features contribute consistently to responses and how multiple features interact. Because we definethese stimuli mathematically, we can completely parameterize their shape characteristics for analysispurposes, and we can test graded variations along any parameter, gradually morphing the objects to approacha neuron's tuning peak through successive approximation (see next section) or to examine any particular 3Dshape tuning characteristic in secondary post hoc tests (Fig. 5). Most importantly, we can generate a widevariety of stimuli that broadly sample 3D object shape space in a relatively unbiased fashion not tied to a singlespecific hypothesis. (However, we do not claim that our randomization method produces stimulus sets that arein any sense "white"—i.e., completely unbiased sampling of a finite space, like pixel noise—it is not even clearhow whiteness would be defined, since there is no generally accepted method for dimensionalizing 3D shapespace.)

PHS 398/2590 {Rev. 05/01) Page 19 Continuation Format Page


Figure 5. Parametric variation in 3D shape characteristics. Our stimulus generation method makes itpossible to systematically vary 3D shape characteristics in secondary tests (based on initial results withrandom stimuli). The rows in this figure exemplify different kinds of variation in overall shape (top 3 rows;length, thickness/surface curvature, medial axis curvature) and shape of a specific part (bottom 3 rows; length,thickness/surface curvature, medial axis curvature).

(v) We have developed and tested a genetic algorithm for guiding search through 3D shape spaceusing neural responses as feedback. The potential range of 3D shapes is virtually infinite, so a purelyrandom search would be highly inefficient for accurate measurement of tuning properties. The highdimensionality of 3D shape space also effectively prohibits an approach based on pre-defined stimulus sets (ofthe sort we have used to study 2D shape coding). Instead, we will use a genetic algorithm92"95 (Fig. 6) to guidesuccessive generations of quasi-random 3D stimuli toward high-response regions of shape space. After eachgeneration of 50 stimuli is tested, those stimuli evoking higher responses (e.g. Fig. 6A) undergo randommutation (i.e., random perturbation of the parameters defining their NURBS construction) of parts (Fig. 6B),and overall shape (Fig. 6C), and crossover with other high-response stimuli (e.g. Fig. 6D) to form differentcombinations of parts (Fig. 6E,F), to produce progeny that will be tested in the next generation. The probability



and number of progeny scale with normalized response to the progenitor, so that the highest response stimuliare represented by the greatest number of variants in subsequent generations.

A B C D E F

Figure 6. Random mutation in a genetic algorithm-based search through 3D shape space.(A) Hypothetical high-response stimulus. (B) Next-generation stimulus derived by parts-level mutation ofstimulus A. (C) Next-generation stimulus derived by whole-object mutation of stimulus A. (D) Anotherhypothetical previous-generation high-response stimulus. (E,F) Next-generation stimuli resulting fromcrossover of parts between stimuli A and D.

The goal of the genetic algorithm is not to find a single best stimulus, but to sample densely around theneuron's tuning peak or peaks in 3D shape space. Random mutation in each succeeding generation ensuresthat sampling remains distributed, not focused on a single point. All generations contribute to the analysis, butlater generations sample more densely near the tuning peak. To avoid entrapment of the search near localsub-maximal tuning peaks, every generation includes a percentage of completely random stimuli unrelated tothe previous generation. Entrapment is also avoided by starting with large mutation distances in earlygenerations to ensure wide exploration, followed by gradual decrease in mutation distances to sample finergradations in high response regions. At the same time, loss of promising stimuli due to mutation is avoided byconferring longevity (across generations) on stimuli in the highest response percentages.

Within generations, sampling efficiency is optimized by scaling the number of repetitions for each stimuluswith the cumulative average response rate to that stimulus. A stimulus evoking no response is tested onlyonce. A stimulus evoking a certain threshold fraction of the current maximum response is tested a secondtime. If its average response rate across the two repetitions exceeds a slightly higher threshold, it is testedagain, and so on up to 5 repetitions. This procedure is optimal because response variance scales withresponse rate (so that high response stimuli require more repetitions for equal accuracy) and because it favorshigher accuracy in the more relevant regions of shape space.

We have tested this genetic algorithm in extensive simulations to determine efficiency and estimate optimalparameters for mutation and crossover probability, mutation distance (degree of perturbation), spatial extent ofmutation (from single parts to whole objects), longevity (what fraction of high-response stimuli surviveunchanged across generations), and stimulus repetition (normalized response thresholds for repeatedpresentation within a generation). These parameters will be further adjusted based on results in earlyexperiments. Our simulations show that the genetic algorithm settles toward a hidden high-response region of3D shape space within 10-30 generations. This would require between approximately 1000-4000 totalstimulus presentations. Our previous experiments on 2D shape coding in IT (Brincat & Connor, Nat. Neurosci.2004, APPENDIX) using pre-defined stimulus sets involved 3000-10000 presentations per neuron, so theefficiency of the genetic approach compares favorably, especially considering the much higher dimensionalityof 3D shape space.

An example simulation is presented in Fig. 7. The hidden tuning principle in this case was based on surfacefragment positions. This kind of tuning corresponds to one specific theory of 3D shape coding: that objectsare represented in terms of their piece-wise, point-to-point positional proximity to learned shape exemplars96.In this simulation, there were two hidden excitatory regions in the z = 0 (fixation) plane at the top and right, andone inhibitory region at the left. Inputs from these regions were integrated by linear summation. Generation 0comprised 50 random shapes (Fig. 7A). The maximum response within this generation was 20.66 (in arbitrary



B

50

125i

20Generation

20

85•fe-

ll 10E

0 Generation 0

ffl Generation 18

J J • •-15 -5 5 15 25 35

Response45

Figure 7. Genetic algorithm-based search for 3D tuning properties. (A) Stimulus generation 0. (B)Stimulus generation 18. (C) Increase in maximum response rate with generation number. (D) Change indistribution of response rates between generations 0 and 18. (E) Reverse correlation result. See main text fordetails.



response units, Fig. 7C) and the distribution of response rates was weighted toward the low range (Fig. 7D,blue bars). Negative responses are due to the simple but unrealistic model for calculating responses as a sumof excitatory and inhibitory effects. In a real experiment, where negative spike rates are impossible, inhibitoryeffects are revealed by their suppressive interaction with excitatory effects (Brincat & Connor, Nat. Neurosci.2004, APPENDIX).

In subsequent generations, the maximum response increased in several abrupt steps (Fig. 7C) and plateauedaround 45 in generation 18. Generation 18 contained a number of high response stimuli with shapes meetingthe hidden specifications (Fig. 7B, top row), and the distribution of response rates was shifted toward a higherrange (Fig. 7D, red bars). Most of the generation 18 stimuli evoked responses below the maximum, but this isessential for sampling the entire tuning surface.

Sampling effectiveness was visualized with reverse correlation based on all stimuli from all generations (Fig.7E). Each stimulus sampled a large number of points in the analysis space (the x,y,z domain), and each ofthose points was weighted by the response to the stimulus. The analysts space was binned and the averageweight of sample points in each bin was calculated. A slice through this averaged volume centered on the zplane reveals the excitatory (red) and inhibitory (blue) subregions specified by the hidden tuning specifications.In a real experiment, this main test search would be followed up by secondary tests to systematically exploretuning characteristics in the region of the tuning peak (see RESEARCH DESIGN).

(vi) We have built and tested a 3D stimulation system for displaying flat, shaded, textured,stereoscopic and random dot stereogram (RDS) stimuli. In our previous 3D experiments, we used aNuVision LCD shutter system to present stereoscopic stimuli. This system suffered from bleed-through(crosstalk) between eye channels and low maximum luminance due to two layers of polarizing filters, making itunsuitable for presentation of realistic, complex 3D object stimuli in the proposed studies. Therefore, duringthe past year we have built and tested a new system comprising two monitors controlled by an FX3000graphics card in a dual processor workstation running real-time Linux to ensure precise and consistent controlover stimulus onsets and offsets. Luminance detectors are attached to each screen to verify framepresentation times during recording. Our tests show that less than 0.1% of frames are delayed whendisplaying the stereoscopic, shaded stimuli described above. Stimuli are rendered in OpenGL with anycombination of depth cues (binocular disparity, shading, texture gradients). In addition to sequential staticpresentation, these stimuli can be dragged across the screen under mouse control for hand-plotting receptivefields.

Images from the two monitors are reflected by circular mirrors mounted directly in front of the eyes (Fig. 8).These are "cold" mirrors that reflect visible light from the monitors but transmit infrared light so that eye positioncan be measured with infrared-sensitive ISCAN video cameras mounted in front of each eye. The mirrorsswivel, and the monitors are mounted on arc-shaped tracks, so that viewing geometry can be adjusted forinterpupillary distance such that vergence angle and accommodation correspond as in natural viewing. Theangular and positional alignment of monitors and mirrors is periodically checked with lasers, and stereoscopicperceptibility is verified by human observers. The monkey's headpost is mounted to the same frame as themirrors to ensure consistent alignment of the eyes.



Figure 8. 3D visual stimulation system. See main text for a description.

(vii) We have trained one monkey on an RDS task and collected behavioral data that verify binocularfusion and stereoscopic depth vision. This provides an essential control for interpreting neuralresponses to stereoscopic stimuli. The task requires the monkey to find and fixate a small square definedonly by depth offset (binocular disparity) within a large RDS display. The monkey was first trained to fixate aluminance defined square at randomly varying positions within the RDS, and then the luminance contrast wasgradually reduced to require use of the depth/disparity cues. Performance on this task exceeds 90% correct,verifying that the monkey is not stereo-blind and that the display is correctly aligned with the eyes to facilitatestereoscopic depth perception. This animal's skull is implanted with an acrylic cap, headpost, and recordingchamber (see RESEARCH DESIGN AND METHODS) so that recording experiments can begin immediately(Fall 2004).

d. RESEARCH DESIGN AND METHODS

In this section, each major part (experiments, single-cell analysis, population analysis) is headed by a shortsummary followed by a series of detailed methodological explanations.

Experiments (AIMS 1 and 2): Summary. We will study neurons throughout early to late stages of monkeyinferotemporal cortex—PIT, CIT, AIT—focusing on the dorsal subdivisions of these regions, which appear toemphasize 3D representation77'78. In our main test, we will study neurons with successive generations ofrandom, complex 3D objects, guided toward high-response regions of shape space by a genetic searchalgorithm. This main test will provide approximately 1000 data points, with sampling concentrated near theneuron's tuning peak(s). In secondary tests, we will use systematically varying stimuli to study in detail the



3D shape tuning characteristics indicated by the main test analyses. These secondary tests will measurewidth (selectivity vs. tolerance) and shape (smoothness vs. abrupt, categorical boundaries) of tuning forvariations in local surface characteristics, part shape, part position, object elongation, object thickness, and 3Dorientation.

Experiments (AIMS 1 & 2): Recording Locations and Methods. Recording methods will be similar to thoseused in our previous studies of 2D shape coding in IT cortex (Brincat & Connor, Nat. Neurosci. 2004,APPENDIX). The experimental animals will be rhesus macaque monkeys (M. mulatta). Access to water will berestricted (see VERTEBRATE ANIMALS). During training and recording sessions the animal will be seated in atwo-monitor computer-driven stereoscopic viewing system (see PRELIMINARY STUDIES), with the headimmobilized by means of a steel headpost attached to the skull with an acrylic cap anchored by orthopedicscrews. Animals will be trained to fixate a 0.1° white spot within 0.5° of visual angle for a period of 5 sec inorder to receive a juice reward. Eye position will be monitored using an ISCAN video eye-tracking system.

PIT/CIT/AIT neurons will be sampled mainly from the lower bank of the superior temporal sulcus because thisdorsal aspect of IT appears more sensitive to 3D shape information77'78, but to a lesser extent from the surfaceof the inferior temporal gyrus for comparison. Recording sites will range in the anterior-posterior direction fromapproximately -5 to +25 mm relative to stereotaxic 0. Anatomical analyses of functional differences betweenareas will be based primarily on regressions of functional measurements onto anterior-posterior recordingposition. An anatomical MRI scan will be used to visualize sulcus location and guide electrode penetrationsusing stereotaxic landmarks.

Neural activity will be recorded with 125-nm-diameter epoxy-coated tungsten electrodes (A-M Systems,Carlsborg, WA) with impedances of 1-5 MQ. Electrodes will be inserted transdurally by means of a customguide tube system with two degrees of freedom in angle of insertion so that any region of temporal cortex canbe targeted from a chamber mounted at the top of the head. Electrode position will be controlled with astepping motor microdrive (National Aperture, Salem, NH). Electrical waveforms will be amplified and filteredand single units will be discriminated on the basis of two independently adjustable time/amplitude windows(Bak Electronics). The digital output of the window discriminator will be collected through a NationalInstruments Card in a Windows PC using custom software.

After isolating the neuron electronically, we will use a hand-plotting program with colored 3D shape stimuliunder mouse control to establish an optimal stimulus color and an adequate stimulus shape, size and positionfor preliminary tests. These initial values will be used in automated tests to optimize stimulus color, positionand size for subsequent tests. Stimuli will be presented in one of 8 colors: red, green, blue, yellow, cyan,magenta, white and black. Each color will be adjusted to an approximate luminance of 20 cd/m2, except forblue (15 cd/m2) and black, and displayed against a background gray of 2.5 cd/m2.

AH stimuli in the main and secondary tests will be presented in random order at a rate of 1 stimulus per second(750 msec stimulus-on periods separated by 250 msec delays). Five stimuli will be presented during each 5second fixation trial. Stimuli will be centered at the receptive field center determined in the preliminary positiontest, and scaled based on the preliminary size test. Stimuli overlapping the fixation point will be positioned indepth so that the surface at that point lies at the fixation plane (in order to produce a more natural viewingsituation in which a point on the object surface is being fixated). Stimuli not overlapping the fixation point willbe positioned such that the median surface disparity value lies in the fixation plane.

Experiments (AIMS 1 & 2): Main Test—Exploration of 3D Shape Space with Random Stimuli Guided bya Genetic Search Algorithm. As described in detail under PRELIMINARY STUDIES, the main test willconsist of successive generations of 50 smooth, spline-based stimuli generated by randomly varying theposition, size, orientation, and local Gaussian perturbations of a series of stacked circular NURBS control pointarrays (Fig. 4). The highest response stimuli from each generation will be used to generate progeny in thenext generation based on random mutation of object parts and overall shape, as well as crossover betweenhigh response stimuli (Fig. 6). (Crossovers are necessarily limited to shape parts with compatible orientations.)The cumulatively highest response stimuli across the entire test will be represented in each generation, andPHS 398/2590 (Rev. 05/01) Page 25 Continuation Format Page


each generation will also include a percentage of completely novel random 3D shapes. The parameterscontrolling rate and extent of mutation, crossover, and longevity will be based initially on a priori simulationresults (PRELIMINARY STUDIES) but then adjusted based on results during early recordings, to produce thefastest progress toward a final response plateau. Simulations indicate that 10-30 generations will be required.Early recording, in which a larger number of generations will be presented, will be used to refine this estimateand determine an optimal method for determining when a final plateau has been achieved based on maximumresponse rates and response variation within generations and across recent generations. For all neurons, aseparate set of 50 random stimuli (consistent across cells) will be presented (5 repetitions) at the beginning ofthe main test for the purpose of estimating population responses to those 50 stimuli. (These stimuli will not beused as part of the genetic search process or for analysis of tuning properties.)

Experiments (AIMS 1 & 2): Secondary Tests of Specific 3D Shape Tuning Characteristics. Preliminaryanalyses of main test results (see below) will be used to identify 3D shape tuning characteristics for systematicexploration in secondary tests. These may include any of the hypothetical tuning dimensions discussed below,including length, width, thickness, and curvature of parts and whole objects (see Fig. 5 for examples). Therelevant dimension(s) will be tested by systematically varying NURBS control point positions. In addition, wewill use one or more optimal stimuli from the main test to measure sensitivity/tolerance to changes in 3Dorientation and stimulus size. Our initial tests of 3D orientation sensitivity will be at 45° increments around thecardinal x, y and z axes (Fig. 9), but we expect selectivity to be typically narrow, in which case we willsubsequently test smaller ranges with finer sampling.

PS!

Figure 9. Secondary test of 3D orientation sensitivity. A hypothetical stimulus drawn from the main test isrotated in 45° increments around the z (top row), y (middle) and x (bottom) axes.

As a control for effects due to the 2D luminance pattern of the shaded stimuli, we will present selected optimaland non-optimal stimuli illuminated from 9 different implicit directions (Fig. 10) to verify that tuning for 3D shapeis consistent. If early results show that lighting direction typically has major effects on responses, we willincorporate it as a variable in the main test. As a control for artifacts associated with stereoscopic presentation(e.g. changes in vergence angle), we will present optimal and non-optimal stimuli with shading cues alone, toverify that tuning for 3D shape is consistent. We will also collect continuous eye position data to control forvergence and other eye movement-related artifacts.



Figure 10. Secondary test of implicit lightingdirection. The main test stimuli (center) imply alight source behind the viewer. The side panelstimuli imply lighting from right, left, above, andbelow, at 45° and 90° angles with respect to theline of sight. To ensure that no part of the objectis invisible, a dim light source aligned with the lineof sight is included in all conditions to fill inshadows partially.

Finally, we will test for tolerance to changes in stereoscopic depth by presenting selected optimal and non-optimal stimuli across a 2° range of binocular disparities at 0.25° increments. For stimuli overlapping thefixation point, the nearest stereoscopic depth will be that at which the overlapping surface point lies in thefixation plane (so that the fixation point is never inside or behind the stimulus). Where possible, this test will berun on stimuli that do not overlap the fixation point, in which case the median surface disparity value will rangefrom +1.0 to -1.0°. Our studies of 3D orientation tuning in area V4 (Hinkle & Connor, Nat. Neurosci. 2002,APPENDIX) suggest that for many neurons 3D shape tuning will be robust to depth changes across this range.

Single-Cell Analysis (AIM 1): Summary. As in our previous work on 2D shape coding, we will use iterativecurve-fitting to derive models in multi-dimensional shape space that do the best job of predicting neuralresponses across a large dataset. In the proposed study, the dataset will comprise the main test (-1000random stimuli widely sampling 3D shape space) and secondary tests (~500 stimuli from systematic tests of3D shape dimensions based on the main test results) described above. Each complex random stimulus is atest of multiple features, and different stimuli test those types of features in various combinations. Thesecondary stimuli provide dense sampling along the shape dimensions most relevant for the particular neuron,and serve to measure tolerance for variation in basic dimensions like orientation. The pattern of neuralresponses to these ~1500 stimuli will provide a rich set of constraints that force models to discover whichspecific 3D shape characteristics have consistent effects on the cell and how those effects interact. We willtest models that vary in coding dimensions, tuning function shape, and integration mechanisms as describedbelow. Prediction accuracy of models will be measured using cross-validation (predicting responses tostimulus subsets not used in the fitting procedure) to ensure against overfitting. We will use regression to lookfor tuning trends across anatomical location. We expect to see increases in tuning complexity and possibly ashift from parts-based toward holistic coding in the posterior to anterior direction (PIT to CIT to AIT). Weexpect to see increased sensitivity to 3D (as opposed to 2D) shape in the dorsal parts of IT (lower bank ofsuperior temporal sulcus, as opposed to inferior temporal gyrus).

Single-Cell Analysis (AIM 1): Iterative Curve-Fitting. As in our previous analyses of 2D shape tuning(APPENDIX), we will use the Gauss-Newton algorithm in MATLAB (MathWorks, Natick, MA) to adjust theparameters of models based on multiple tuning function components in multi-dimensional shape domains so



as to minimize the sum of squared errors betweep observed and predicted responses to large sets of stimuli.Observed neural response rates will be calculated by counting spike occurrences within the stimuluspresentation period. Background response rate will be derived in the same way from null stimulus periodsinterspersed randomly among stimulus presentations in all tests and subtracted from observed rate for mostanalyses. Our previous results in IT indicate that background rates are extremely low for almost all neurons.

In previous analyses, we have used multiple starting positions for the iterative search for each tuning functioncomponent in order find the global maximum solution. To speed analyses in the present study, we will attemptto use reverse correlation (e.g. Fig. 7E) as a basis for inferring optimal starting positions (e.g., by finding localmaxima in the reverse correlation volume to use as starting points for Gaussian tuning subunit positions). Theaccuracy of this approach will be verified in early analyses by comparison with the multiple starting pointapproach. If the reverse correlation approach proves inadequate, we will rely on multiple starting points as inour previous analyses.

Correlation between predicted and observed responses will be the basic measure of model performance.Partial F-tests will be used to measure significance of individual tuning function parameters and significance ofdifferences between models. The Bayesian Information Criterion will be used to compare the relativeprobabilities of competing models, taking account of model complexity and weighing that against the amount ofexplained variance. Generality of models will be tested by cross-validation (predicting responses to a stimulussubset not used in the fitting procedure).

Single-Cell Analysis (AIM 1): Coding Dimensions (AIM 1a). Using the iterative model-fitting proceduresdescribed above, we will test a wide variety of hypothetical 3D shape tuning dimensions—i.e. measurable 3Dshape characteristics that might systematically influence neural responses. We will test dimensions related todifferent parts or features of objects: 2D occlusion boundaries (external and internal edges, which typicallyhave a 3D shape), 3D surface patches, medial axes, and volumetric parts. For each of these, we will test arange of metric characteristics—absolute and relative position, orientation, curvature (in multiple directions forsurfaces), curvature orientation, curvature derivative, thickness variation (for volumetric parts). We will test allof these tuning dimensions individually and in every possible combination, searching for the metric 3D shapedomain that yields the best prediction of observed responses, in an attempt to approximate the implicit codingdimensions of IT cortex itself.

Parts-level analyses will rely on dense sampling of stimulus values along boundaries, surfaces, medial axesand volumes of 3D object stimuli to provide a set of parts-related points in the analysis space that representthe object. Predicted responses will be based on those points that fall closest to the tuning functioncomponents in the current iteration of the model. In previous analyses, we have used higher-level derivativesto segment 2D objects into boundary parts that were then each represented by a point in the shape domain.The dense sampling approach will provide a more accurate and complete representation of the object, and willavoid the difficult issues involved in segmentation of 3D objects into parts. Dense sampling will slowcomputational speed, but we hope to make that up by reducing the necessary number of starting positions foriterative searches, as described above.

Hypothetical position dimensions for object parts will include absolute (retinotopic) position, position relativeto object center of mass, position relative to object geometric center, and (for model terms that represent non-linear interactions between multiple tuning regions) relative positions of two or more object parts.

Orientation for 2D occlusion boundary fragments will be the direction of a boundary normal vector pointingaway from the object interior. Surface orientation will similarly be based on a surface normal pointing awayfrom the interior. It will involve two dimensions—the direction of the projection of the normal onto the x,y plane,and the angular deviation away from the x,y plane. For model terms that represent non-linear interactionsbetween multiple tuning components, relative orientation will also be considered as a dimension.

Curvature (rate of change in orientation) will be mapped from the original -coto oorange to a scale of-1.0 to1.0 using a squashing function (for details see Pasupathy & Connor. J. Neurophysiol. 2001, APPENDIX).PHS 398/2590 (Rev. 05/01) Page 28 Continuation Format Page

Principal Investigator/Program Director (Last, First, Middle}! Connor, Charles Edward

Curvature of 2D occlusion boundaries and medial axis components will be one-dimensional. Curvature ofsurface patches will be measured with two dimensions, one corresponding to the maximum cross-sectionalcurvature, the other to the orthogonal cross-sectional curvature. These curvatures will +/+ for a protrusion, -/-for a depression, +/- for a saddle-shape, +/0 for a ridge, -/O for a channel, and 0/0 for flat. The orientation ofthe maximum curvature will also be a dimension in these analyses (to distinguish, for example, ridges orientedin different directions). The derivative of curvature will also be tested (a non-zero curvature derivativecharacterizes horn-shapes and other spiral elements).

Volumetric parts will be characterized by local size in 3 orthogonal directions—length, height, and thickness—relative to cardinal axes or to the local medial axis (in which case the orientation and curvature of the local axiswill also be analysis dimensions). Rate of change (1st derivative) of these measures will also be considered asanalysis dimensions, to distinguish, for example a cylinder (zero 1st derivative of width) from a cone (non-zero1st derivative of width). The 2nd derivative of these volumetric size values will also be considered, todistinguish, for example, a cone (zero 2nd derivative of width) from a bullet-shape (non-zero 2nd derivative ofwidth).

These dimensions do not exhaust the possible range of 3D shape descriptors, but they represent the simplest,most easily derived geometrical measures most commonly discussed in the theoretical literature, and theyrelate to 2D tuning dimensions with high predictive value in our previous analyses of 2D shape coding in V4and IT (APPENDIX). Tuning for more complex shape characteristics would be apparent as specific patternswithin these simpler dimensions, which could then be modeled by integration across multiple tuning regions(see below) or by constructing more complex dimensions suggested by the initial analyses. Tuning for holisticshape would likewise be represented by integration across multiple tuning regions in these parts-leveldimensions.

Single-Cell Analysis (AIM 1): Tuning Functions (AIM 1b). The nature of the best-fitting tuning functions in3D shape dimensions will reveal fundamental characteristics of 3D shape coding mechanisms. Smoothfunctions (e.g. Gaussians) suggest a basis function coding scheme97'98, where the variable response rates ofindividual neurons represent a range of shape values along some dimension or dimensions, and exact valuesfor the current stimulus are represented as hills of activity in the population of units (tuned for different regionsalong the shape dimensions). Tuning functions with abrupt discontinuities would represent categorical tuning99'10°, which could be prominent for an extremely complex domain like 3D shape, and might be more typical athigher stages in anterior regions of IT. Broad tuning functions (e.g. Gaussians with large standard deviations)would reflect a highly distributed coding scheme with broad tolerance for shape and orientation changes, whichcould support generalization of object recognition across different views. In contrast, narrow tuning functionswould reflect a sparse, efficient, compact representation of highly specific shapes101.

As in our previous analyses of 2D shape (APPENDIX), multi-dimensional Gaussian functions (products of one-dimensional Gaussians in the individual dimensions) will be used as the basic subunits in our initial models.Gaussian tuning seems to be a ubiquitous feature of neural representation, our 2D coding studies have shownthat highly predictive models can be constructed from multiple Gaussian subunits, and we expect similarresults for 3D shape. Because tuning for 3D shape, especially at more anterior stages in IT cortex, may bemore categorical, with more abrupt boundaries, we will also use if necessary combinations of sinusoidalfunctions (for each dimension, 2 sinusoids with a common maximum amplitude where they join), which can beadjusted to have sharp roll-offs. If our initial reverse correlation analyses suggest other more complex tuningfunction shapes, we will try fitting responses with more flexible polynomial functions and constructing higher-dimensional domains in which tuning function shape might become simpler.

Single-Cell Analysis (AIM 1): Integration Mechanisms (AIM 1c). The ways in which shape information isintegrated across object parts in the best-fitting models will test fundamental theoretical ideas about 3D shapecoding. Many theories posit that objects are represented, even at the highest levels, in terms of theirconstituent parts, by ensembles of neurons tuned for parts-level information7'11'12'14'15'21-30. In our models ofneural tuning, this would be reflected by a limit to the degree of spatial integration across object parts withinindividual neurons, even at the highest stages in CIT/AIT. In contrast, some theories argue that shape isPHS 398/2590 (Rev. 05/01) Page 29 Continuation Format Page


ultimately represented in a holistic fashion, with each neuron carrying information about overall object shape96'102. In the neural tuning models, this would be reflected by integration of shape information across all objectregions, probably at the highest stages in CIT/AIT. True holistic representation would require non-linearintegration—e.g. best-fitting models in which the major terms are products of excitatory terms corresponding todifferent object regions54. In such cases, where integration was almost all non-linear, it would be possible torepresent multiple object regions with multiple sets of dimensions, creating a higher-dimensional space wheretuning for a specific range of overall shapes could be represented by a single Gaussian.

Between the single part and whole object tuning extremes, there is a wide range of possible integrationmodels, involving combinations of linear and nonlinear mechanisms. Our early results in studying 2D shapecoding suggest such a continuum (Brincat & Connor, Nat. Neurosci. 2004, APPENDIX). The order of the best-fitting non-linearities corresponds to the complexity and specificity of part-shapes explicitly represented byindividual neurons. For example, a highly significant fourth-order term (e.g. a product of excitatory inputs fromfour different regions of the shape domain) would indicate selectivity for a very specific part or combination ofparts. Mainly linear tuning is compatible with highly distributed, implicit coding mechanisms that could exhibitwide generalization across shape changes. Mainly nonlinear tuning suggests sparse, compact, efficient,explicit coding mechanisms that represent objects through the activity of a few highly selective neurons.

Integration mechanisms can also vary in mode of excitatory/inhibitory interaction. Inhibition can be linear(subtractive) or non-linear (divisive). Non-linear inhibition could apply to the overall response or to specificexcitatory terms. A fundamental coding question is the extent to which inhibitory effects are general (broadtuning subunits that provide only a general bias to the overall response) vs. specific (narrow tuning subunitsthat provide specific shape information for sculpting out selectivity).

We will explore all these possibilities by testing models with different numbers of tuning subunits, differentlinear and non-linear excitatory combinations of different orders, different types of non-linearities (productterms, exponential terms, threshold rectification) and different kinds of excitatory/inhibitory interactions(subtractive, divisive) between different combinations of model terms.

Population Analysis (AIM 2): Summary. The ultimate representation of a 3D object is the activity pattern itevokes across large neural populations. Population coding analyses provide the ultimate confirmation that anygiven set of coding dimensions, tuning functions, and integration mechanisms (at the individual cell level)provide sufficient information (at the population level) to represent an entire object. Testing different populationcoding mechanisms addresses basic questions about how information distributed across individual cells couldbe integrated into a coherent, explicit signal for a complex 3D object103. In our previous work on 2D shape inarea V4 and IT, we have shown that it is possible to estimate the population representation of an object andreconstruct the original stimulus from this estimate (PRELIMINARY RESULTS; Pasupathy & Connor, Nat.Neurosci. 2002, APPENDIX). In the proposed project, we will perform similar analyses of 3D shaperepresentation at the population level, based on a separate set of 50 random 3D shape stimuli presented toevery neuron at the beginning of the main test. These 50 stimuli will not be used to fit tuning function modelsto the individual cells, making it possible to demonstrate that a given population coding scheme generalizes toshapes not used in defining the population code.

Population Analysis (AIM 2): Population Coding Mechanisms. In most of our previous analyses of 2Dshape population coding (PRELIMINARY STUDIES; APPENDIX), we have hypothesized a basis functioncoding scheme, in which exact numerical values (e.g. for curvature) are represented by a broad peak of activityacross a population of neurons with broad tuning functions that together span the stimulus space. The activitypeak can be estimated by vector averaging of weighted tuning peaks, averaging of weighted tuning functions,or fitting a smooth (e.g. Gaussian) function to the population activity distribution, all of which give similar resultsunder most circumstances97, or by a winner-take-all calculation (which can yield a very different answer underspecific circumstances). Our analyses have required extrapolation from previous single-peak basis functionanalyses (e.g. in motor cortex104) in that our population activity profiles contain multiple local peaks representeddifferent components of a complex stimulus. To find the multiple peaks, after vector or function averaging, weconvolve the population profile with a spatially restricted filter that identifies local maxima. We will apply similarPHS 398/2590 (Rev. 05/01) Page 30 Continuation Format Page

Principal Investigator/Program Director (Last, First, Middle): ConnOP, Charles Edward

basis function analysis methods to our 3D data. These analyses will employ different types of tuning functionanalyses (see above) for individual cells, so that those individual tuning analyses can be tested at thepopulation level. We will compare results based on averaging (of tuning peaks or functions) with results basedon (local) winner-take-all analysis, since these represent two very different population representation schemes.They differ, however, only under circumstances where two activity peaks imply conflicting information (e.g. twodifferent curvatures at the same position). This circumstance has not occurred in our 2D data, but it mightoccur in the more complex 3D shape domain.

The general alternative to basis function coding is optimization methods (e.g. maximum likelihood estimation)that use mathematical techniques to find the best or most probable solution based on the available data103.Basis function coding has a clear and simple neural implementation, but may not always yield the best resultderivable from the data. Optimization can yield better results, but may not have obvious neuralimplementations103, although a simple neural implementation of maximum likelihood estimation has beendeveloped98. We will test optimization decoding methods to determine whether a significantly more accurateresult can be derived from our population data using methods other than basis function analysis. Standardoptimization analyses designed for simple sensory coding situations are not straightforwardly applicable to ourhigh-dimensional, multi-peaked population responses. Instead, we will use iterative function-fitting in thepopulation response domain, based on error terms that weight various single-cell tuning function componentsin different ways, in an attempt to derive more optimal decoding solutions.

Population Analysis (AIM 2): Reconstruction of 3D Shape Stimuli. We will attempt to use populationdecoding results (described above) to reconstruct the original stimuli. This is the ultimate test of shape codingmechanisms, and the shape differences between original and reconstructed stimuli can be quantified (e.g. bysummed point-to-point Euclidean distance) to assess the validity of those mechanisms. In our previousanalyses of 2D shape population coding in area V4, we used an iterative algorithm for adjusting a spline-basedreconstruction to match the decoded population information as closely as possible (Pasupathy & Connor, Nat.Neurosci. 2002, APPENDIX). A similar procedure based on NURBS will be used here to reconstruct 3D stimulifrom population activity analyses.

Projected Results. Schedule and Alternative Strategies: As in our previous studies of 2D shape coding(APPENDIX), the strategy outlined above for investigating 3D shape coding is predicated on collecting themaximum amount of information from each individual neuron. Thus, we typically study only one cell per day,using most of the available time and behavioral trials to collect on the order 1000-4000 data points. (Ourexperience in studying IT neurons with the custom guide tube system described above shows that we canconsistently hold well-isolated neurons for the required period of time.) Because of the high dimensionality ofshape, especially 3D shape, this is the only way to adequately characterize shape tuning in a reliable,quantitative fashion.

Due to this time-consuming but necessary 1 cell per day approach, we project that several years ofexperiments will be required to collect a complete sample from the full posterior/anterior extent of IT.Moreover, because such a large range of hypothetical coding mechanisms must be tested, and becauseiterative fitting of high-dimensional models is computationally intensive, we project that several years ofanalysis will be required to completely characterize the resulting data. We envision a 4-year schedule ofrecordings and analysis, in two stages.

The first stage of recording and analysis will focus on more posterior regions in PIT and CIT, where we expecttuning characteristics to be more metric and amenable to quantitative characterization. This first investigationwill establish an analytical basis for approaching the potentially more difficult anterior regions. Recording fromthe 1st monkey will begin in Fall 2004 and should continue through Winter 2004-2005. Results from the 1st

monkey will be analyzed as much as possible during Spring 2005 to assess whether any major experimentalmodifications are necessary before recording from a second animal. The 2nd monkey will be prepared andtrained during early 2005 so that recording can begin in Summer 2005 and continue through the end of 2005.Based on our experience with analysis of 2D shape coding using similarly large datasets, we expect thatcomplete analysis of these PIT/CIT results will require most of the following year (2006), during which the



dataset will be supplemented by recording from a 3rd monkey if necessary. We expect to be writing severalmanuscripts on 3D shape coding at the individual neuron and population levels in PIT/CIT by early 2007. WeThese reports will describe models of metric 3D shape tuning with high predictive value for a wide range ofcomplex 3D stimuli. The specific characteristics of these models—tuning dimensions, tuning functioncharacteristics, integration mechanisms, population decoding algorithms—will establish many of thefundamental coding principles for 3D shape in high-level visual cortex and help distinguish between competingtheories of 3D shape representation.

The second stage of recording and analysis will focus on more anterior regions in CIT and AIT, relying on thepreceding results to help refine the experimental approaches and focus the analytical efforts. We expect totrain and record from 2-3 monkeys during 2007 and early 2008. Analyses of this dataset should also requireon the order of a year, during 2008-2009, leading to several publications in late 2009. We expect these reportsto show how 3D shape representation progresses at higher stages in IT, possibly in the directions of greaterintegration across object parts, greater complexity of tuning functions and/or dimensions, and greaterspecificity of tuning, reflecting a more sparse, explicit coding scheme.

Because complex 3D shape representation has not yet been investigated at the neural coding level, thestrategies outlined above are necessarily based on analogies to our previous 2D shape coding studies and apriori theoretical ideas about how 3D shape might be encoded. Results from initial experiments may indicatethat alternative strategies are required. It may be that 3D shape responses, especially at later stages inCIT/AIT, are too weak or variable to characterize quantitatively. If so, we will try every stimulus manipulationpossible to increase response strength (including greater variation of more dimensions in the random stimulussets, especially color gradients, texture, motion) and/or to decrease response variability (especially byincreasing presentation times to ensure that variability is not due to inadequate time for depth percepts toconsolidate). If no such stimulus manipulations produce adequately robust responses, that could indicate thatattention is required due to the complexity and perceptual subtlety of 3D shape stimuli. In that case, we willtrain monkeys to attend to the 3D shape stimuli in the context of a standard delayed match to sample task.This would be a last resort, since an attentional task would seriously retard collection of the large number ofdata points required for quantitative analysis of shape tuning functions. Luckily, our previous 2D shape resultsand other studies of complex 2D shape representation (as well as many psychological studies of implicitrecognition) argue that attention is not required for complex shape processing in the absence of competingstimuii.

It might turn out instead that 3D shape responses are robust and consistent, but not metric and quantifiablewith smooth tuning functions in geometric dimensions. Again, this might be especially true at later stages inCIT/AIT. This would suggest that neurons are encoding non-geometric information, probably related tolearning and memory, as has been demonstrated for neurons near the anterior pole of the temporal lobe105. Inthis case, we would train monkeys on a standard shape categorization task with large numbers of stimuligrouped without regard to geometric similarity, in order to show how responses to geometrically disparatestimuli may be explainable in terms of learned categorical relationships, and to characterize the posterior toanterior transition from metric coding to categorical coding. We think this eventuality is unlikely except atextremely anterior or medial recording locations, since the available evidence indicates that most IT responsesare related to metric shape, not to learned object categories99-10°.

f. VERTEBRATE ANIMALS

1) Description. Four to eight rhesus macaque monkeys (M. mulatta) will be used in this project. The animalsmay be male or female, and will range in age from 1-5 yrs. and in weight from 3-12 kg. Procedures includebehavioral training, surgery, and single unit recording from cerebral cortex.• Training. The animal will be transferred each day from its cage to a standard design primate chair

using the widely accepted pole and collar method. The animal will be rewarded for correct fixationbehavior with water or juice, and will be allowed to obtain as much liquid as desired in this manner. If



the total amount obtained on a given day is less than its average daily intake during training the animalwill be supplemented up to its average amount. Experience has shown that animals are typicallysatiated and maintained in good health by a daily water intake on the order of 100 ml plus 10-15 ml foreach kg of body weight over 4 kg. The safest course, however, is to gauge the appropriate amount foreach animal individually by carefully monitoring the amount of water earned during the early (relativelyeasy) stages of training. The animal will be monitored for signs of dehydration such as weight loss,constipation, lethargy and loss of appetite. Water will be immediately provided ad lib in the cage if theanimal shows signs of dehydration or suffers any other serious health problem. The standard diet ofmonkey chow will be supplemented with fresh and dried fruit. The animal will be given water ad lib fora period of at least 24 hours every 3-4 weeks. Free access to water will be provided weekly wheneverpossible, but during critical periods in the training process weekly interruptions in the behavioralregimen can seriously compromise the animal's performance (many animals require 3-4 days to re-achieve normal performance levels after ad lib water) and thus prolong the total period of waterdeprivation necessary to complete training. Weekly interruptions during recording would greatlydecrease the data yield from each animal and thus increase the number of animals required for thestudy. Controlling water intake in this way is the accepted and only reasonable method for motivatingbehavior in single cell recording experiments. Years of experience with this method in numerouslaboratories has shown that no ill effects result.

• Surgery. One surgery will be performed prior to training (further minor surgeries will be performedduring recording periods as described below). The surgery will involve implantation of a head restraintdevice (a steel or titanium post) and steel or titanium recording chamber anchored to the skull withacrylic dental cement and small orthopedic bone screws. Head restraint is necessary for stableinsertion of electrodes into the brain to record single cell activity. This type of surgery is a standardprocedure used by most labs performing this kind of research. It will be performed in the surgical suiteon the second floor of Krieger Hall at the Johns Hopkins University Homewood Campus. The animalwill be initially sedated with ketamine (10 mg/kg IM) and then anesthetized with sodium pentobarbital(25 mg/kg IV). Anesthesia will be maintained with further IV injection of sodium pentobarbital asneeded. After surgery a prophylactic dose of Bicillin (300,000 IU IM) and an opiate analgesic(buprenex, 0.01 mg/kg IM) will be administered. The animal will be allowed to recover from anesthesiaunder continuous observation and then returned to its cage and allowed to recover from surgery for aperiod of not less than 7 days.

• Recording. Each day of the experiment the animal will be transferred to the primate chair and broughtto the lab as before. The head will be fixed, and the cap will be removed from the recording chamber.A microdrive and guide tube apparatus will be attached to the recording chamber and used to drive oneor more thin (80-125 micron diameter) epoxy or glass-coated wire electrodes through the dura via asmall craniotomy previously drilled through the skull under ketamine anesthesia. The craniotomy is aminor procedure that is routinely done in humans under local anesthesia. It is not considered painfulbecause the skin has been retracted and the periosteum removed from the skull during the surgery toimplant the head restraint device. The craniotomy will be cleaned each day with saline and Nolvasan(0.05% chlorhexidine gluconate, a mild disinfectant). The electrodes will be used to record theelectrical activity of single cells while the animal performs its standard behavioral task. At the end ofthe 3-5 hour recording session the electrodes will be withdrawn, the craniotomy cleaned and resealed,and the animal returned to its cage. Experiments will continue in this manner for a period of 2-4 monthsper hemisphere. Water intake will be controlled as described above. Ad lib water will be given for 24hours every 3-4 weeks during training and recording. In between studies the animal will be given waterad lib for a period of weeks or months (depending on the overall experimental schedule of the lab) andpossibly retrained to perform a variant behavioral task. All the procedures described here are standardtechniques for single cell recording.

2) Justification. The brain mechanisms underlying visual processing can be fully studied only by recording theresponses of individual cells. Human psychological studies can reveal what kind of visual processingoccurs, and functional brain imaging studies (using PET and MRI) can reveal roughly where it occurs, butonly single cell experiments can show how it is accomplished. Single cell experiments are necessarilyinvasive and therefore usually inappropriate for human studies, so an animal model is required. The


Principal Investigator/Program Director {Last, First, Middle): Connor, Charles Edward

macaque monkey is the only widely available research animal with a visual system comparable to thehuman visual system. The cat has been a useful species for studying lower level visual processing, buthigher level functions are studied almost exclusively in monkeys. Four to eight rhesus monkeys (Macacamulatta) will be used in this project. It is standard for publications in this filed to be based on data from atleast 2-3 animals, and this study is expected to lead to multiple publications.

3) Veterinary care. The animals are cared for by veterinarians in the Department of Comparative Medicine atJohns Hopkins University Medical School. They are always monitored on a daily basis by theexperimenters.

4) Discomfort and pain. The only potential pain involved in the procedures described above is post-surgicalpain. This is controlled with an opiate analgesic (buprenex). Insertion of electrodes into the brain is notpainful (there is no sensitivity to tissue damage in the brain itself)- There is potential discomfort and mentaldistress associated with transfer between the cage and the chair, head restraint, and water restriction.Each of these elements is essential to the performance of this type of experiment. Fear and discomfortinvolved in transfer between cage and chair is minimized by use of the pole and collar technique, in whichthe monkey is allowed to climb into the chair or cage on its own (though restrained from escaping by ametal pole attached to a collar around the monkey's neck). Head restraint is limited to a maximum of 6hours in a day, and care is taken to ensure that the monkey is in a comfortable position. Water restrictionis ameliorated by the ability of the animal to obtain as much water as it is willing to work for, by carefulmonitoring of the monkey's physical state to prevent dehydration, by dietary supplementation with freshand dried fruit to ensure adequate nutrition, and by regular intervals of free access to water. Any seriousdiscomfort or distress inevitably leads to a marked drop in behavioral performance; when this occurstraining or recording is discontinued and the animal is given water ad lib.

5) Euthanasia. Animals will be euthanized by IV injection of a large dose of sodium pentobarbital (90 mg/kg).This method is selected because it involves no pain or distress to the animal, and it is consistent with therecommendations of the Panel on Euthanasia of the American Veterinary Medical Association.

g. LITERATURE CITED

1. Farah.M.J. Visual Agnosia (MIT Press, Cambridge, Massachusetts, 1990) .

2. Albert.M.K. & Tse.P.U. The role of surface attraction in perceiving volumetric shape. Perception

29,409-420 (2000) .

3. AlbrightJ.D. Why do things look as they do? Trends Neurosci. 17, 175-177 (1994) .

4. Bennett.B.M., Hoffman.D.D., Nicola.J.E., & Prakash.C. Structure from two orthographic views of

rigid motion. J. Opt Soc. Am. fAJ6,1052-1069 (1989) .

5. Berbaum.K., Bever.T., & Chung,C.S. Extending the perception of shape from known to

unknown shading. Perception 13,479-488 (1984) .

6. Bergstrom.S.S., Gustafsson.K.A., & Jakobsson.T. Distinctness of perceived three-dimensional

form induced by modulated illumination: effects of certain display and modulation conditions.

Percept. Psychophys. 53, 648-657 (1993) .

7. Biederman.l. Recognition-by-components: a theory of human image understanding. Psycho/.

Rev. 94, 115-147 (1987) .

8. Biederman.l. & Ju.G. Surface versus edge-based determinants of visual recognition. Cognit.

Psycho/. 20, 38-64 (1988) .



9. Blostein.D. & Ahuja.N. Shape from Texture: Integrating Texture-Element Extraction and Surface

Estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence 11, 1233-1251

(1989) .

10. Braunstein.M.L, Hoffman.D.D., Shapiro.LR., Andersen.G.J., & Bennett.B.M. minimum points

and views fMor the recovery of three-dimensional structure. J. Exp. Psycho/. Hum. Percept.

Perform. 13, 335-343 (1987) .

11. Bulthoff.H.H., Edelman.S.Y., & Tarr.M.J. How are three-dimensional objects represented in the

brain? Cereb. Cortex 5, 247-260 (1995) .

12. Bulthoff.H.H. & Edelman.S. Psychophysical support for a two-dimensional view interpolation

theory of object recognition. Proc. Natl. Acad. Sci. U. S. A 89, 60-64 (1992) .

13. Christou.C.G. & Koenderink,J.J. Light source dependence in shape from shading. Vision Res.

37, 1441-1449 (1997) .

14. Dickinson.S.J., Pentland.A.P., & Rosenfeld.A. 3-D Shape Recovery Using Distributed Aspect

Matching. IEEE Transactions on Pattern Analysis and Machine Intelligence 14, 174-198 (1992) .

15. Dickinson.S.J., Pentland.A.P., & Rosenfeld.A. From Volumes to Views: An Approach to 3-D

Object Recognition. Image Understanding 55, 130-154 (1992) .

16. Edelman.S. Computational theories of object recognition. Trends in Cog. Sci. 1, 296-303

(1997) .

17. Edelman.S. & Bulthoff.H.H. Orientation dependence in the recognition of familiar and novel

views of three-dimensional objects. Vision Res. 32, 2385-2400 (1992) .

18. Graham.M. & Rogers.B. Simultaneous and successive contrast effects in the perception of

depth from motion-parallax and stereoscopic information. Perception 11, 247-262 (1982) .

19. Grossberg.S. & Swaminathan.G. A laminar cortical model for 3D perception of slanted and

curved surfaces and of 2D images: development, attention, and bistability. Vision Res. 44, 1147-

1187 (2004) .

20. Hayward.W.G., Tarr.M.J., & Corderoy.A.K. Recognizing silhouettes and shaded images across

depth rotation. Perception 28, 1197-1215 (1999) .

21. Hoffman.D.D. & Richards.W.A. Parts of recognition. Cognition 18, 65-96 (1984) .

22. Johnston.A. & Passmore.P.J. Shape from shading. I: Surface curvature and orientation.

Perception 23, 169-189 (1994) .

23. Johnston.A. & Passmore.P.J. Shape from shading. II. Geodesic bisection and alignment.

Perception 23, 191-200 (1994) .

24. Kersten.D., Mamassian.P., & Yuille.A. Object Perception as Bayesian Inference. Annu. Rev.

Psycho/. 55, 271-304 (2004) .



25. Kleffner.D.A. & Ramachandran.V.S. On the perception of shape from shading. Percept,

Psychophys. 52,18-36 (1992) .

26. Koenderink.J.J., van Doorn.A.J., Kappers,A.M., & Todd.J.T. The visual contour in depth.

Percept. Psychophys. 59, 828-838 (1997) .

27. Koenderink.JJ. What does the occluding contour tell us about solid shape? Perception 13, 321-

330 (1984) .

28. Langer.M.S. & Bulthoff.H.H. A prior for global convexity in local shape-from-shading. Perception

30,403-410 (2001) .

29. Langer.M.S. & Bulthoff.H.H. Depth discrimination from shading under diffuse lighting.

Percept/on 29, 049-660 (2000) .

30. Marr.D. Visual information processing: the structure and creation of visual representations.

Philos. Trans. R. Soc. LondBBiol. Sci. 290, 199-218 (1980) .

31. Kumar.T. & Glaser.D.A. Shape analysis and stereopsis for human depth perception. Vision Res.

32,499-512 (1992) .

32. Nakayama.K., Shimojo.S., & Silverman.G.H. Stereoscopic depth: its relation to image

segmentation, grouping, and the recognition of occluded objects. Perception 18, 55-68 (1989) .

33. Norman.J.F., Dawson.T.E., & Raines,S.R. The perception and recognition of natural object

shape from deforming and static shadows. Perception 29,135-148 (2000) .

34. Palmer.S., Simone.E., & Kube.P. Reference frame effects on shape perception in two versus

three dimensions. Perception 17, 147-163 (1988) .

35. Malik.J., Anderson,B.L., & Charowhas.C.E. Stereoscopic occlusion junctions. Nat. Neurosci. 2,

840-843 (1999) .

36. Rogers.B. & Graham,M. Motion parallax as an independent cue for depth perception.

Percept/on 8, 125-134 (1979) .

37. Ungerleider,LG. & Mishkin.M. Two cortical visual systems in Analysis of visual behavior (eds.

Ingle.D.G., Goodale.M.A. & Mansfield.R.J.Q.) 549-586 (MIT Press, Cambridge, MA, 1982) .

38. Gross.C.G., Rocha-Miranda.C.E., & Bender.D.B. Visual properties of neurons in inferotemporal

cortex of the Macaque. J. Neurophysiol. 35, 96-111 (1972) .

39. Desimone.R., Schein.S.J., Moran.J., & Ungerleider.L.G. Contour, color and shape analysis

beyond the striate cortex. Vision Res. 25, 441-452 (1985) .

40. Desimone.R., Albright,T.D., Gross.C.G., & Bruce.C. Stimulus-selective properties of inferior

temporal neurons in the macaque. J. Neurosci. 4, 2051-2062 (1984) .

41. Sereno.M.I. eta/. Borders of multiple visual areas in humans revealed by functional magnetic

resonance imaging. Science268, 889-893 (1995) .

42. Engel.S.A. eta/. fMRI of human visual cortex. Nature3G9, 525 (1994) .



43. Engel.S.A., Glover.G.H., & Wandell,B-A. Retinotopic organization in human visual cortex and

the spatial precision of functional MRI. Cereb. Cortex!, 181-192 (1997) .

44. DeYoe.E.A. eta/. Mapping striate and extrastriate visual areas in human cerebral cortex. Proc.

Nat/. Acad. Sci. U. S. A 93, 2382-2386 (1996) .

45. Kourtzi.Z. & Kanwisher.N. Cortical regions involved in perceiving object shape. J. Neurosci. 20,

3310-3318 (2000) .

46. Grill-Spector.K. eta/. A sequence of object-processing stages revealed by fMRI in the human

occipital lobe. Hum. Brain Mapp. 6, 316-328 {1998) .

47. Grill-Spector.K., Kourtzi.Z., & Kanwisher.N. The lateral occipital complex and its role in object

recognition. Vision Res. 41,1409-1422 (2001) .

48. Kanwisher.N., Chun.M.M., McDermott.J., & Ledden.P.J. Functional imaging of human visual

recognition. Brain Res. Cogn Brain Res. 5, 55-67 (1996) .

49. Kanwisher.N., McDermott.J., & Chun.M.M. The fusiform face area: a module in human

extrastriate cortex specialized for face perception. J. Neurosci. 17, 4302-4311 (1997) .

50. Fujita,!., Tanaka.K., Ito.M., & Cheng.K. Columns for visual features of objects in monkey

inferotemporal cortex [see comments]. Nature 360, 343-346 (1992) .

51. Kobatake.E. & Tanaka.K. Neuronal selectivities to complex object features in the ventral visual

pathway of the macaque cerebral cortex. J. Neurophysiol. 71, 856-867 (1994) .

52. Tanaka.K., Saito.H., Fukada.Y., & Moriya.M. Coding visual images of objects in the

inferotemporal cortex of the macaque monkey. J. Neurophysiol. 66,170-189 (1991) .

53. Tsunoda.K., Yamane.Y., Nishizaki.M., & Tanifuji.M. Complex objects are represented in

macaque inferotemporal cortex by the combination of feature columns. Nat. Neurosci. 4, 832-838

(2001) .

54. Baker.C.I., Behrmann.M., & Olson,C.R. Impact of learning on representation of parts and

wholes in monkey inferotemporal cortex. Nat. Neurosci. 5, 1210-1216 (2002) .

55. Sigala.N. & Logothetis.N.K. Visual categorization shapes feature selectivity in the primate

temporal cortex. Nature4*\5, 318-320 (2002) .

56. Gallant.J.L, Connor.C.E., & Van Essen.D.C. Neural activity in areas V1, V2 and V4 during free

viewing of natural scenes compared to controlled viewing. NeuroreportS, 1673-1678 (1998) .

57. Gallant.J.L., Braun.J., & Van Essen.D.C. Selectivity for polar, hyperbolic, and Cartesian gratings

in macaque visual cortex. Science 259, 100-103 (1993) .

58. Baylis.G.C., Rolls,E.T., & Leonard.C.M. Selectivity between faces in the responses of a

population of neurons in the cortex in the superior temporal sulcus of the monkey. Brain Res. 342,

91-102 (1985) .

PHS 398/2590 (Rev. 05/01) Page 37 Continuation Forniat Page


59. Perrett.D.I., Rolls.E.T., & Caan.W. Visual neurones responsive to faces in the monkey temporal

cortex. Exp. Brain Res. 47, 329-342 (1982) .

60. Glennerster.A., McKee.S.P., & Birch.M.D. Evidence for surface-based processing of binocular

disparity. Curr. Biol. 12, 825-828 (2002) .

61. He.Z.J. & Nakayama.K. Visual attention to surfaces in three-dimensional space. Proc. Natl.

Acad. Sci. U. S. A 92, 11155-11159 (1995) .

62. AnzaiA, Ohzawa,!., & Freeman,R.D. Neural mechanisms underlying binocular fusion and

stereopsis: position vs. phase. Proc. Natl. Acad. Sci. U. S. A 94, 5438-5443 (1997) .

63. Burkhalter.A. & Van Essen.D.C. Processing of color, form and disparity information in visual

areas VP and V2 of ventral extrastriate cortex in the macaque monkey. J. Neurosci. 6, 2327-2351

(1986) .

64. Gumming,B.G. & Parker.AJ. Binocular neurons in V1 of awake monkeys are selective for

absolute, not relative, disparity. J. Neurosci. 19, 5602-5618 (1999) .

65. Poggio.G.F., Motter.B.C., Squatrito.S., & Trotter.Y. Responses of neurons in visual cortex (V1

and V2) of the alert macaque to dynamic random-dot stereograms. Vision Res. 25, 397-406

(1985) .

66. Poggio.G.F., Gonzalez.F., & Krause.F. Stereoscopic mechanisms in monkey visual cortex:

binocular correlation and disparity selectivity. J. Neurosci. 8,4531-4550 (1988) .

67. von der.H.R., Zhou.H., & Friedman.H.S. Representation of stereoscopic edges in monkey visual

cortex. Vision Res. 40, 1955-1967 (2000) .

68. Bradley.D.C., Chang.G.C., & Andersen.R.A. Encoding of three-dimensional structure-from-

motion by primate area MT neurons. Nature'392, 714-717 (1998) .

69. DeAngelis.G.C., Gumming,B.G., & Newsome.W.T. Cortical area MT and the perception of

stereoscopic depth. Nature 394, 677-680 (1998) .

70. Maunsell.J.H. & Van Essen.D.C. Functional properties of neurons in middle temporal visual

area of the macaque monkey. II. Binocular interactions and sensitivity to binocular disparity. J.

Neurophysiol. 49, 1148-1167 (1983) .

71. Roy.J.P., Komatsu.H., & Wurtz.R.H. Disparity sensitivity of neurons in monkey extrastriate area

MST. J. Neurosci. 12, 2478-2492 (1992) .

72. Sakata,H. eta/. Neural coding of 3D features of objects for hand action in the parietal cortex of

the monkey. Philos. Trans. R. Soc. LondB Biol. Sci. 353, 1363-1373 (1998) .

73. Tsao.D.Y. eta/. Stereopsis activates V3A and caudal intraparietal areas in macaques and

humans. Neuron 39, 555-568 (2003) .

74. Uka.T. & DeAngelis.G.C. Contribution of Area MT to Stereoscopic Depth Perception; Choice-

Related Response Modulations Reflect Task Strategy. Neuron 42, 297-310 (2004) .



75. Janssen.P., Vogels.R., Liu.Y., & Orban.G.A. At least at the level of inferior temporal cortex, the

stereo correspondence problem is solved. Neuron 37, 693-701 (2003) .

76. Janssen.P., Vogels.R., & Orban.G.A. Macaque inferior temporal neurons are selective for

disparity-defined three-dimensional shapes. Proc. Nat!. Acad. Sci. U. S. A 96, 8217-8222 (1999) .

77. Janssen.P., Vogels.R., Liu.Y., & Orban.G.A. Macaque inferior temporal neurons are selective

for three-dimensional boundaries and surfaces. J. Neurosci. 21, 9419-9429 (2001) .

78. Janssen.P., Vogels.R., & Orban.G.A. Selectivity for 3D shape that reveals distinct areas within

macaque inferior temporal cortex. Science 288, 2054-2056 (2000) .

79. Janssen.P., Vogels.R., & Orban.G.A. Three-dimensional shape coding in inferior temporal

cortex. Neuron 27, 385-397 (2000) .

80. Liu.Y., Vogels.R., & Orban.G.A. Convergence of depth from texture and depth from disparity in

macaque inferior temporal cortex. J. Neurosci. 24, 3795-3800 (2004) .

81. Orban.G.A., Sunaert.S., Todd.J.T., Van Hecke.P., & Marchal.G. Human cortical regions

involved in extracting depth from motion. Neuron 24, 929-940 (1999) .

82. Sereno.M.E., Trinath.T., Augath.M., & Logothetis.N.K. Three-dimensional shape representation

in monkey cortex. Neuron 33, 635-652 (2002) .

83. Hinkle.D.A. & Connor.C.E. Disparity tuning in macaque area V4. Neuroreportl2, (2001) .

84. Watanabe.M., Tanaka.H., Uka.T., & Fujita.l. Disparity-selective neurons in area v4 of macaque

monkeys. J. Neurophysiol. 87, 1960-1973 (2002) .

85. Uka,T., Tanaka.H., Yoshiyama.K., Kato.M., & Fujita.l. Disparity selectivity of neurons in monkey

inferior temporal cortex. J. Neurophysiol. 84, 120-132 (2000) .

86. Hasselmo.M.E., Rolls.E.T., Baylis.G.C., & Nalwa.V. Object-centered encoding by face-selective

neurons in the cortex in the superior temporal sulcus of the monkey. Exp. Brain Res. 75,417-429

(1989) .

87. Logothetis.N.K. & Pauls.J. Psychophysical and physiological evidence for viewer-centered

object representations in the primate. Cereb. CortexS, 270-288 (1995) .

88. Logothetis.N.K., Pauls.J., & Poggio.T. Shape representation in the inferior temporal cortex of

monkeys. Curr. Biol. 5, 552-563 (1995) .

89. Logothetis.N.K., Pauls.J., Bulthoff.H.H., & Poggio.T. View-dependent object recognition by

monkeys. Curr. Biol. 4, 401-414 (1994) .

90. Touryan,J. & Dan.Y. Analysis of sensory coding with complex stimuli. Curr. Opin. Neurobiol. 11,

443-448 (2001) .

91. Touryan.J., Lau.B., & Dan.Y. Isolation of relevant visual features from random stimuli for cortical

complex cells. J. Neurosci. 22, 10811 -10818 (2002) .



92. Blanco,A., Delgado.M., & Pegalajar.M.C. A real-coded genetic algorithm for training recurrent

neural networks. Neural Netw. 14, 93-105 (2001) .

93. Li.J.P., Balazs.M.E., Parks.G.T., & Clarkson,P.J. A species conserving genetic algorithm for

multimodal function optimization. Evol. Comput. 10, 207-234 (2002) .

94. Wager.T.D. & Nichols.T.E. Optimization of experimental design in fMRI: a general framework

using a genetic algorithm. Neuroimage. 18, 293-309 (2003) .

95. Rovithakis.G.A., Chalkiadakis,!., & Zervakis.M.E. High-Order Neural Network structure selection

for function approximation applications using Genetic Algorithms. IEEE Trans. Syst. Man Cybern. B

Cybem. 34, 150-158 (2004) .

96. Edelman.S. Representation and recognition in vision (MIT Press, Cambridge, Massachusetts,

1999) .

97. Deneve.S., Latham,P.E., & Pouget.A. Efficient computation and cue integration with noisy

population codes. Nat. Neurosci. 4, 826-831 (2001) .

98. Zhang,K., Ginzburg,!., McNaughton.B.L, & Sejnowski.T.J. Interpreting neuronal population

activity by reconstruction: unified framework with application to hippocampal place cells. J.

Neurophysio/.79,^0^7-^044 (1998) .

99. Freedman.D.J., Riesenhuber.M., Poggio.T., & Miller.E.K. Categorical representation of visual

stimuli in the primate prefrontal cortex. Sc/ence291, 312-316 (2001) .

100. Freedman.D.J., Riesenhuber.M., Poggio,!., & Miller.E.K. A comparison of primate prefrontal

and iTemporal cortices during visual categorization. J. Neurosci. 23, 5235-5246 (2003) .

101. Pouget.A., Deneve.S., Ducom.J.C., & Latham,P.E. Narrow versus wide tuning curves: What's

best for a population code? Neural Comput. 11, 85-90 (1999) .

102. Sugihara.T,, Edelman.S., & Tanaka.K. Representation of objective similarity among three-

dimensional shapes in the monkey. Biol. Cybem. 78, 1-7 (1998) .

103. Pouget.A., Zhang.K., Deneve.S., & Latham,P.E. Statistically efficient estimation using

population coding. Neural Comput. 10, 373-401 (1998) .

104. Georgopoulos.A.P., Caminiti.R., Kalaska.J.F., & Massey.J.T. Spatial coding of movement: A

hypothesis concerning the coding of movement direction by motor cortical populations. Exp. Brain

Res. Supp. 7, 327-336 (1983) .

105. Sakai.K. & Miyashita.Y. Neural organization for the long-term memory of paired associates.

Nature 354, 152-155 (1991) .


Principal Investigator/Program Director (last, First, Middle): Connor, Charles E.

CHECKLISTTYPE OF APPLICATION (Check all that apply.)

IX] NEW application. (This application is being submitted to the PHS for the first time.)

I I SBIR Phase 1 I I SBIR Phase II: SBIR Phase I Grant No.

I I STTR Phase I I I STTR Phase II: STTR Phase I Grant No.

|~~l SBIR Fast Track

CH STTR Fast Track

REVISION of application number:(This application replaces a prior unfunded version of a new, competing continuation, or supplemental application.)

._, INVENTIONS AND PATENTSI I COMPETING CONTINUATION of grant number: (Competing continuation appl. and Phase II only)

(This application is to extend a funded grant beyond its current project period.)

I I SUPPLEMENT to grant number:

(This application is for additional funds to supplement a currently funded grant.)

I I CHANGE of principal investigator/program director.

Name of former principal investigator/program director:

I I FOREIGN application or significant foreign component.

No I I Previously reported

Yes. If "Yes." —- l~~l Not previously reported

1. PROGRAM INCOME (See instructions.)All applications must indicate whether program income is anticipated during the period(s) for which grant support is request. If program income isanticipated, use the format below to reflect the amount and source(s).

Budget Period Anticipated Amount Source(s)

2. ASSURANCES/CERTIFICATIONS (See instructions.)The following assurances/certifications are made and verified by thesignature of the Official Signing for Applicant Organization on the FacePage of the application. Descriptions of individual assurances/certifications are provided in Section III. If unable to certify compliance,where applicable, provide an explanation and place it after this page.

•Human Subjects; -Research Using Human Embryonic Stem Cells-•Research on Transplantation of Human Fetal Tissue -Women andMinority Inclusion Policy -Inclusion of Children Policy Vertebrate Animals'

•Debarment and Suspension; -Drug- Free Workplace (applicable to new[Type 1] or revised [Type 1] applications only); -Lobbying; -Non-Delinquency on Federal Debt; -Research Misconduct; -Civil Rights(Form HHS 441 or HHS 690); -Handicapped Individuals (Form HHS 641or HHS 690); -Sex Discrimination (Form HHS 639-A or HHS 690); -AgeDiscrimination (Form HHS 680 or HHS 690); -Recombinant DNA andHuman Gene Transfer Research; -Financial Conflict of Interest (exceptPhase I SBIR/STTR) -STTR ONLY: Certification of Research InstitutionParticipation.

3. FACILITIES AND ADMINSTRATIVE COSTS (F&A)/ INDIRECT COSTS. See specific instructions.

[3 DHHS Agreement dated: 9/17/04 | | No Facilities And Administrative Costs Requested.

Regional Office.

Date

I DHHS Agreement being negotiated with

| No DHHS Agreement, but rate established with

CALCULATION* (The entire grant application, including the Checklist, will be reproduced and provided to peer reviewers as confidential information.)

a. Initial budget period:

b. 02 year

c. 03 year

d. 04 year

e. 05 year

'Check appropriate box(es):

I I Salary and wages base

Amount of base $

Amount of base $

Amount of base $

Amount of base $

Amount of base $

250,000 x Rate applied

250,000 x Rate applied 63.50

250,000 x Rate applied 64.00

250,000 x Rate applied

x Rate applied

63.00 % = F&A costs $

= F&A costs

= F&A costs

= F&A costs

= F&A costs

157,500

64.00

158,750160,000

160,000

TOTAL F&A Costs 636.250 |

Modified total direct cost base I | Other base (Explain)

[ I Off-site, other special rate, or more than one rate involved (Explain)Explanation (Attach separate sheet, if necessary.):

4. SMOKE-FREE WORKPLACE Yes |_| No (The response to this question has no impact on the review or funding of this application.)

PHS 398 (Rev. 05/01) Page 4J. Checklist Form Page

Documents

PI: CONNOR, CHARLES E 9774 42 121 ication · Johns Hopkins University Krieger Mind/Brain Institute 3400 N Charles Street / 338 Krieger Hall Baltimore MD 21218 E-MAIL ADDRESS: [email protected]