Download ppt - Grounding Knowledge in the Brain’s Modal Systems

1

Grounding Knowledge in the Brain’s Modal Systems

Lawrence W. BarsalouDepartment of Psychology

Emory University

June 2010

Research supported by

National Science Foundation grantsSBR-9421326, SBR-9796200, SBR-9905024, BCS-0212134

DARPA grants FA8650-05-C-7256, FA8650-05-C-7255

Emory fMRI seed grants

Grounding Knowledge

Grounding Knowledge 2

The conceptual system

1. represents knowledge about experience in the world

2. organizes knowledge categorically• concepts in memory represent categories in the world

3. provides representational support across cognitive tasks• online processing

•high-level perception•categorization of perceived entities and events• inferences that go beyond the information given

• offline processing• in memory, language, and thought•conceptualization of entities not present

• guides learning•provides interpretations of novel material•expertise grows with conceptual vocabulary


The traditional approach: Semantic memorye.g., Tulving, 1972; Collins & Loftus, 1975

•modular• distinct from episodic memory and the brain’s modal systems (e.g., vision)

•amodal• non-perceptual representations


The transduction principle in amodal conceptual systems

•amodal symbols are transduced from modal states• constitute knowledge about categories

• a modular system with unique operating principles

soft

barks

legstail

pat


§¥ÞŧЭ

The transduction principle in amodal symbol systems

•amodal symbols are not linguistic symbols• the conceptual symbols that underlie language

• transduction underlies many common approaches to representation• feature lists, semantic nets, schemata, frames, production systems, etc.


“dog”

Representing knowledge in amodal symbol systems

•amodal symbols later represent categories in their absence• constitute the knowledge that underlies memory, language, thought

• no representations in modal systems required or involved

§¥ÞŧЭ


An alternative approach

•non-modular• concepts utilize sensory-motor and other modal systems (e.g., affect)

•modal• modal simulations represent concepts


Capturing neural activity in the brain’s modal systems Damasio (1989), Barsalou (1999, 2003, 2005), Simmons & Barsalou (2003)

•modal states are captured during online experience• by conjunctive neurons in hierarchically-organized association areas

•capture is partial• not complete


“dog”

Running simulations to represent knowledge

•simulations (reenactments) represent categorical knowledge• may often be unconscious, not necessarily conscious (as in imagery)

• always partial, may be distorted

• could be exemplars, averages of exemplars, etc.

Empirical evidence for simulation:A general computational mechanism in the brain

•evidence across disciplines• cognitive psychology• social psychology• developmental psychology• cognitive neuropsychology• cognitive neuroscience

•evidence across processes• perception (perceptual anticipation)• working memory (imagery and rehearsal)• implicit memory (sensory-motor priming)• explicit memory (recollection)• knowledge representation (conceptualization)• language (meaning)• thought (envisioning possible scenarios)• social cognition (mirroring and empathy)


For a recent review see:Barsalou, L.W. (2008). Grounded cognition. Annual Review of Psychology, 59, 617-645.

Conjecture:Multiple control systems.

One representation system.


Common misperceptions

•NOT a classic empiricist theory• in principle, simulations for categories could be genetically determined

• strong genetic constraints determine feature systems and association areas

•anticipating important categories in evolutionary history

•NOT a recording system, symbolic operations are central (Pylyshyn, 1973)

• doesn’t simply capture records of experience

• instead, interpretation of experience lies at the core of this account

•via symbolic operations

•implemented with mechanisms not presented today

•knowledge does NOT solely reflect perception of the external world• also perception of mental states, meta-cognition, affect, etc. (“introspection”)

• central to abstract concepts


Overview of research to be presented

1. Examples of simulation

2. Simulation in situated action

3. Simulation in natural abstract categories

4. Simulation in symbolic operations• predication

• conceptual combination

12Grounding Knowledge










Shape inferences in comprehensionZwaan, Stanfield, and Yaxley (2002)

•hypothesis• if readers simulate the meaning of a text to understand it, then text representations should have perceptual properties, even when these properties are not mentioned

•method• a sentence was presented

• a picture was presented and participants had to name it as quickly as possible

•key manipulation• whether or not the pictured object had a shape that matched the shape of the object implied in the sentence

The bird sat quietly in the tree (implies a bird with its wings folded)

The bird flew quickly across the sky (implies a bird with its wings flapping)

Examples of sentences Examples of pictures

Mentioned Not Mentioned


Results

•conclusion• when comprehending sentences, participants simulated the scenes described,thereby committing to a particular shape of the objects mentioned

Nam

ing

RT

(m

s)

Picture Shape


Simulating actions to represent verb meaningHauk, Johnsrude, and Pulvermüller (2004)

•method• participants read isolated words in an fMRI scanner (2.5 sec rate)

• subsets of words referred to face, arm, or leg movements (mixed with non-motor words)

•e.g., “lick,” “pick,” “kick”

• participants later performed actual motor movements (localizer task)

•i.e., moved their tongue, finger, or foot

•prediction• if simulations represent word meanings,then words for different body part movementsshould activate the respective regions of the motor system

• these activations should also lie near thosefor the localizer task


Results

• the predicted somatotopic order of word activations appeared• activations occurred in the motor system for the action words

•relative to reading strings of hash marks

• leg activations were vertically highest, then arm, then face

• leg and arm activations overlapped for the word and localizer tasks

• face discrepancies could indicate less correspondence between words and the localizer task










Conceptual processing is situatedBarsalou (2003), Smith & Semin (2004), Yeh & Barsalou (2006)

•situations frame conceptual representations• concepts are not learned and represented in a vacuum

• concepts are learned and represented in a situated manner

•background situations prepare agents for situated action• provide useful inferences about:

•settings

•agents and objects

•actions and events

•mental states

• tailored to different courses of situated action for the same concept

•e.g., chairs in living rooms vs. offices vs. jets

•situational inferences delivered via multimodal simulations• across the relevant modalities


Situating physical objects Wu and Barsalou (2009)

• task and results• participants produced the features of objects (e.g., apple)

• produced non-requested information about settings and mental states

• suggests that they situated their conceptualizations of the objects


Situating manipulable objects Chao & Martin (2000)

•participants viewed manipulable objects• activation occurred in motor and parietal areas associated with manipulating objects (but did not occur for non-graspable objects)

•participants situated the manipulable objects with respect to action• on categorizing a visual picture, motor inferences were produced

•review of related findings• Lewis (2006)

Taste inferences for foodsSimmons, Martin, & Barsalou (2005)

•presented participants with pictures of foods and houses• relatively tasty foods from the undergraduate perspective

•no fruits and vegetables

• 1-back task

• food pictures should activate categorically-related inferences

• taste areas should become active

Situated Simulation 22


64, -4, 20 tasting sucrose - deAraujo et al. (2003), p.2063 - R. Operculum 36, 0, 16 tasting chocolate - Small et al. (2001), p.1724 - R Insula/Operculum54, 12, 10 tasting umami - deAraujo et al. (2003), p.316 - R Insula/Operculum45, 3, 5 tasting glucose - Francis et al. (1999), p.457 - operculum45, 1, -9 tasting sucrose - deAraujo et al. (2003), p.2063 – Anterior Insula36, -6, 9 viewing food pictures - Simmons, Martin, & Barsalou- R Insula/operculum

Activations in primary gustatory cortex for foods(frontal operculum)

Z = 20 Z = 16 Z = 10 Z = 9 Z = 5 Z = -9


Z = -30

Z = -24

Z = -20

Z = -18

Z = -10

Z = -6

-4, 51, -30 abstract reward - O'Doherty ,et al. (2001)-10 ,42, -24 abstract reward - O'Doherty ,et al. (2001)-4, 30, -20 abstract reward - O'Doherty ,et al. (2001)-9, 26, -18 tasting glucose - Frances, Rolls, et.al. (1999)-32, 50, -10 flavor center - de Araujo et, Rolls, Kingelbach, et al. (2003)-18, 48, -10 property verification, - Simmons, Pecher, Hamann, et al. (submitted)-34, 26, -6 tasting umami- de Aaujo, et al. (2003)-21, 33, -18 viewing food pictures- Simmons, Martin, & Barsalou

Activations in the taste reward area for foods(orbitofrontal cortex)


Situating social objectsGil-da-Costa, Braun, Lopes, Hauser, Carson, Herscovitch, & Martin (2004)

•monkeys listened to recorded coos and screams of other monkeys• compared PET activations to those for unfamiliar sounds (musical instruments)

•results• activations in auditory areas (perception)• activations for situated inferences

•visual areas• inferior temporal (faces)• superior temporal (expression, motion)

•frontal and limbic areas• medial prefrontal (mental states?)• amgydala (emotion)• hippocampus (emotional memory)

• the monkeys simulated the situations associated with the sounds• provides continuity with the human conceptual system (Barsalou, 2005)

A

D EC

BA

D EC

B









Representing abstract concepts with simulationBarsalou (1999)

•concepts are typically situated• not represented in vacuum, but in a setting with agents, objects, events, etc.

•concrete concepts• objects, settings, and actions in situations• relatively “local” in time and space• perceived externally

•abstract concepts• complex configurations of information distributed across settings and events• internally perceived content especially important

• e.g., mental states, affect, cognitive operations

•simulating abstract concepts • simulating the associated configuration of information,including mental states and events


Schwanenflugel (1991)

28

Exploratory study Barsalou & Wiemer-Hastings (2005)

•method• participants produced properties for abstract and concrete concepts

• TRUTH, FREEDOM, INVENTION vs. SOFA, BIRD, CAR

•results• participants produced broad situational content for both types of concepts

•people generally situate both kinds of concepts

• abstract concepts activated more mental state and setting/event properties, whereas concrete concepts activated more entity properties

•the two kinds of concepts rely on different situational information

Proportions of property types

Concept type Entity Setting/Event Mental State Concrete .26 .46 .21 Abstract .15 .52 .28


29

Assessing simulation in abstract concepts with fMRIWilson, Simmons, Martin, & Barsalou (in preparation)

• two phases of the experiment

• localizer phase• identified brain areas that perform abstract forms of processing

• blocked design

•priming phase• assessed the semantic content of two abstract concepts

• fast event-related design

•hypothesis• simulations of abstract processing will represent the abstract concepts

LocalizerPhase

Semantic PrimingPhase


30

• thoughts localizer• participants viewed blocks of complex scenes

• for each picture, participants answered the following question to themselves

•“What are the thoughts of people in the picture?”

•counting localizer• participants viewed blocks of complex scenes

• for each picture, participants answered the following question to themselves

•“How many entities are there in the picture?”

LocalizerPhase


Note. Localizer blocks were also included for two concrete localizers, color and motion, not discussed here.


31

LocalizerPhase


• thoughts – counting• medial prefrontal, precuneus

• bilateral anterior and superior temporal

•counting – thoughts• bilateral intraparietal sulcus

p < .0001, corrected, random effects31Grounding Knowledge

L

R

x = -45

x = 48

x = -5

y = -60L

32

•semantic priming trials• fast-event related design

•concepts ordered randomly

•random ISI jitter

•catch trials to deconvolveword primes and pictures

• possible responses: “Word applies” or “Word doesn’t apply”

• pictures promoted deepsemantic processing

•hypothesis• simulations underlie meaning

LocalizerPhase


convince

5 sec(fMRI images of interest)

Response

2.5 sec

Note. Semantic priming trials were also included for two concrete concepts red and rolling, not discussed here.

arithmetic

5 sec(fMRI images of interest)

Response

2.5 sec


33

•convince – arithmetic• dark blue areas

• medial prefrontal, precuneus, superior temporal

• no arithmetic – convince activations in localizer areas

• simulations underlie meaning

•arithmetic – convince• orange areas

• intraparietal sulcus

• no convince – arithmeticactivations in localizer areas

• simulations underlie meaning

LocalizerPhase


p < .05, corrected, random effects33Grounding Knowledge

L y = -45

A

L

R

x = -45

x = 48

x = -5

y = -60L

3434Grounded Cognition

Assessing simulation in abstract concepts with fMRIWilson, Barrett, Simmons, Barsalou (in preparation)

Physical Threat Situation You row on a lake to experience the feel of storm waves. As you head toward the middle of the lake, white-capped waves break across your small boat with increasing frequency. As the waves become increasingly rough, water pours into the boat. The boat sinks. You try to keep your head above water, as wave after wave crashes over you. Your wet clothes feel heavy on your body, making it difficult to stay afloat.

Social Threat SituationYou’re leading an important group presentation at work. You’re unprepared for your boss’s questions because a couple of co-workers didn’t pull their weight in preparing for the meeting. The presentation finishes awkwardly. Your boss thanks you coolly. His associates sit looking at each other, wondering what to say next. You can feel the sweat forming under your arms.

OBSERVE

TASK“How easy was it for you to experience

OBSERVE in the context of the situation?”

Very easy (3), Somewhat easy (2), Not easy (1)

Other concepts used: PLAN FEAR ANGER

Grounded Cognition 35

BOLD responses only to words,p < .05, corrected, random effects

Visual processing(bilateral superior occipital)

Auditory processing(bilateral superior temporal)

Object processing(bilateral fusiform, BA 37)

Observe – (Fear, Anger)physical situationssocial situationsboth

x = -32 x = 55x = -47

R R Rz = 26 z = 5z = -8

OBSERVE activations as a function of situation

Physical Social Overlap

1. L. Sup. Temporal extends into pole, insula

2. R. Sup. Temporal extends into pole

3.Bilateral Mid Occipital extends into inf. parietal

4. L. ITG/MTG extends into fusiform, PHC

6. R. ITG/MTG

7. Precuneus

8. Mid Cingulate

9. R. Middle Frontal

10. R. Insula

z = -12

z = 28

y = -17

8% 83% 9%

Grounded Cognition

BOLD responses only to words,p < .05, corrected, random effects










Symbolic operations

• typically viewed as a problem for grounded views• assumed to be possible only in amodal symbolic systems

• examples of symbolic operations• predication “Pumpkins are orange” ORANGE (pumpkins)

• conceptual combination “The cat is on the sofa” ON (cat, sofa)

• simulation-based accounts of symbolic operationsBarsalou, L.W. (1999). Perceptual symbol systems. Behavioral and Brain Sciences, 22,

577-609.

Barsalou, L.W. (2003). Abstraction in perceptual symbol systems. Philosophical Transactions of the Royal Society of London: Biological Sciences, 358, 1177-1187.

Barsalou, L.W. (2008). Grounding symbolic operations in the brain’s modal systems. In G.R. Semin & E.R. Smith (Eds.), Embodied grounding: Social, cognitive, affective, and neuroscientific approaches (pp. 9-42). New York: Cambridge University Press.










Grounding color semantics in the visual systemSimmons, Ramjee, Beauchamp, McRae, Martin, & Barsalou (2007)

•participants verified “possible” properties of conceptsin an fMRI scanner

• color properties

• motor properties

•hypothesis• verifying color properties produces color simulations in the brain’s color perception system

•builds on previous findings in the literature• e.g., Chao & Martin (1999), Oliver & Thompson-Schill (2003)

MILK..

white..

(true)

WATER..

white..

(false)

DOOR KNOB..

turned..

(true)

•10 participants, event-related design (168 trials across 7 runs)• concept-only catch trials used to deconvolve concepts and properties


Color localizer task Establishing the color perception areas

• the Farnsworth-Munsell color judgment task (adapted for fMRI)• participants judged whether or not hues are ordered from lightest to darkest

• for multiple chromatic and achromatic wheels

• performed after the property verification scanning runs

Chromatic Achromatic

Ordered

Not ordered• blocked design, 4 runs• 3 chromatic blocks / run• 3 achromatic blocks / run

LocalizerProperty

Verification


Chromatic > Achromatic Wheels

Areas active for color perception

p < .05, corrected, random effects

Left

Front

LocalizerProperty

Verification


Color > Motor Properties

Areas active for verifying color properties

p < .05, corrected, random effectsLeft

Front

LocalizerProperty

Verification


Overlap

Chromatic > Achromatic Wheels

Color > Motor Properties

Overlapping activations for verifying color properties and

perceiving color

A subsequent ROI analysis on the fusiform cluster identified by the color perception task (-33, -36, -9) found color > motor properties, p < .05, corrected, random effects.

Tootell et al. (2004) argue that this overlapping area is a primary color processing area in macaques

Left

Front









46

Occlusion effects in conceptual combinationWu & Barsalou (2009)

• in perception, occluded features are less salient than unoccluded ones• for LAWN, dirt and roots are less salient than green and blades

• in conceptual processing, is there an analogous occlusion effect?• if simulation is used to combine meanings, there should be

•nouns task• some participants generated features for nouns having occluded features

•neutral instructions (nothing mentioned about imagery)• simulation prediction:

• unoccluded features should be produced more than occluded featuresLAWN green, blades > dirt, roots

•noun phrases (NPs) task• other participants generated features for NPs with revealing modifiers

•either novel or familiar NPs (neutral instructions)• simulation prediction:

•occluded features should be produced more often than for isolated nounsROLLED-UP LAWN dirt, roots > LAWN dirt, roots


47

Results

•occlusion affected conceptual processing• external features more likely than internal features for nouns• internal features become more likely for both novel and familiar NPs


• not the result of rules associated with modifiers• e.g., occluded features are not produced more often for ROLLED UP SNAKE than for SNAKE


Assessing conceptual combination with fMRIJames, Simmons, Barbey, Hu, & Barsalou (in preparation)

• two critical kind of trials• independent " . . "

• combination " − . "

• fast event-related design• trial types interspersed randomly

• random ISI jitter

• catch trials to deconvolvemodifiers and head nouns

• familiarity responses at " . "• “Occurs once a month or more” or

• “Occurs less than once a month”

distressed . reverend .

Modifier1 sec

Head Noun1 sec

3 sec

FamiliarityJudge Modifier

FamiliarityJudge Head Noun

3 sec

distressed − reverend .

Modifier1 sec

Head Noun1 sec

3 sec

FamiliarityJudge Noun Phrase

3 sec

Independent Trials

Combination Trials


Examples of modifiers and head nouns

•mental state modifiersdistressed reverend

pleasing cloves

•motion modifierssoaring balloon

swaying oak

• location modifiersocean shrimp

auditorium piano

• no modifier or head noun repeated

• head nouns counter-balanced forlength, frequency, category, typicality

• words counter-balanced across participants so that every word occurred in both the independent and combination conditions

distressed . reverend .

Modifier1 sec

Head Noun1 sec

3 sec

FamiliarityJudge Modifier

FamiliarityJudge Head Noun

3 sec

distressed − reverend .

Modifier1 sec

Head Noun1 sec

3 sec

FamiliarityJudge Noun Phrase

3 sec

Independent Trials

Combination Trials

Mental State modifiers – Motion and Location Modifiers

Motion modifiers – Mental State and Location Modifiers

Location modifiers – Mental State and Motion Modifiers



Simulations in modal areas represented the modifiers

• areas more active for one modifier modality than for the other two, p < .05, corrected, random effects

IndependentMental State Modifiers

(distressed, pleasing)

IndependentLocation Modifiers(e.g., ocean, auditorium)

IndependentMotion Modifiers

(soaring, swaying)

medial pre-frontal(mental states area)

L temporal(motion area)

LR parahippocampus(location area)

L L

L

L

x=-9 y=-54

z=-11

y=-33

z=17 z=-3

L

Mental State modifiers – Motion and Location Modifiers

Motion modifiers – Mental State and Location Modifiers

Location modifiers – Mental State and Motion Modifiers



Mental State Modifiers(distressed, pleasing)

Location Modifiers(e.g., ocean, auditorium)

Motion Modifiers(soaring, swaying)

Combined Modifiers

Independent Modifiers

Combined Modifiers (All) – Independent Modifiers (All)



Combined modifiers − Independent modifiers

•conjecture• the conceptual system holds off committing to a specific sense of the modifier

• a right hemisphere network infers a situation that contains the modifier and a predicted head noun

• areas more active for combined modifiers across modalities than for independent modifiers across modalities, p < .05, corrected, random effects• the same areas were generally active for individual modalities, but less robustly

R inferior parietalR supramarginalR superior temporal

y=-51 y=44 z=28L L

R medial frontal

z=43 x=40L x=27

Combined Nouns (All) – Independent Nouns (All)


• areas more active for combined nouns across modalities than for independent nouns across modalities, p < .05, corrected, random effects• similar areas were generally active for individual modalities, but less robustly


Combined nouns – Independent nouns

•conjecture• this bilateral multimodal network simulates a specific situation in which the modifier and head noun concepts have been integrated

LR parahipp gyrusLR fusiform gyrusLR lingual gyrus

L superior temporalLR precuneusL inferior frontal

L

L pre-central gyrusLR post central gyrusLR inferior parietal

L L L

L Ly=-53

z=45 z=18

x=-14

z=2

y=-38


Conclusions

•simulation underlies a wide variety of conceptual processes• representation of object categories

• representation of abstract categories

• symbolic operations (predication, conceptual combination)

•simulations of concepts are situated• represent background information about situations relevant to goal pursuit



Contributors

• post docs• Anna Borghi• Andy James• Diane Pecher• Rene Zeelenberg

• graduate students• Aron Barbey• Sergio Chaigneau• Linda Confalonieri• Shlomit Finkelstein• Carla Harenski• Irene Kan• Zhaohui Liu• Barbara Luka• Jesse Prinz• Daniel Richardson• Ava Santos• Kyle Simmons• Karen Solomon• Saskia van Dantzig• Christy Wilson• Ling-Ling Wu• Wenchi Yeh

• undergraduates• Melissa Armstrong• Joy Lynn Brasfield• Shurin Hase• Vimal Ramjee• Christy Wilson

• faculty• Lisa Barrett• Michael Beauchamp• Cynthia Breazeal• Andy Butler• Zach Estes• Rob Goldstone• Stephan Hamann• Xiaoping Hu• Alex Martin• Ken McRae• Paula Niedenthal• Giuseppe Pagnoni• Linda Smith• Michael Spivey• Sharon Thompson-Schill• Katja Wiemer-Hastings• Phil Wolff