The Embodied Cognition Literature

The Embodied Cognition Literature

Michael J. SpiveyDepartment of Cognitive ScienceUniversity of California, Merced

16th International Summer School in Cognitive Science, New Bulgarian University, Sofia, Bulgaria, 2009

…cont’d

Sensorimotor Influences in Cognition

• Motor Properties Influence Vision

• Visual Properties Influence Concepts

• Language Influences Visual Properties

• Language and Motor Influence Each Other

• Your Entire Body Participates in Thought

•

•

•

Sensorimotor Influences in Cognition

QuickTime™ and a decompressor

are needed to see this picture.

Scheerer & Lyons (1957)

Sensorimotor Influences in Cognition

ConceptsHaptic

Oculomotor

Visual

Oro-Facial

Auditory

Linguistic

Skeletalmotor

Memory

Sensorimotor Influences in Cognition

• Motor Properties Influence Vision

• Visual Properties Influence Concepts

• Language Influences Visual Properties

• Language and Motor Influence Each Other

• Your Entire Body Participates in Thought

•

•

•

Barsalou, Solomon, & Wu (1999)

List the featuresof a watermelon

LargeGreen Sweet

List the featuresof half a watermelon

Red SweetSeeds

Sensorimotor Influences in Cognition

ConceptsHaptic

Oculomotor

Visual

Oro-Facial

Auditory

Linguistic

Skeletalmotor

Memory

Sensorimotor Influences in Cognition

• Motor Properties Influence Vision

• Visual Properties Influence Concepts

• Language Influences Visual Properties

• Language and Motor Influence Each Other

• Your Entire Body Participates in Thought

•

•

•

Stimulus-Response CompatibilityTucker & Ellis (1998)

Stimulus-Response CompatibilityTucker & Ellis (1998)

Stimulus-Response CompatibilityTucker & Ellis (1998)

Stimulus-Response CompatibilityTucker & Ellis (1998)

The Importance of BeingSomatosensory Input

Bosbach, Cole, Prinz, Knoblich (2005)

Sensorimotor Influences in Cognition

ConceptsHaptic

Oculomotor

Visual

Oro-Facial

Auditory

Linguistic

Skeletalmotor

Memory

Sensorimotor Influences in Cognition

• Motor Properties Influence Vision

• Visual Properties Influence Concepts

• Language Influences Visual Properties

• Language and Motor Influence Each Other

• Your Entire Body Participates in Thought

•

•

•

Mental SimulationsStanfield & Zwaan (2001)

John put the pencil in the cup.John put the pencil in the drawer.

QuickTime™ and a decompressor

are needed to see this picture.

QuickTime™ and a decompressor

are needed to see this picture.

Mental SimulationsZwaan, Stanfield & Yaxley (2001)

The ranger saw the eagle in the sky.The ranger saw the eagle in the nest.

QuickTime™ and a decompressor

are needed to see this picture.QuickTime™ and a decompressor

are needed to see this picture.

Mental SimulationsZwaan, Madden, Yaxley, & Aveyard (2004)

The pitcher hurled the softball at you.You hurled the softball at the pitcher.

QuickTime™ and a decompressor

are needed to see this picture.QuickTime™ and a decompressor

are needed to see this picture.QuickTime™ and a decompressor

are needed to see this picture.QuickTime™ and a decompressor

are needed to see this picture.

Mental SimulationsMeteyard, Bahrami, & Vigliocco (2007)

QuickTime™ and aVideo decompressor

are needed to see this picture.“bomb, drop, fall…”

“climb, rise, fly…”

“cook, sew, write…”Upward:

Control:

Downward:

QuickTime™ and aVideo decompressor

are needed to see this picture.

Mental SimulationsMeteyard, Bahrami, & Vigliocco (2007)

“bomb, drop, fall…”

“climb, rise, fly…”

“cook, sew, write…”Upward:

Control:

Downward:

QuickTime™ and aVideo decompressor

are needed to see this picture.

QuickTime™ and aVideo decompressor

are needed to see this picture.

Mental SimulationsMeteyard, Bahrami, & Vigliocco (2007)

QuickTime™ and aVideo decompressor

are needed to see this picture.“bomb, drop, fall…”

“climb, rise, fly…”

“cook, sew, write…”Upward:

Control:

Downward:

QuickTime™ and aVideo decompressor

are needed to see this picture.

Verb Image Schemas

• An intimate relationship between language and space– Language is embodied and metaphorical (Gibbs, 2006;

Lakoff, 1999)– Spatial registration hypothesis (Chatterjee, 2001;

Coslett, 1999)– Language is not describable without reference to

cognitive processes such as attentional mechanisms (Langacker, 1991; Talmy, 1983)

– Perceptual symbol systems: cognitive representations are schematics, or simulations of perceptual experiences (Barsalou, 1999)

Richardson, Spivey, Barsalou, & McRae (2003)

Verb Image Schemas - Offline Data• Forced Choice Norming Study

Richardson, Spivey, Barsalou, & McRae (2003)

Verb Image Schemas• Free Drawing Norming Study

Richardson, Spivey, Barsalou, & McRae (2003)

argued with

respected

Verb Image Schemas• Did subjects agree with each other?

– 70% agreement in forced choice; Std errors about 10˚ in freeform.

• Did the tasks tap the same representation?– Axis/aspect angle correlation, R = 0.71

• Did they agree with linguistic intuitions?– Significant effect of expected axis in both studies

Verb Image Schemas - Online Data• Experiment 1 - Visual discrimination

– Hypothesis: comprehending a verb will activate a form of spatial representation that will be extended along either a horizontal or vertical axis, as shown by norming data.

– Non-compatible mental imagery will interfere with the detection or discrimination of overlapping visual stimuli (Perky, 1910; Craver & Arterberry, 2001)

– Participants presented with a spoken sentence containing a horizontal or vertical verb, and identified a rapidly flashed visual stimulus.

Richardson, Spivey, Barsalou, & McRae (2003)

Verb Image Schemas - Online Data

• Performance on the visual task was impaired when the visual stimulus appeared along the axis of the verb’s image schema

Richardson, Spivey, Barsalou, & McRae (2003)

Verb Image Schemas - Online Data• Experiment 2 - Memory• Hypothesis: the spatial structure associated with a verb will influence

the encoding of visual stimuli• Imagery improves memory (Paivio, 1969), and visual stimuli are

remembered better when they are presented in the same spatial locations at presentation and test (Santa, 1977; Zimmer 1998)

• Participants heard a sentence and saw pictures of the agent and patient of the sentence, presented centrally. At a later test stage, they were shown pairs of pictures and asked if they had been previously been presented together. At test, the pictures were presented in a horizontal or vertical alignment.

Richardson, Spivey, Barsalou, & McRae (2003)

Verb Image Schemas - Online DataRichardson, Spivey, Barsalou, & McRae (2003)

Verb Image Schemas - Online DataRichardson, Spivey, Barsalou, & McRae (2003)

Verb Image Schemas - Online Data

• At test, picture pairs were recognized faster if they were presented in the same orientation as the associated verb’s image schema

Richardson, Spivey, Barsalou, & McRae (2003)

Verb Image Schemas • In two offline tasks, naïve participants categorized verbs as having a

horizontal or vertical image schema• There was a good degree of consistency between tasks and between

participants• The offline tasks provided norms for experiments involving online

verb comprehension• In both a visual discrimination and a memory task, reaction times

showed an interaction between the horizontal/vertical nature of the verb’s image schema, and the horizontal/vertical position of the visual stimuli

• Such spatial effects of verb comprehension provide evidence for the perceptual-motor character of linguistic representations (see also Bergen, Lindsay, Matlock & Narayanan, 2007).

Richardson, Spivey, Barsalou, & McRae (2003)

Sensorimotor Influences in Cognition

ConceptsHaptic

Oculomotor

Visual

Oro-Facial

Auditory

Linguistic

Skeletalmotor

Memory

attic

basement

Iconicity Affects Semantic Relatedness Judgments (Zwaan & Yaxley, 2003)

basement

attic

Iconicity Affects Semantic Relatedness Judgments (Zwaan & Yaxley, 2003)

Sensorimotor Influences in Cognition

ConceptsHaptic

Oculomotor

Visual

Oro-Facial

Auditory

Linguistic

Skeletalmotor

Memory

Sensorimotor Influences in Cognition

• Motor Properties Influence Vision

• Visual Properties Influence Concepts

• Language Influences Visual Properties

• Language and Motor Influence Each Other

• Your Entire Body Participates in Thought

•

•

•

One-Referent Context Two-Referent Context

(Spivey, Tanenhaus, Eberhard, & Sedivy, 2002)

“Put the apple on the towel in the box”“Put the apple that’s on the towel in the box”

Ambiguous:

Unambiguous:

Motor Constraints During Language

Pour the egg in the bowl onto the flour.Pour the egg that’s in the bowl onto the flour.

Compatible Incompatible

(Chambers, Tanenhaus, & Magnuson, 2004)

Motor Constraints During Language(Chambers, Tanenhaus, & Magnuson, 2004)

Motor Facilitation During Language

The Action-Sentence Compatibility Effect(Glenberg & Kaschak, 2002)

QuickTime™ and a decompressor

are needed to see this picture.

Motor Facilitation During Language

Close the drawer.Open the drawer.

QuickTime™ and a decompressor

are needed to see this picture.

The Action-Sentence Compatibility Effect(Glenberg & Kaschak, 2002)

Motor Facilitation During Language

QuickTime™ and a decompressor

are needed to see this picture.

Andy delivered the pizza to you.You delivered the pizza to Andy.

The Action-Sentence Compatibility Effect(Glenberg & Kaschak, 2002)

Motor Facilitation During Language

Liz told you the story.You told Liz the story.

QuickTime™ and a decompressor

are needed to see this picture.

The Action-Sentence Compatibility Effect(Glenberg & Kaschak, 2002)

Motor Facilitation During Language

(For variation in the temporal dynamics,see Kaschak & Borregine, 2008)

The Action-Sentence Compatibility Effect (Glenberg & Kaschak, 2002)

QuickTime™ and a decompressor

are needed to see this picture.

Motor Activation During Language(Pulvermüller, 1999)

Motor Activation During Language(Pulvermüller, 1999)

Language Networks

Motor Activation During Language(Pulvermüller, 1999)

Language Networks Associated with Visual Objects

Motor Activation During Language(Pulvermüller, 1999)

Language Networks Associated with Actions

Motor Activation During Language(Martin, Wiggs, Ungerleider, & Haxby, 1996)

Motor Activation During Language(Pulvermüller, Härle, & Hummel, 2001)

QuickTime™ and a decompressor

are needed to see this picture.

Motor Activation During Language(Pulvermüller, Härle, & Hummel, 2001)

Motor Activation During Language(Pulvermüller, Härle, & Hummel, 2001)

Sensorimotor Influences in Cognition

ConceptsHaptic

Oculomotor

Visual

Oro-Facial

Auditory

Linguistic

Skeletalmotor

Memory

Sensorimotor Influences in Cognition

• Motor Properties Influence Vision

• Visual Properties Influence Concepts

• Language Influences Visual Properties

• Language and Motor Influence Each Other

• Your Entire Body Participates in Thought

•

•

•

Your Entire Body Participates in Thought• Subliminal Priming of Old Age (Bargh et al., 1996)

• Associations with Approach or Avoid Gestures(Cacioppo and colleagues)

• A Fleeting Moment of Zen and movement/memory

• Orofacial muscles influence judgments (and IAT)(Cacioppo and colleagues)

• Push/Pull response to pos/neg words (Chen &Bargh)

Sensorimotor Influences in Cognition

ConceptsHaptic

Oculomotor

Visual

Oro-Facial

Auditory

Linguistic

Skeletalmotor

Memory