Language Embedded Language Embedded

in Systems in Systems of Perception and of Perception and

ActionAction

Rolf A. ZwaanRolf A. Zwaan

Florida State UniversityFlorida State University

Experiential TracesExperiential Traces

Experiential TracesExperiential Traces

“referential” linguistic

Experiential AssociationsExperiential Associations

“intra-referential” intra-linguistic

Experiential AssociationsExperiential Associations

cross-domain

Hebbian LearningHebbian Learning

Co-occurrence primary factor in the Co-occurrence primary factor in the forming of associationsforming of associations Intra-linguistic;Intra-linguistic; Intra-referential;Intra-referential; Cross-domain (grounding).Cross-domain (grounding).

Experiential TracesExperiential Traces

Linguistic TracesLinguistic Traces AuditoryAuditory MotorMotor VisualVisual TactileTactile

Referential TracesReferential Traces VisualVisual MotorMotor AuditoryAuditory OlfactoryOlfactory TactileTactile SomatosensorySomatosensory

LSA/Encyclopedia_all

00.050.1

0.150.2

0.250.3

0.350.4

0.450.5

0 1000 2000 3000 4000distance from Miami (miles)

cosine

Capturing Capturing Second-order Second-order CorrelationsCorrelations

r2 = .30

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0 1 2 3

distance from the Netherlands (# of intervening countries)

LSA cosine

r2 =.65

Capturing Capturing Second-order Second-order CorrelationsCorrelations

Utility of Linguistic Utility of Linguistic Traces/AssociationsTraces/Associations

Additional layer of associations.Additional layer of associations. May enhance “fluency.”May enhance “fluency.”

““Referential bootstrapping.”Referential bootstrapping.”

Referential Referential BootstrappingBootstrapping

Name this animal:Name this animal: What has horns like a giraffe,What has horns like a giraffe, a deer head,a deer head, a a

horse neck,horse neck, and legs like a zebra? and legs like a zebra? http://www.westerville.k12.oh.us/ZOO/Orlich/pages/Okafrtpg.htmhttp://www.westerville.k12.oh.us/ZOO/Orlich/pages/Okafrtpg.htm

Referential Referential BootstrappingBootstrapping

ResonanceResonance

Experiential traces “resonate” with Experiential traces “resonate” with linguistic input.linguistic input. Visual resonance.Visual resonance. Motor resonance.Motor resonance.

The role of resonance in The role of resonance in comprehension.comprehension.

PredictionsPredictions

Prediction 1Prediction 1 Language comprehension affects Language comprehension affects

subsequent perceptual performance.subsequent perceptual performance. Prediction 2Prediction 2

Incidentally acquired visual Incidentally acquired visual representations affect subsequent representations affect subsequent language processing.language processing.

Prediction 3Prediction 3 Language comprehension affects Language comprehension affects

concurrent motor performance.concurrent motor performance.

Comprehension Affects Comprehension Affects Perceptual Performance Perceptual Performance

(Zwaan, Stanfield, & Yaxley, 2002)(Zwaan, Stanfield, & Yaxley, 2002)

The ranger saw the eagle in the nest.[SAW, RANGER, EAGLE][IN, EAGLE, NEST]

The ranger saw the eagle in the sky. [SAW, RANGER, EAGLE][IN, EAGLE,

SKY]

The ranger saw the eagle in its nest. The ranger saw the eagle in the sky.

Picture Recognition Picture Recognition and Naming Latenciesand Naming Latencies

(Zwaan, Stanfield & Yaxley, 2002)(Zwaan, Stanfield & Yaxley, 2002)

600

650

700

750

800

Recognition Naming

MatchMismatch

RT (ms)

Language may Involve Language may Involve Dynamic Dynamic

RepresentationsRepresentations(Zwaan. Madden, Yaxley, & Aveyard, 2004)(Zwaan. Madden, Yaxley, & Aveyard, 2004)

The midfielder passed the ball to you.

Zwaan, Madden, Yaxley, & Aveyard, 2004

Prediction 2Prediction 2

Incidentally acquired visual Incidentally acquired visual representations affect subsequent representations affect subsequent language processing.language processing.

The Visual-Memory The Visual-Memory Paradigm Paradigm (Aveyard, Zwaan, Radach, & (Aveyard, Zwaan, Radach, &

Vorstius, in preparation)Vorstius, in preparation) Memory PhaseMemory Phase

Word-picture verification (4 times)Word-picture verification (4 times) Reading Phase Reading Phase (ostensibly unrelated)(ostensibly unrelated)

Eye movements trackedEye movements tracked

Memory PhaseMemory Phase

eagle

Memory PhaseMemory Phase

Reading PhaseReading Phase

The ranger was in the park on his The ranger was in the park on his morning round. It was a bright day. morning round. It was a bright day. In the sky there was an eagle, In the sky there was an eagle, soaring above the prairie.soaring above the prairie.

First FixationsFirst Fixations

300

350

400

450

500

550

600

650

In the sky an eagle wassoaring

above the…

MatchMismatch

Gaze DurationsGaze Durations

500

550

600

650

700

750

800

850

In the sky an eagle wassoaring

above the…

MatchMismatch

Visual World Visual Visual World Visual MemoryMemory

SituatedSituated Visual SearchVisual Search Auditory Auditory

PresentationPresentation DV = eye fixations DV = eye fixations

during display during display viewingviewing

Re-situatedRe-situated Memory SearchMemory Search Visual Visual

PresentationPresentation DV = eye fixations DV = eye fixations

during readingduring reading

Prediction 3Prediction 3

Motor traces are activated by Motor traces are activated by linguistic input.linguistic input.

Action Compatibility Action Compatibility EffectEffect

(Glenberg & Kaschak, 2002)(Glenberg & Kaschak, 2002) Sensibility judgmentsSensibility judgments

Close the drawer.Close the drawer. Andy delivered the pizza to you.Andy delivered the pizza to you. Liz told you the story.Liz told you the story. Break the cloud.Break the cloud.

Action Parameters Action Parameters and Motor Resonanceand Motor Resonance

Actions have various parameters.Actions have various parameters. Direction.Direction. Force.Force. Manner (e.g., open hand vs. closed hand).Manner (e.g., open hand vs. closed hand). Speed.Speed. Duration.Duration.

Which action parameters exhibit Which action parameters exhibit resonance in language comprehension?resonance in language comprehension?

Measuring Response Measuring Response ForceForce

Measuring Response Measuring Response ForceForce

05000

100001500020000

2500030000350004000045000

Time

Force (kPa)

10%

T

90%A

Atmospheric pressure = 100 kPa

Response ForceResponse Force(Zwaan & Taylor, in prep.)(Zwaan & Taylor, in prep.)

Sensibility judgments.Sensibility judgments. Verbs.Verbs.

He pushed the car.He pushed the car. (high) (high) He started the car.He started the car. (low) (low) He admired the car.He admired the car. (no) (no)

Nouns.Nouns. He broke the baseball bat. He broke the baseball bat. (high)(high) He broke the bread. He broke the bread. (low)(low) He broke the promise. He broke the promise. (no)(no)

Response Force: VerbsResponse Force: Verbs

15000

16000

17000

18000

19000

20000

21000

Force

HighLowNo

ResponseAmplitude(kPa)

Response Force: NounsResponse Force: Nouns

15000

15500

16000

16500

17000

17500

18000

18500

19000

Force

HighLowNo

ResponseAmplitude(kPa)

Manner of ActionManner of Action

Sustain vs. ReleaseSustain vs. Release He gripped the hammer.He gripped the hammer. He dropped the hammerHe dropped the hammer..

PredictionsPredictions Sustain faster than Release (ACE).Sustain faster than Release (ACE). Sustain longer than Release (ACE).Sustain longer than Release (ACE).

Sustain vs. ReleaseSustain vs. Release

Response SpeedResponse Speed

1000

1100

1200

1300

1400

1500

1600

Sustain Release

SustainRelease

05000

100001500020000

2500030000350004000045000

Time

Force (kPa)

10%

Sustain vs. ReleaseSustain vs. Release

Response DurationResponse Duration

Sustain vs. ReleaseSustain vs. Release

Response Duration >10%Response Duration >10%

200

220

240

260

280

300

320

Sustain Release

SustainRelease

Motor ResonanceMotor Resonance

What is the role of motor What is the role of motor representations in language representations in language comprehension?comprehension? Instrumental Instrumental (Glenberg)(Glenberg) Ornamental Ornamental (many skeptics)(many skeptics) Useful/necessary in specific cases Useful/necessary in specific cases

(Jackendoff)(Jackendoff)

Motor ResonanceMotor Resonance

What is the role of motor What is the role of motor representations in language representations in language comprehension?comprehension? Useful/necessary when actions areUseful/necessary when actions are

ConcreteConcrete PunctatePunctate SimpleSimple

Let’s first ask: How is motor resonance Let’s first ask: How is motor resonance modulated by linguistic input?modulated by linguistic input?

How is motor resonance How is motor resonance modulated by linguistic modulated by linguistic

input?input? ((Zwaan & Taylor, 2006, JEP General)Zwaan & Taylor, 2006, JEP General)

Reading by rotating.Reading by rotating. Every 4 deg. new frame.Every 4 deg. new frame.

He /realized /that /the music /was He /realized /that /the music /was /too loud/so he //too loud/so he /turned downturned down/the /the /volume./volume.

60 psychology undergraduates.60 psychology undergraduates.

Reading-by-RotationReading-by-Rotation

300

310

320

330

340

350

360

370

380

He...he turned down the volumeSentence Region

Reading Time

MatchMismatch

Is Motor Resonance Is Motor Resonance

Short-lived or Focus-bound?Short-lived or Focus-bound? (Taylor & Zwaan, in prep.)(Taylor & Zwaan, in prep.)

Perspective changePerspective change.. After/lighting/the candles/for

the/romantic/evening,/he/dimmed/the/lights. Maintaining focus on the action.Maintaining focus on the action.

He lit/the candles/for the/romantic/evening./He noticed/the bright/lights,/which he/dimmed/slowly.

Reading-by-rotating.Reading-by-rotating.

Motor Resonance is Focus-Motor Resonance is Focus-boundbound

Strong Test Strong Test of Linguistic Focus of Linguistic Focus

HypothesisHypothesis Action-modifying vs. subject-Action-modifying vs. subject-

modifying adverbsmodifying adverbs … … which he turned down slowly.which he turned down slowly. ……which he turned down obediently.which he turned down obediently.

Stay tuned.Stay tuned.

Conclusions & OutlookConclusions & Outlook

Language is embedded in systems of Language is embedded in systems of perception and action.perception and action. Rapid exchange of information.Rapid exchange of information.

But it needs to be determined to But it needs to be determined to what extent it is in bed with these what extent it is in bed with these systems.systems. Perceptual & motor resonance Perceptual & motor resonance

ornamental or instrumental?ornamental or instrumental?

AcknowledgementsAcknowledgements Grants from the National Science Foundation Grants from the National Science Foundation

(NSF) and the National Institute of Mental (NSF) and the National Institute of Mental Health (NIMH)Health (NIMH)

Graduate students (past & present)Graduate students (past & present) Rob Stanfield, Carol Madden, Rich Yaxley, Mark Rob Stanfield, Carol Madden, Rich Yaxley, Mark

Aveyard, Larry TaylorAveyard, Larry Taylor Postdoctoral Fellows (past & present)Postdoctoral Fellows (past & present)

Barbara Kaup, Dave Therriault, Johanna Kaakinen, Barbara Kaup, Dave Therriault, Johanna Kaakinen, Tobias RichterTobias Richter

Faculty CollaboratorsFaculty Collaborators Mike Kaschak, Ralph Radach, Katinka Dijkstra Mike Kaschak, Ralph Radach, Katinka Dijkstra