Thinking with your body and other things

Thinking with your Body

and Other Things

David Kirsh

Cognitive Science

UCSD

HOW DO WE

MOVE THOUGHT FORWARD?

Hand drags the eye - pacer

How do we use our …

Eyes

Hands

Bodies

Tools

Nearby objects

To help us think?

Thinking with our eyes

Will this design work?

Which square becomes the cube’s top?

X D

C

B

A F

E

D is the top square

D D

Mental Simulation

D C

B A F

E

Some thinking involves

mental simulation while looking

Projection: another element of

visual thinking

See things that aren’t there

Move 11 to where 15 is

Initially visual search to locate 11 and 15

1 1

1 5

We project faint mental images onto external or internal visual anchors or indices (FINSTs)

11 to 15

How many moves can you reliably project?

1

1 1

Now 1 to 11

Visual search

Projection

Manipulation then Create change

Project

Project Create Interactive strategy

THESIS

Thinking often involves

making models or simulations

through ‘active’ perception,

projection and manipulation.

Thinking often involves

manipulating external

structures (as well as internal)

projections.

Agenda

Main Theoretical Claims

Thinking through the Body

Study of Dance Making

Thinking through Instruments

Action vs. Observation

Thinking through Ordinary Objects,

Bottles, Representations, …

Agenda

Main Theoretical Claims

Thinking through the Body

Study of Dance Making

Thinking through Instruments

Action vs. Observation

Thinking through Ordinary Objects,

Bottles, Representations, …

ONE: THINKING IS OFTEN

INTERACTIVE

Claim

How do we use our …

Eyes

Hands

Bodies

Tools

Nearby objects

To Think

Prove:

All three medians of a triangle

always intersect at a single point.

Can you do it just in your head?

Do the medians intersect at a single point?

Conceive then Imagine

Do the medians intersect at a single point?

Triangle

Medians Intersect

Imagination is fallible

All three medians intersect at a point.

Easier if you create an external representation

Conceive then

Externalize

Create 3rd median

Still too hard to project reliably so create

THINKING IS OFTEN INTERACTIVE

Claim One

PROJECT CREATE INTERACTIVE

STRATEGY

TWO: PROJECTION CAN BE IN

ANY MODALITY

Claim

Look and use your hands to feel its width

Kinesthetic projection

Look and use your hands to feel its width

Kinesthetic projection

Anchored on visually perceived object

She sells seashells by the seashore

Read this silently

Read this silently

She sells seashells by the seashore

Auditory projection onto words

WE PROJECT IN MANY

MODALITIES

Claim Two

KINESTHETIC, SOUND, VISION . . .

THREE: WE CREATE STRUCTURE

TO

PROJECT ONTO

IN MANY MODALITIES

Claim

Thinking through Gesture Mental Abacus

Source: Brooks et al, Gesture in Mental Abacus Calculation, 2012

Visual projection onto hand movement or

Kinesthetic projection onto hands

Numbers are encoded visually rather than as sounds or words

Thinking through Body Movement

Work done with Paul Maglio 1994

Where should this

piece go?

Thinking by

moving objects

Work done with Paul Maglio 1994

Physical Rotation

is faster than

Mental Rotation

Work done with Paul Maglio 1994

Physical Rotation

reveals different

shapes

We project the location

Work done with Paul Maglio 1994

Will these

Make this

Solving it in the world is more cost-effective

• More reliable

• Usually faster

• Less effortful

• Scales up better

Oops

WE CREATE STRUCTURE TO PROJECT

ONTO

IN MANY MODALITIES

Claim Three

HAND MOTION, SOUND, MOVING OBJECTS . . .

Interim Summary

We think interactively through

projection creation

Thinking often involves building models or

simulations

– to make sense of things or

– to think things through

Interim Summay

We think interactively through

projection creation

Thinking often involves building models or

simulations

– to make sense of things or

– to think things through

Interim Summary

This interactive strategy is multimodal

Gestures + Mental Imagery

(projection)

Body Movement + Mental Imagery “

Manipulate objects + Mental Imagery “

Writing + Mental Imagery

Drawing + Mental Imagery “

FOUR: INTERACTIVE THINKING REQUIRES A TIGHT

TEMPORAL COUPLING BETWEEN INTERNAL

AND EXTERNAL PROCESSES

Claim

Hand - ‘Mind’ Coordination

Tight Temporal Coupling

Only useful for moments

Kinesthetic imagery - projection

Thinking involves tight temporal coupling between components

Executes this

Thinking this helps shape performance

Dancer: Noah Gelber

Tight’ish temporal coupling

• ‘Read these words’

– inner speech is tightly coupled

• Write what you’re thinking

– to build on your writing interactively inner processes cannot

decouple from outer ones

• write read write

• Conversation – joint activity breaks down if delays are

extended

TIGHT TEMPORAL COUPLING BETWEEN

INTERNAL AND EXTERNAL

Claim Four

MENTAL IMAGERY MOVEMENT

FIVE: WHEN WE EXTERNALLY MODEL WE

MENTALLY PROJECT TO SOME ASPECT

OR

DIMENSION OF THE TARGET THING

Claim

When making a caricature you think of some aspect of the real thing

Models are simpler than the thing modeled

WE MENTALLY PROJECT TO SOME ASPECT

Claim Five

LOOK AT IMAGE SEE JOWLS, DETERMINATION

Agenda

✔ Main Theoretical Claims

Thinking through the Body

Study of Dance Making

Thinking through Instruments

Action vs. Observation

Thinking through Ordinary Objects, Bottles, Representations, …

Agenda

✔ Main Theoretical Claims

Thinking through the Body

Study of Dance Making

Thinking through Instruments

Action vs. Observation

Thinking through Ordinary Objects, Bottles, Representations, …

Case Study

Thinking with our body

Body Mediated Cognition

How we studied

dance making

Timeline – ‘Dyad’ 2009

Wayne McGregor

Jan 26th Feb 13th

UCSD

Aug 25th Sept 7th

London

Oct 13th

Premiere Total = 6 weeks

Random Dance

Timeline – ‘Far’ 2010

Sept 13th Nov 13th

London

Nov 17th

Premiere Total = 6 weeks

Timeline – ‘Undance’ 2011

Aug 30th Oct 23rd

London

Nov 30th

Premiere

Total = 6 weeks

Timeline – ‘Atomos’ 2013

July 21st Oct 8th

London

Oct 9th

Premiere

Total = 11 weeks

Observation Set-Up - UCSD

Rear2

Rear1

W Cam

Daily2

Daily1

Mandeville

Observers

Mandeville Auditorium Cal-IT2

Daily 2 W Cam (b)

Daily 1 (a)Rear 1 (b)

Rear 1 (c)Rear 2

Cal-IT2

W Cam (a)

ArtistEntrance

Rear 1 (a)

Daily 1 (b)Rear 1 (a)

Student'sEntrance

Dyad UCSD

Field Notes Observations 5 hours/day 6 -11 weeks

Time, Activity code, Description of activity

Interviews Before and after session

2 hours per day, for 23 days

Interviews

After session each day usually in two’s

Data Collected at Each Capture

• 20 TB of video of dancers and choreographer

• Dozens of interviews with choreographer and

dancers

• Still images

• Dancer notes

• Associate choreographer’s notes

• Student notes of ongoing activity

• Music used

MARKING in dance

Thinking with the body

Marking - during practice

What is Marking?

A dance phrase is practiced, explored or reviewed in a less

energetic manner than doing it ‘full-out’.

Small marking

Marking for time

Marking

• Imperfect model of real phrase

• a sketch, abstraction

• Attend to specific aspect of a movement

• Dancers often practice real phrases by marking

Marking: a universal phenomenon

• Tennis swing – work on aspect

• Cello – on the arm

• Staged Plays – an Italian run-through

• Imperfect modeling – aspectual – as a

learning/practice technique

Teachers have students swing without hitting a ball in order to grasp shapes or make small adjustments

Not even a tennis racket

Planning grips and placements in rock

climbing by marking when on the ground

Similar Phenomena

Planning a downhill run when skiing moguls

Similar Phenomena

Antoine: Full

Antoine: Large marking

Antoine: Small marking

Agnes: Full vs. Marked

I’m not using energy but I’m thinking what’s my pathway

Agnes: Large vs. Small marking

Tombe Pas de Bouree

Experiment to explore value of marking

Which condition facilitate learning most?

Mark Full-Out Simulate in the head

Hypothesis

Mark

Full-Out

Simulate in the head

>

>

Experimental Design

MEASURE MEASURE PRACTICE TEACH

Experimental Design

Mark

Lie on floor

Full-Out

Trial 2

Trial 3

Trial 1

Mark

Lie on floor

Full-Out Mark

Lie on floor

Full-Out

Group 1 Group 2 Group 3

Performance Measures

1 2 3 4 Technicality Memory Dynamics

Precision of positions Completeness of detail Speed, Force, Acceleration

Timing

Tempo, duration

RESULTS

Higher is better

P = .018

Marking was best!

Measure Mark >Full Full>Mark Mark>Sim Full>Sim

Memory .7334 < .0001 < .0001

Technicality .0029 < .0001 .0005

Timing .0194 < .0001 < .0001

Dynamics - .145 .0003 < .0001

Mem, Tech, Timing .0189 - < .0001 < .0001

Higher is better

P Values

Results by Measured Dimension

Upshot: Marking Study

• Marking is a movement reduction

system

• Other movement reduction systems that

might facilitate thought:

• Whispering or Subvocalizing

• Gesturing

• Physical miming with objects

How Does It Work?

Marked aspect anchors

projection

Marked phrase

‘Imagery”

In world

In mind

Movement

Projection in mind

PROJECTION

Projection vs. Imagination

Reality oriented

See what is present

Perception

Augment reality

Anchored Imagery

Virtual Reality

Imagery has no size or location

Projection Imagination

Projection: Claims

1. Mental Projection is more powerful than

mental imagery alone

2. We can project beyond what we can

readily imagine

3. External structure helps us

Experiment to explore projection

Tic Tac Toe

Blank Sheet

Imagination Condition

Projection Condition

Projection Condition - 2

X O

Tic Tac Toe experiment – 3 by 3

Projection conditions

Blank Sheet

Imagination Condition

Blank Table Table + X O

Experimental Conditions

X O

Training

1 2 3

4 5 6

7 8 9

Blank Sheet

Within Subject Design, with Practice Phase

Imagination Condition Practice

Blank

X O

Projection conditions

Table Table + X O

Projection ≠ Memory Offload of State

Board remains the same over time

Memory not offloaded because no persistent structure

X

O

4,8 4,9

4,6

4,0

4,5

5,0

5,5

6,0

Blank Table Table + XO

Overall Means

Not significant N = 25

Secs

X O

Lower is better

Results 3 by 3

4.9s ≈ 4.8s

Surprise!

≈

Table is no faster than blank

Results 3 by 3: Individual Differences

4,4

5,8

4,7

4,0

4,5

5,0

5,5

6,0

Blank Table Table + XO

Significant p = .002

X O More than half were significantly better using

Imagination alone

Secs

X

Anchoring Costs

Anchoring = Spatializing a component of mental image

X Placement costs

Do benefits ever overcome

anchoring costs?

If the imagery task is hard

enough everyone will benefit

Conjecture:

4 by 4 Board

1 2 3 4

5 6 7 8

9 10 11 12

13 14 15 16

4 by 4 Experiment: harder imagery task

Imagination Projection

Practice

Blank Table Table + X O

3 Conditions

Blank Page

X O

1 2 3 4

5 6 7 8

9 10 11 12

13 14 15 16

Results 4 by 4

Table is worth the cost of coordination

If task is hard enough

Table faster than XO p=.01

N = 25

4 by 4 Mean Time per move

p = .002

> p = .002

9.1s 10.7s

Is it better for everyone?

Bad visualizers are helped much more!

Strong grid > blank p = .11

Weak grid > blank p = .01

*

Lower is better * Weak 10.4 < 14.1 Strong 9.3 < 10.1

>

Strong trending to benefit

Blank XO Grid Blank XO Grid Blank XO Grid

Upshot: Projection

• Projection: real process

– distinct from perception and imagination

• As problems get harder we cannot easily

imagine the answer

• So we rely on creating external structures

to project onto and scaffold imagination

SIX: IF WE CAN THINK WITH OUR BODIES

THEN WE CAN THINK WITH THINGS

THAT WE ARE HIGHLY PRACTICED

WITH

Claim

Agenda

1. Thinking with the Body

Study of Dance Making

Thinking with Tools

Action vs. Observation

Thinking with Ordinary Objects, Representations …

Conclusions

Thesis

Eyes

Hands

Bodies

Tools

Nearby objects

To help us think

We Use our

Players can Mark with their Instrument

As if it is their body

So ... Experts can think with

their

instruments

Can they have the same

thoughts without?

Why an expert might have the

same thought without playing

Mirror Neurons!

Super Experts

• Play enough, you eventually simulate playing without an

instrument in hand - i.e. internalization

– Instrument-mediated cognition

is possible without the instrument

• Perlman’s auditory perception

is so expert when he listens

it is as if he is physically playing

Neural support that observing is

enough

• Motor resonance, mirror neuron, action observation network

– Watching or listening stimulates covert actions as if dancing or playing

• Covert actions involve motor planning, just as overt action does

• The real difference is that covert action does not activate muscular control

• So experts simulate playing

Support from Enactive Perception

• Enactive account: observer sees the counterfactual futures in the present – what it would look like if I moved, turned it, played it …

• Observer’s phenomenological experience includes possible ways a phrase may be continued

So in auditory perception agents

enact a possible future

YET

Do mirror neurons simulate at a fine

enough resolution?

empirical question … but

Listening is not Performing

physicality of the piece

Performer Situation ≠ Observer

Situation • Performer has responsibility to succeed

• observer doesn’t

• Performer must decide: how to attack a note, its mood,

emotionality

• These concepts are ad hoc, situated, and embodied

• unavailable even to an an expert observer.

• Hence agents project a future that is conceptually and

experientially richer than the future projected by an

observer

• Simulation is a lower resolution than actual playing

With a violin in hand musical engagement is

mediated differently than in imagination and

active listening

Instrument mediated conclusion

Agenda

1. Thinking with the Body

Study of Dance Making

Thinking with Instruments

Action vs. Observation

Thinking with Ordinary Objects, Representations …

Conclusions

Thesis

Eyes

Hands

Bodies

Tools

Nearby objects

To help us think

We Use our

Thinking with Illustrations by deforming

them on a stretchy surface

D’Arcy Thompson’s famous fish geometry

Distorting the plane equals distorting growth rates – explains different forms

Stretching, twisting, shearing

Remember this …?

Solving it in the world is more cost-effective

• More reliable

• Usually faster

• Less effortful

• Scales up better

Cognition flows to wherever costs

are lower

Internal Processes

External Processes

In closely coupled system, process and structure migrates

to wherever computational and physical costs are lowest

External Processes

Physical is cheaper

Here processes flow to the physical

Thinking by interacting with 3D models

We use the model as tool for visualization – focusing attention on things that suddenly make sense

Driving thought forward

Objects in motion can do work for us

We can think causally with objects

If we interact we can:

• Compare rotation speeds

• Try out speeds

• Get a feel for force

• Visualize effects

Agenda

1. Thinking with the Body

Study of Dance Making

Thinking with Instruments

Action vs. Observation

Thinking with Ordinary Objects, Representations …

Conclusions

THINKING IS MULTI-MODAL

Bodies & Things

Eyes & Hands

Bodies & Kinesthetics

HIGHLY PERCEPTUAL Visual Probing

Kinesthetic Probing

USUALLY INVOLVES PROJECTION

INSIDE AND OUTSIDE

PROCESSES ARE TIGHTLY

COUPLED

OFTEN INVOLVES PHYSICAL OR

MENTAL SIMULATION OR

CONSTRUCTION

SIMULATION AND PROJECTION ARE

HARD SO INTERACTING IS USUALLY

PART OF THINKING

Eye hand mediated cognition

Body mediated cognition

Tool mediated cognition

External Rep’n mediated Cognition

Object mediated Cognition

WITH THE RIGHT STRUCTURES AND

MOVEMENTS WE CAN PUSH

THOUGHT FORWARD FASTER

158 slides

Thank you for concentrating!

Acknowledgements

Thanks to

Dafne Muntanyola

Wayne McGregor

Odette Hughes

Random Dance Company

Funding:

NSF grant: Distributed Creative Cognition in Dance

UCSD core grant

R-Research, London

UCSD classes

The End

Enough