107
Probabilistic mechanisms in human sensorimotor control Daniel Wolpert, University College London movement is the only way we have of Interacting with the world Communication: speech, gestures, writing sensory, memory and cognitive processes future motor outputs Q. Why do we have a brain? Sea Squirt A. To produce adaptable and complex movements

Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Embed Size (px)

Citation preview

Page 1: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Probabilistic mechanisms in human sensorimotor controlDaniel Wolpert, University College London

• movement is the only way we have of– Interacting with the world – Communication: speech, gestures, writing

• sensory, memory and cognitive processes future motor outputs

Q. Why do we have a brain?

Sea Squirt

A. To produce adaptable and complex movements

Page 2: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

1000

1500

2000

2500

3000

3500

Why study computational sensorimotor control?

Year

Pag

es r2=0.96

1980 1990 2000 2010 2020 2030 2040 2050500

Experiments

Theory

Principles of Neural Science, Kandel et al.

Page 3: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

vs.

What to move where

vs.

Moving

The complexity of motor control

Page 4: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Noise makes motor control hard

Noise = randomness

The motor system is Noisy

Perceptual noise– Limits resolution

Motor Noise– Limits control

NoisyPartial

Noisy

AmbiguousVariable

Page 5: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

David Marr’s levels of understanding (1982)

1) the level of computational theory of the system

2) the level of algorithm and representation, which are used make computations

3) the level of implementation: the underlying hardware or "machinery" on which the computations are carried out.

Page 6: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Tutorial Outline– Sensorimotor integration

• Static multi-sensory integration• Bayesian integration• Dynamic sensor fusion & the Kalman filter

– Action evaluation• Intrinsic loss function• Extrinsic loss functions

– Prediction• Internal model and likelihood estimation • Sensory filtering

– Control• Optimal feed forward control• Optimal feedback control

– Motor learning of predictable and stochastic environments

Review papers on www.wolpertlab.com

Page 7: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Multi-sensory integration

Multiple modalities can provide information about the same quantity• e.g. location of hand in space

– Vision– Proprioception

• Sensory input can be– Ambiguous– Noisy

• What are the computations used in integrating these sources?

Page 8: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Ideal Observers

Maximum likelihood estimation (MLE)

i ix x ε= +

1 2( , ..., | )nP x x x x1

( | )n

ii

P x x=

=∏

2(0, )i iNε σ=

( )2

211

ˆ withn

ii i i n

i jj

x w x w σ

σ

−=

=

= =∑∑

( ) 12 2 2ˆ 1

nx i kj

kσ σ σ−

−=

= < ∀∑

2x

1x

x

Consider signals , {1 }in x i n= …

Two examples of multi-sensory integration

Page 9: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Visual-haptic integration (Ernst & Banks 2002)

Two alternative force choice size judgment• Visual• Haptic• Visual-haptic (with discrepancy)

Page 10: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Visual-haptic integration

Measure•Visual reliability •Haptic reliability•Predict

•Visual + Haptic noise•Weighting of

2Vσ2Hσ

2 Hσ

Prob

abili

ty

Size

Prob

abili

ty

Size difference

2 Hσ

Hσ Hσ

Page 11: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Visual-haptic integration

( )2

21

2

2 2 1

ii n

jj

VH V H

V H

w

w w w

σ

σ

σσ σ

−=

=

= = −+

∑( )

2 212 2ˆ 2 21

n V Hx ij

V H

σ σσ σσ σ

−−

== =

+∑

Weights Standard deviation (~threshold)

Optimal integration of vision and haptic information in size judgement

Page 12: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Visual-proprioceptive integrationClassical claim from prism adaptation

“vision dominates proprioception”

Page 13: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Reliability of proprioception depends on location

(Van Beers, 1998)

Reliability of visual localization is anisotropic

Page 14: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Integration models with discrepancyWinner takes all

Linear weighting of mean

Optimal integration

ˆ (1 )V Hw w= + −x x x

ˆ V HA B= +x x x

1 1 1

1 1

( )

( )PV P V

PV PV P P V Vµ µ µ

− − −

− −

Σ = Σ +Σ

= Σ Σ +Σ

Page 15: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Prisms displace along the azimuth•Measure V and P•Apply visuomotor discrepancy during right hand reach•Measure change in V and P to get relative adaptation

Vision 0.33Prop 0.67

(Van Beers, Wolpert & Haggard, 2002)

Page 16: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Visual-proprioceptive discrepancy in depth

AdaptationVision 0.72Prop 0.28

Visual adaptation in depth > visual adaptation in azimuth (p<0.01)> Proprioceptive adaptation in depth (p<0.05)

Proprioception dominates vision in depth

Page 17: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Priors and Reverend Thomas Bayes

“I now send you an essay which I have found among the papers of our deceased friend Mr Bayes, and which, in my opinion, has great merit....”

Essay towards solving a problem in the doctrine of chances. Philosophical Transactions of the Royal Society of London, 1764.

1702-1761

Page 18: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Bayes rule

A and B

( , )P A B = ( | )P A B ( | ) ( )P B A P A=

Belief instate sensory inpuAFTER t

Posterior

( )P B

P(state|sensory input)

Prior

BEFOREBelief in state sensory input

NeuroscienceA= State of the world B=Sensory Input

A = Disease B = Positive blood test

Likelihood

Evidence

P (sensoryinput|state) P(state)=P(sensory input)

Agiven B

Page 19: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Bayesian Motor Learning

= Optimal estimate (Posterior)

Bayes rule

+ Task statistics (Prior)Not all locations are

equally likely

Sensory feedback (Evidence)Combine multiple cuesto reduce uncertainty

Estimate

Evidence

Prior

P (sensorP(state| y input|ssensory tate Pinput) (st )) ate∝

Real world tasks have variability, e.g. estimating ball’s bounce location

Does sensorimotor learning use Bayes rule?If so, is it implemented• Implicitly: mapping sensory inputs to motor outputs to minimize error?• Explicitly: using separate representations of the statistics of the prior and sensory noise?

Page 20: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

(Körding & Wolpert, Nature, 2004)

Prior

Lateral shift (cm)

Prob

abili

ty

0 1 2

Task in which we control 1) prior statistics of the task 2) sensory uncertainty

Page 21: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

(Körding & Wolpert, Nature, 2004)

Prior

Lateral shift (cm)

Prob

abili

ty

0 1 2

Task in which we control 1) prior statistics of the task 2) sensory uncertainty

Page 22: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Sensory FeedbackLikelihood

Generalization

Learning

Page 23: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

After 1000 trials

2 cm shift

1cm

No visual feedback

Prob

abili

ty

0 1 2

Lateral shift (cm)

Page 24: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

0 1 2

Bayesian Compensation

Lateral shift (cm)

Models

Full Compensation

Lateral shift (cm)0 1 2

0 1 2Lateral Shift (cm)

Ave

rage

Err

or

Bia

s (c

m) 1

0

-1 0 1 2Lateral Shift (cm)

Ave

rage

Err

or

Bia

s (c

m) 1

0

-1

Mapping

0 1 2Lateral shift (cm)

0 1 2Lateral Shift (cm)

Ave

rage

Err

or

Bia

s (c

m) 1

0

-1

Page 25: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Supports model 2: Bayesian

Results: single subject0

Full

Bayes

Map

0 1 2Lateral Shift (cm)

Ave

rage

Err

or

Bia

s (c

m)

Page 26: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Supports model 2: Bayesian

Results: 10 subjects0

Full

Bayes

Map

0 1 2Lateral Shift (cm)

Ave

rage

Err

or

Bia

s (c

m)

-0.5 1 2.5

1

0

lateral shift [cm]Lateral shift (cm)

Infe

rred

Prio

r(n

orm

aliz

ed)

Page 27: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Bayesian integrationSubjects can learn

• multimodal priors• priors over forces• different priors one after the other

(Körding& Wolpert NIPS 2004, Körding, Ku & Wolpert J. Neurophysiol. 2004)

Page 28: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Statistics of the world shape our brain

Objects Configurations of our body

• Statistics of visual/auditory stimuli representation visual/auditory cortex• Statistics of early experience what can be perceived in later life

(e.g. statistics of spoken language)

Page 29: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Statistics of action

• 4 x 6-DOF electromagnetic sensors• battery & notebook PC

With limited neural resources statistics of motor tasks motor performance

Page 30: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Phase relationships and symmetry bias

Page 31: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Multi-sensory integration

• CNS – In general the relative weightings of the senses is

sensitive to their direction dependent variability– Represents the distribution of tasks– Estimates its own sensory uncertainty– Combines these two sources in a Bayesian way

• Supports an optimal integration model

Page 32: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Loss Functions in Sensorimotor system

What is the performance criteria (loss, cost, utility, reward)?

• Often assumed in statistics & machine learning – that we wish to minimize squared error for analytic or algorithmic tractability

• What measure of error does the brain care about?

Target Position

Post

erio

rPr

obab

ility

PriorLikePosterior lihood

P(sensory iP (state|sensory inpu nput|state) P(statet) )∝

[ ] ( , ) ( | _ ) _

ˆ ( ) arg min [ ]B actions

E Loss Loss state action P state sensory input dsensory input

Bayes estimator x s E Loss

=

=∫

Page 33: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Loss function

312 2

Target

Scenario 1 Scenario 2

( )f error

2Loss error=

12Loss error=

Loss error=

Loss=4+4=8 Loss=1+9=10

Loss=2+2=4 Loss=1+3=4

Loss=1+1.7=2.7Loss=1.4+1.4=2.8

Page 34: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Virtual pea shooter

MeanStarting location

Position (cm)Prob

abili

ty

-0.2 0 0.2

(Körding & Wolpert, PNAS, 2004)

Page 35: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Probed distributions and optimal means

Possible Loss functions

2Loss ( )error=MEAN

-2 -1 0 1 2Error (cm)

ρ=0.2

ρ=0.3

ρ=0.5

ρ=0.8

Distributions

MODEMaximize Hits

-2 -1 0 1 2Error (cm)

MEDIANLoss error=

Robust estimator

Page 36: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Shift of mean against asymmetry (n=8)

Mean squared error with robustness to outliers

Page 37: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

0.1 1.0 1.3 1.5 1.7 1.9 2.1 2.3 2.5 2.7 2.9 3.1 3.3 3.5 3.7 3.9α=

Loss ierrorα=∑

Personalised loss function

Page 38: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Bayesian decision theory

Increasing probability of avoiding keeper

Increasing probability of being within the net

Page 39: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Imposed loss function (Trommershäuser et al 2003)

0 -100 -500+100+100 +100

Page 40: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Optimal performance with complex regions

Page 41: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

State estimation• State of the body/world

– Set of time-varying parameters which together with• Dynamic equations of motion• Fixed parameters of the system (e.g. mass)

– Allow prediction of the future behaviour

• Tennis ball– Position– Velocity– Spin

Page 42: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

State estimation

NOISE NOISE

Observer

Page 43: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Kalman filter• Minimum variance estimator

– Estimate time-varying state– Can’t directly observe state but only measurement

1t t t tA B+ = + +x x u w

1t t tC+ = +y x v

1ˆ ˆ ˆ[ ]t t t t t tA B K C+ = + + −x x u y x

Page 44: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

State estimation

1

Forward DynamicModel

ˆ ˆ ˆ[ ]t t t t t tA B K C+ = + + −x x u y x

Page 45: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Kalman Filter

• Optimal state estimation is a mixture– Predictive estimation (FF)– Sensory feedback (FB)

Page 46: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Eye position

Location of object based on retinal location and gaze direction

PerceptActual

Motor command FM Eye

Position

Page 47: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Sensory likelihood

(Wolpert & Kawato, Neural Networks 1998Haruno, Wolpert, Kawato, Neural Computation 2001)

P (sensorP(state| y input|ssensory tateinput) P(st )) ate∝

Page 48: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Sensory predictionOur sensors report• Afferent information: changes in outside world• Re-afferent information: changes we cause

+ =

Internal source

Externalsource

Page 49: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

TicklingSelf-administered tactile stimuli rated as less ticklish than externally administered tactile stimuli. (Weiskrantz et al, 1971)

Page 50: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Does prediction underlie tactile cancellation in tickle?

0

0.5

1

1.5

2

2.5

3

Self-producedtactile stimuli

Robot-producedtactile stimuli

Condition

Tic

kle

ratin

g ra

nk

Gain control or precise spatio-temporal prediction?

P<0.001

Page 51: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Spatio-temporal prediction

0

0.5

1

1.5

2

2.5

3

3.5

4

Delay 0ms Delay100ms

Delay200ms

Delay300ms

Robot-produced

tactilestimuliCondition

Tic

kle

ratin

g ra

nk

P<0.001

P<0.001

0

0.5

1

1.5

2

2.5

3

3.5

4

0 degrees 30 degrees 60 degrees 90 degrees External

Condition

Tic

kle

ratin

g ra

nk

P<0.001P<0.001

(Blakemore, Frith & Wolpert. J. Cog. Neurosci. 1999)

Page 52: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

The escalation of force

Page 53: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Tit-for-tat

Force escalates under rules designed to achieve parity: Increase by ~40% per turn

(Shergill, Bays, Frith & Wolpert, Science, 2003)

Page 54: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Perception of force

70% overestimate in force

Page 55: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Perception of force

Page 56: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Labeling of movements

Large sensorydiscrepancy

Page 57: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Defective prediction in patients with schizophrenic

• The CNS predicts sensory consequences

• Sensory cancellation in Force production

• Defects may be related to delusions of control

Patients Controls

Page 58: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Motor LearningRequired if:

• organisms environment, body or task change• changes are unpredictable so cannot be pre-specified • want to master social convention skills e.g writing

Trade off between: – innate behaviour (evolution)

• hard wired• fast• resistant to change

– learning (intra-life)• adaptable• slow• Maleable

Page 59: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Motor Learning

Actual behaviour

Page 60: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Predicted outcome can be compared to actual outcome to generate an error

Supervised learning is good for forward models

Page 61: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Weakly electric fish (Bell 2001)Produce electric pulses to • recognize objects in the dark or in murky habitats• for social communication.

The fish electric organ is composed of electrocytes, • modified muscle cells producing action potentials • EOD = electric organ discharges• Amplitude of the signal is between 30 mV and 7V • Driven by a pacemaker in medulla, which triggers each discharge

Page 62: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Sensory filteringSkin receptors are derived from the lateral line system

Removal of expected or predicted sensory input is one of the very general functions of sensory processing. Predictive/associative mechanisms for changing environments

Page 63: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Primary afferent terminate in cerebellar-like structures

Primary afferents terminate on principal cells either directly or via interneurons

Page 64: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Block EOD discharge with curare

Specific for Timing (120ms), Polarity, Amplitude & Spatial distribution

Page 65: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Proprioceptive Prediction

Tail bend affects feedbackPassive Bend phase locked to stimulus:

Bend

Page 66: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Learning ruleChanges in synaptic strength requires principal cell spike discharge

Change depends on timing of EPSP to spike

Anti-Hebbian learning

T1

T2

T2-T1

• Forward Model can be learned through self-supervised learning• Anti-hebbian rule in Cerebellar like structure of he electric fish

Page 67: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Motor planning (what is the goal of motor control)

Duration Hand Trajectory

Joint Muscles

• Tasks are usually specified at a symbolic level• Motor system works at a detailed level, specifying muscle activations• Gap between high and low-level specification• Any high level task can be achieved in infinitely many low-level ways

Page 68: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Eye-saccades Arm- movements

Motor evolution/learning results in stereotypyStereotypy between repetitions and individuals

Time (ms)

• Main sequence• Donder’s law• Listings Law

• 2/3 power law• Fitts’ law

Page 69: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Models

HOW models– Neurophysiological or black box models– Explain roles of brain areas/processing units in

generating behaviorWHY models

– Why did the How system get to be the way it is? – Unifying principles of movement production

• Evolutionary/Learning– Assume few neural constraints

Page 70: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

The Assumption of OptimalityMovements have evolved to maximize fitness

– improve through evolution/learning– every possible movement which can achieve a task has a cost– we select movement with the lowest cost

Overall cost = cost1 + cost2 + cost3 ….

Page 71: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Optimality principles

• Parsimonious performance criteriaElaborate predictions

• Requires– Admissible control laws– Musculoskeletal & world model– Scalar quantitative definition of task

performance – usually time integral of f(state, action)

Page 72: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Open-loop

• What is the cost– Occasionally task specifies cost

• Jump as high as possible• Exert maximal force

– Usually task does not specify the cost directly• Locomotion well modelled by energy minimization• Energy alone is not good for eyes or arms

Page 73: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

What is the cost?

Saccadic eye movements

• little vision over 4 deg/sec• frequent 2-3 /sec• deprives us of vision for 90

minutes/day

⇒Minimize time

Page 74: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Arm movementsMovements are smooth – Minimum jerk (rate of change of acceleration) of the hand

(Flash & Hogan 1985)

2 2

04 5 3

0 04 5 3

0 0

( ) ( )

( ) ( )(15 6 10 )

( ) ( )(15 6 10 )/

T

T

T

Cost x t y t dt

x t x x x

y t y y yt T

τ τ τ

τ τ ττ

= +

= + − − −

= + − − −=

Page 75: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Smoothness• Minimum Torque change (Uno et al, 1989)

Shouldertorque

Elbowtorque2 2

0( ) ( )

T

s eCost t t dtτ τ= +∫

Page 76: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

The ideal cost for goal-directed movement

• Makes sense - some evolutionary/learning advantage• Simple for CNS to measure• Generalizes to different systems

– e.g. eye, head, arm

• Generalizes to different tasks – e.g. pointing, grasping, drawing

→ Reproduces & predicts behavior

Page 77: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Motor command noise

MotorSystem

Noise

Position

Error minimized byrapidity

Page 78: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Fundamental constraint=Signal-dependent noise

• Signal-dependent noise: – Constant coefficient of variation– SD (motor command) ~ Mean (motor command)

• Evidence from– Experiments: SD (Force) ~ Mean (Force)– Modelling

• Spikes drawn from a renewal process• Recruitment properties of motor units

(Jones, Hamilton & Wolpert , J. Neurophysiol., 2002)

Page 79: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Task optimization in the presence of SDN

Given SDN, Task Given SDN, Task ≡≡ optimizing optimizing f(statisticsf(statistics))

An average motor command ⇒ probability distribution (statistics) of movement.

Controlling the statistics of actionControlling the statistics of action

Page 80: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Finding optimal trajectories for linear systems

2 2 2( ) ( )u t k u tσ =

time

TM

x(t)0

[ ( )] ( ) ( )M

E x M u p M d Aτ τ τ= − =∫

2

0 0

2 2 2 2

0 0

[ ( )] [ ( ) ( )] [ ( )] ( )

( ) ( ) ( ) ( )

M M

M M

Var x T Var u p T d Var u p T d

k u p T d w u d

τ τ τ τ τ τ

τ τ τ τ τ τ

= − = −

= − =

∫ ∫∫ ∫

time

p(t)

Impulse responseSignal-dependent noise

( ) ( )

0[ ( )] ( ) ( ) 0

Mn nE x M u p M dτ τ τ= − =∫

Cost

Constraints

System

A

Linear constraints with quadratic cost:can use quadratic programming or isoperimetric optimization

Page 81: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Saccade predictions

SDN

Motor command

Jerk

3rd order linear system

Page 82: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

0 100 2000

20

Time (ms)

Deg

rees

Prediction: very slow saccade

22 degree saccade in 270 ms (normally ~ 70 ms)

Page 83: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Head free saccade

0 100 200 300 400 500 600 700 8000

20

40

60

80

100

120

Deg

rees

Time (ms)

Head

Gaze

Eye

Τ1=0.3 Τ2=0.3

Τ1=0.15 Τ2=0.08 Free parameter, eye:head noise

0 100 200 300 400 500 600 700 8000

20

40

60

80

100

120

Time (ms)

Deg

rees

Eye

Head

Gaze

(Tomlinson & Bahra, 1986)

Page 84: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Coordination: Head and eyeFor a fixed duration (T), Var(A)=k A2

Var(A)=k A2 Var(A)= k (A/2)2 + k (A/2)2

= k A2 /2Var(A)=k A2

Eye only Head onlyEye &Head

Page 85: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Movement extent vs. target eccentricity

Gaze amplitude

Ang

ular

dev

iatio

n at

ac

quis

ition

Eye

Head

Gaze amplitude

0 20 40 60 80 100 120 1400

10

20

30

40

50

60

70

80

90

100

0 100 200 300 400 500 600 700 8000

20

40

60

80

100

120

} }Head

Eye

Page 86: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Arm movements

Drawing (⅔ power law) f=path errorObstacle avoidance f= limit probability of collision

Smoothness

Non smooth movement ⇒ requires abrupt change in velocity⇒ given low pass system⇒ large motor command⇒ increased noise

Smoothness ⇒ accuracy

Feedforward control•Ignores role of feedback•Generates desired movements•Cannot model trial-to-trial variability

Page 87: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Optimal feedback control (Todorov 2004)

– Optimize performance over all possible feedback control laws– Treats feedback law as fully programmable

• command=f(state)• Models based on reinforcement learning optimal cost-to-go functions• Requires a Bayesian state estimator

Page 88: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Minimal intervention principle

• Do not correct deviations from average behaviour unless they affect task performance– Acting is expensive

• energetically• noise

– Leads to • uncontrolled manifold• synergies

Uncontrolled manifold

Page 89: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Optimal control with SDN• Biologically plausible theoretical underpinning

for both eye, head, arm movements

• No need to construct highly derived signals to estimate the cost of the movement

• Controlling statistics in the presence of noise

Page 90: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

What is being adapted?

• Possible to break down the control process:

• Visuomotor rearrangements• Dynamic perturbations• [timing, coordination , sequencing]

• Internal models captures the relationship between sensory and motor variables

Page 91: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Altering dynamics

Page 92: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Altering Kinematics

Page 93: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Representation of transformations

Look-upTable

Non-physicalParametersθ=f(x,ω)

PhysicalParametersθ=acos(x/L)

High storageHigh flexibility

Low Generalization

Low storageLow flexibility

High Generalization

θ

x

L ÷asin

x

θ

L

x

θ

x θ1 103 35. .. .

Page 94: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Generalization paradigm

• Baseline– Assess performance over

domain of interest – (e.g. workspace)

• Exposure– Perturbation: New task– Limitation: Limit the

exposure to a subdomain

• Test– Re-assess performance

over entire domain of interest

Page 95: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Difficulty of learning

(Cunningham 1989, JEPP-HPP)

• Rotations of the visual field from 0—180 degrees

Difficulty• increases from 0 to 90• decreases from 120 to 180

• What is the natural parameterization

Page 96: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Viscous curl field

(Shadmehr & Mussa-Ivaldi 1994, J. Neurosci.)

Page 97: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Representation from generalization: Dynamic

1. Test: Movements over entire workspace

2. Learning– Right-hand workspace – Viscous field

3. Test: Movements over left workspace

Two possible interpretations force = f(hand velocity) or torque=f(joint velocity)

Joint-based learning of dynamics

(Shadmehr & Mussa-Ivaldi 1994, J. Neurosci.)

Left hand workspaceBefore After with Cartesian field

Page 98: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Visuomotor coordinates

z

yx

(x,y,z)

Cartesian

z

y

x

Spherical Polar

θφ

r(r,φ,θ)

α1

α2

α3

(α1,α2,α3)

Joint angles

Page 99: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Representation- Visuomotor1. Test: Pointing accuracy to a set of targets

2. Learning– visuomotor remapping– feedback only at one target

3. Test: Pointing accuracy to a set of targets

-50

-40

-30

y = 22.2/16.2 cm y = 29.2/26.2 cm y = 36.2 cm

-20 -10 0 10 20

20

30

40z = -43.6 cm z = -35.6 cm

-20 -10 0 10 20

z = -27.6 cm

-20 -10 0 10 20

x (cm)

Prediction of eye-centred spherical coordinates

(Vetter et al, J. Neurophys, 1999)

Predictions of eye-centred spherical coordinates

• Generalization paradigms can be used to assess– Extent of generalization– Coordinate system of transformations

Page 100: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Altering dynamics: Viscous curl fieldBefore Early with force

Late with force Removal of force

Page 101: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

A muscle activation levels sets the spring constant k (or resting length) of the muscle

Stiffness control

Equilibrium point

Page 102: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Equilibrium point control

• Set of muscle activations (k1,k2,k3…) defines a posture • CNS learns a spatial mapping

– e.g. hand positions muscle activations(x,y,z) (k1,k2,k3…)

Page 103: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Equilibrium control

The hand stiffness can vary with muscle activation levels.

Low stiffness High stiffness

Page 104: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Controlling stiffness

Burdet et al (Nature, 2002)

Page 105: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Stiffness ellipses

• Internal models to learn stable tasks• Stiffness for unpredictable tasks

Page 106: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

Summary– Sensorimotor integration

• Static multi-sensory integration• Bayesian integration• Dynamic sensor fusion & the Kalman filter

– Action evaluation• Intrinsic loss function• Extrinsic loss functions

– Prediction• Internal model and likelihood estimation • Sensory filtering

– Control• Optimal feed forward control• Optimal feedback control

– Motor learning of predictable and stochastic environments

Wolpert-lab papers on www.wolpertlab.com

Page 107: Q. Why do we have a brain? A. To produce adaptable and ...media.nips.cc/Conferences/2004/Tutorials/slides/wolpertSlides.pdf · – Communication: speech, ... 1980 1990 2000 2010 2020

References• Bell, c.(2001) Memory-based expectations in electrosensory systemsCurrent Opinion in

Neurobiology 2001, 11:481–487• Burdet, E., R. Osu, et al. (2001). "The central nervous system stabilizes unstable dynamics by

learning optimal impedance." Nature 414(6862): 446-9.• Cunningham, H. A. (1989). "Aiming error under transformed spatial maps suggest a structure for

visual-motor maps." J. Exp. Psychol. 15:3: 493-506.• Ernst, M. O. and M. S. Banks (2002). "Humans integrate visual and haptic information in a

statistically optimal fashion." Nature 415(6870): 429-33.• Flash, T. and N. Hogan (1985). "The co-ordination of arm movements: An experimentally

confirmed mathematical model " J. Neurosci. 5: 1688-1703.• Shadmehr, R. and F. Mussa-Ivaldi (1994 ). "Adaptive representation of dynamics during learning

of a motor task." J. Neurosci. 14:5: 3208-3224.• Todorov, E. (2004). "Optimality principles in sensorimotor control." Nat Neurosci 7(9): 907-15.• Trommershauser, J., L. T. Maloney, et al. (2003). "Statistical decision theory and the selection of

rapid, goal-directed movements." J Opt Soc Am A Opt Image Sci Vis 20(7): 1419-33.• Uno, Y., M. Kawato, et al. (1989). "Formation and control of optimal trajectories in human

multijoint arm movements: Minimum torque-change model " Biological Cybernetics 61: 89-101.• van Beers, R. J., A. C. Sittig, et al. (1998). "The precision of proprioceptive position sense." Exp

Brain Res 122(4): 367-77.• Weiskrantz, L., J. Elliott, et al. (1971). "Preliminary observations on tickling oneself." Nature

230(5296): 598-9.