107
Probabilistic mechanisms in human sensorimotor control Daniel Wolpert, University College London movement is the only way we have of Interacting with the world Communication: speech, gestures, writing sensory, memory and cognitive processes future motor outputs Q. Why do we have a brain? Sea Squirt A. To produce adaptable and complex movements

Wolpert Slides on Machine learning

  • Upload
    outlaw

  • View
    231

  • Download
    0

Embed Size (px)

DESCRIPTION

on Machine learning

Citation preview

Page 1: Wolpert Slides on Machine learning

Probabilistic mechanisms in human sensorimotor controlDaniel Wolpert, University College London

• movement is the only way we have of– Interacting with the world – Communication: speech, gestures, writing

• sensory, memory and cognitive processes future motor outputs

Q. Why do we have a brain?

Sea Squirt

A. To produce adaptable and complex movements

Page 2: Wolpert Slides on Machine learning

1000

1500

2000

2500

3000

3500

Why study computational sensorimotor control?

Year

Pag

es r2=0.96

1980 1990 2000 2010 2020 2030 2040 2050500

Experiments

Theory

Principles of Neural Science, Kandel et al.

Page 3: Wolpert Slides on Machine learning

vs.

What to move where

vs.

Moving

The complexity of motor control

Page 4: Wolpert Slides on Machine learning

Noise makes motor control hard

Noise = randomness

The motor system is Noisy

Perceptual noise– Limits resolution

Motor Noise– Limits control

NoisyPartial

Noisy

AmbiguousVariable

Page 5: Wolpert Slides on Machine learning

David Marr’s levels of understanding (1982)

1) the level of computational theory of the system

2) the level of algorithm and representation, which are used make computations

3) the level of implementation: the underlying hardware or "machinery" on which the computations are carried out.

Page 6: Wolpert Slides on Machine learning

Tutorial Outline– Sensorimotor integration

• Static multi-sensory integration• Bayesian integration• Dynamic sensor fusion & the Kalman filter

– Action evaluation• Intrinsic loss function• Extrinsic loss functions

– Prediction• Internal model and likelihood estimation • Sensory filtering

– Control• Optimal feed forward control• Optimal feedback control

– Motor learning of predictable and stochastic environments

Review papers on www.wolpertlab.com

Page 7: Wolpert Slides on Machine learning

Multi-sensory integration

Multiple modalities can provide information about the same quantity• e.g. location of hand in space

– Vision– Proprioception

• Sensory input can be– Ambiguous– Noisy

• What are the computations used in integrating these sources?

Page 8: Wolpert Slides on Machine learning

Ideal Observers

Maximum likelihood estimation (MLE)

i ix x ε= +

1 2( , ..., | )nP x x x x1

( | )n

ii

P x x=

=∏

2(0, )i iNε σ=

( )2

211

ˆ withn

ii i i n

i jj

x w x w σ

σ

−=

=

= =∑∑

( ) 12 2 2ˆ 1

nx i kj

kσ σ σ−

−=

= < ∀∑

2x

1x

x

Consider signals , {1 }in x i n= …

Two examples of multi-sensory integration

Page 9: Wolpert Slides on Machine learning

Visual-haptic integration (Ernst & Banks 2002)

Two alternative force choice size judgment• Visual• Haptic• Visual-haptic (with discrepancy)

Page 10: Wolpert Slides on Machine learning

Visual-haptic integration

Measure•Visual reliability •Haptic reliability•Predict

•Visual + Haptic noise•Weighting of

2Vσ2Hσ

2 Hσ

Prob

abili

ty

Size

Prob

abili

ty

Size difference

2 Hσ

Hσ Hσ

Page 11: Wolpert Slides on Machine learning

Visual-haptic integration

( )2

21

2

2 2 1

ii n

jj

VH V H

V H

w

w w w

σ

σ

σσ σ

−=

=

= = −+

∑( )

2 212 2ˆ 2 21

n V Hx ij

V H

σ σσ σσ σ

−−

== =

+∑

Weights Standard deviation (~threshold)

Optimal integration of vision and haptic information in size judgement

Page 12: Wolpert Slides on Machine learning

Visual-proprioceptive integrationClassical claim from prism adaptation

“vision dominates proprioception”

Page 13: Wolpert Slides on Machine learning

Reliability of proprioception depends on location

(Van Beers, 1998)

Reliability of visual localization is anisotropic

Page 14: Wolpert Slides on Machine learning

Integration models with discrepancyWinner takes all

Linear weighting of mean

Optimal integration

ˆ (1 )V Hw w= + −x x x

ˆ V HA B= +x x x

1 1 1

1 1

( )

( )PV P V

PV PV P P V Vµ µ µ

− − −

− −

Σ = Σ +Σ

= Σ Σ +Σ

Page 15: Wolpert Slides on Machine learning

Prisms displace along the azimuth•Measure V and P•Apply visuomotor discrepancy during right hand reach•Measure change in V and P to get relative adaptation

Vision 0.33Prop 0.67

(Van Beers, Wolpert & Haggard, 2002)

Page 16: Wolpert Slides on Machine learning

Visual-proprioceptive discrepancy in depth

AdaptationVision 0.72Prop 0.28

Visual adaptation in depth > visual adaptation in azimuth (p<0.01)> Proprioceptive adaptation in depth (p<0.05)

Proprioception dominates vision in depth

Page 17: Wolpert Slides on Machine learning

Priors and Reverend Thomas Bayes

“I now send you an essay which I have found among the papers of our deceased friend Mr Bayes, and which, in my opinion, has great merit....”

Essay towards solving a problem in the doctrine of chances. Philosophical Transactions of the Royal Society of London, 1764.

1702-1761

Page 18: Wolpert Slides on Machine learning

Bayes rule

A and B

( , )P A B = ( | )P A B ( | ) ( )P B A P A=

Belief instate sensory inpuAFTER t

Posterior

( )P B

P(state|sensory input)

Prior

BEFOREBelief in state sensory input

NeuroscienceA= State of the world B=Sensory Input

A = Disease B = Positive blood test

Likelihood

Evidence

P (sensoryinput|state) P(state)=P(sensory input)

Agiven B

Page 19: Wolpert Slides on Machine learning

Bayesian Motor Learning

= Optimal estimate (Posterior)

Bayes rule

+ Task statistics (Prior)Not all locations are

equally likely

Sensory feedback (Evidence)Combine multiple cuesto reduce uncertainty

Estimate

Evidence

Prior

P (sensorP(state| y input|ssensory tate Pinput) (st )) ate∝

Real world tasks have variability, e.g. estimating ball’s bounce location

Does sensorimotor learning use Bayes rule?If so, is it implemented• Implicitly: mapping sensory inputs to motor outputs to minimize error?• Explicitly: using separate representations of the statistics of the prior and sensory noise?

Page 20: Wolpert Slides on Machine learning

(Körding & Wolpert, Nature, 2004)

Prior

Lateral shift (cm)

Prob

abili

ty

0 1 2

Task in which we control 1) prior statistics of the task 2) sensory uncertainty

Page 21: Wolpert Slides on Machine learning

(Körding & Wolpert, Nature, 2004)

Prior

Lateral shift (cm)

Prob

abili

ty

0 1 2

Task in which we control 1) prior statistics of the task 2) sensory uncertainty

Page 22: Wolpert Slides on Machine learning

Sensory FeedbackLikelihood

Generalization

Learning

Page 23: Wolpert Slides on Machine learning

After 1000 trials

2 cm shift

1cm

No visual feedback

Prob

abili

ty

0 1 2

Lateral shift (cm)

Page 24: Wolpert Slides on Machine learning

0 1 2

Bayesian Compensation

Lateral shift (cm)

Models

Full Compensation

Lateral shift (cm)0 1 2

0 1 2Lateral Shift (cm)

Ave

rage

Err

or

Bia

s (c

m) 1

0

-1 0 1 2Lateral Shift (cm)

Ave

rage

Err

or

Bia

s (c

m) 1

0

-1

Mapping

0 1 2Lateral shift (cm)

0 1 2Lateral Shift (cm)

Ave

rage

Err

or

Bia

s (c

m) 1

0

-1

Page 25: Wolpert Slides on Machine learning

Supports model 2: Bayesian

Results: single subject0

Full

Bayes

Map

0 1 2Lateral Shift (cm)

Ave

rage

Err

or

Bia

s (c

m)

Page 26: Wolpert Slides on Machine learning

Supports model 2: Bayesian

Results: 10 subjects0

Full

Bayes

Map

0 1 2Lateral Shift (cm)

Ave

rage

Err

or

Bia

s (c

m)

-0.5 1 2.5

1

0

lateral shift [cm]Lateral shift (cm)

Infe

rred

Prio

r(n

orm

aliz

ed)

Page 27: Wolpert Slides on Machine learning

Bayesian integrationSubjects can learn

• multimodal priors• priors over forces• different priors one after the other

(Körding& Wolpert NIPS 2004, Körding, Ku & Wolpert J. Neurophysiol. 2004)

Page 28: Wolpert Slides on Machine learning

Statistics of the world shape our brain

Objects Configurations of our body

• Statistics of visual/auditory stimuli representation visual/auditory cortex• Statistics of early experience what can be perceived in later life

(e.g. statistics of spoken language)

Page 29: Wolpert Slides on Machine learning

Statistics of action

• 4 x 6-DOF electromagnetic sensors• battery & notebook PC

With limited neural resources statistics of motor tasks motor performance

Page 30: Wolpert Slides on Machine learning

Phase relationships and symmetry bias

Page 31: Wolpert Slides on Machine learning

Multi-sensory integration

• CNS – In general the relative weightings of the senses is

sensitive to their direction dependent variability– Represents the distribution of tasks– Estimates its own sensory uncertainty– Combines these two sources in a Bayesian way

• Supports an optimal integration model

Page 32: Wolpert Slides on Machine learning

Loss Functions in Sensorimotor system

What is the performance criteria (loss, cost, utility, reward)?

• Often assumed in statistics & machine learning – that we wish to minimize squared error for analytic or algorithmic tractability

• What measure of error does the brain care about?

Target Position

Post

erio

rPr

obab

ility

PriorLikePosterior lihood

P(sensory iP (state|sensory inpu nput|state) P(statet) )∝

[ ] ( , ) ( | _ ) _

ˆ ( ) arg min [ ]B actions

E Loss Loss state action P state sensory input dsensory input

Bayes estimator x s E Loss

=

=∫

Page 33: Wolpert Slides on Machine learning

Loss function

312 2

Target

Scenario 1 Scenario 2

( )f error

2Loss error=

12Loss error=

Loss error=

Loss=4+4=8 Loss=1+9=10

Loss=2+2=4 Loss=1+3=4

Loss=1+1.7=2.7Loss=1.4+1.4=2.8

Page 34: Wolpert Slides on Machine learning

Virtual pea shooter

MeanStarting location

Position (cm)Prob

abili

ty

-0.2 0 0.2

(Körding & Wolpert, PNAS, 2004)

Page 35: Wolpert Slides on Machine learning

Probed distributions and optimal means

Possible Loss functions

2Loss ( )error=MEAN

-2 -1 0 1 2Error (cm)

ρ=0.2

ρ=0.3

ρ=0.5

ρ=0.8

Distributions

MODEMaximize Hits

-2 -1 0 1 2Error (cm)

MEDIANLoss error=

Robust estimator

Page 36: Wolpert Slides on Machine learning

Shift of mean against asymmetry (n=8)

Mean squared error with robustness to outliers

Page 37: Wolpert Slides on Machine learning

0.1 1.0 1.3 1.5 1.7 1.9 2.1 2.3 2.5 2.7 2.9 3.1 3.3 3.5 3.7 3.9α=

Loss ierrorα=∑

Personalised loss function

Page 38: Wolpert Slides on Machine learning

Bayesian decision theory

Increasing probability of avoiding keeper

Increasing probability of being within the net

Page 39: Wolpert Slides on Machine learning

Imposed loss function (Trommershäuser et al 2003)

0 -100 -500+100+100 +100

Page 40: Wolpert Slides on Machine learning

Optimal performance with complex regions

Page 41: Wolpert Slides on Machine learning

State estimation• State of the body/world

– Set of time-varying parameters which together with• Dynamic equations of motion• Fixed parameters of the system (e.g. mass)

– Allow prediction of the future behaviour

• Tennis ball– Position– Velocity– Spin

Page 42: Wolpert Slides on Machine learning

State estimation

NOISE NOISE

Observer

Page 43: Wolpert Slides on Machine learning

Kalman filter• Minimum variance estimator

– Estimate time-varying state– Can’t directly observe state but only measurement

1t t t tA B+ = + +x x u w

1t t tC+ = +y x v

1ˆ ˆ ˆ[ ]t t t t t tA B K C+ = + + −x x u y x

Page 44: Wolpert Slides on Machine learning

State estimation

1

Forward DynamicModel

ˆ ˆ ˆ[ ]t t t t t tA B K C+ = + + −x x u y x

Page 45: Wolpert Slides on Machine learning

Kalman Filter

• Optimal state estimation is a mixture– Predictive estimation (FF)– Sensory feedback (FB)

Page 46: Wolpert Slides on Machine learning

Eye position

Location of object based on retinal location and gaze direction

PerceptActual

Motor command FM Eye

Position

Page 47: Wolpert Slides on Machine learning

Sensory likelihood

(Wolpert & Kawato, Neural Networks 1998Haruno, Wolpert, Kawato, Neural Computation 2001)

P (sensorP(state| y input|ssensory tateinput) P(st )) ate∝

Page 48: Wolpert Slides on Machine learning

Sensory predictionOur sensors report• Afferent information: changes in outside world• Re-afferent information: changes we cause

+ =

Internal source

Externalsource

Page 49: Wolpert Slides on Machine learning

TicklingSelf-administered tactile stimuli rated as less ticklish than externally administered tactile stimuli. (Weiskrantz et al, 1971)

Page 50: Wolpert Slides on Machine learning

Does prediction underlie tactile cancellation in tickle?

0

0.5

1

1.5

2

2.5

3

Self-producedtactile stimuli

Robot-producedtactile stimuli

Condition

Tic

kle

ratin

g ra

nk

Gain control or precise spatio-temporal prediction?

P<0.001

Page 51: Wolpert Slides on Machine learning

Spatio-temporal prediction

0

0.5

1

1.5

2

2.5

3

3.5

4

Delay 0ms Delay100ms

Delay200ms

Delay300ms

Robot-produced

tactilestimuliCondition

Tic

kle

ratin

g ra

nk

P<0.001

P<0.001

0

0.5

1

1.5

2

2.5

3

3.5

4

0 degrees 30 degrees 60 degrees 90 degrees External

Condition

Tic

kle

ratin

g ra

nk

P<0.001P<0.001

(Blakemore, Frith & Wolpert. J. Cog. Neurosci. 1999)

Page 52: Wolpert Slides on Machine learning

The escalation of force

Page 53: Wolpert Slides on Machine learning

Tit-for-tat

Force escalates under rules designed to achieve parity: Increase by ~40% per turn

(Shergill, Bays, Frith & Wolpert, Science, 2003)

Page 54: Wolpert Slides on Machine learning

Perception of force

70% overestimate in force

Page 55: Wolpert Slides on Machine learning

Perception of force

Page 56: Wolpert Slides on Machine learning

Labeling of movements

Large sensorydiscrepancy

Page 57: Wolpert Slides on Machine learning

Defective prediction in patients with schizophrenic

• The CNS predicts sensory consequences

• Sensory cancellation in Force production

• Defects may be related to delusions of control

Patients Controls

Page 58: Wolpert Slides on Machine learning

Motor LearningRequired if:

• organisms environment, body or task change• changes are unpredictable so cannot be pre-specified • want to master social convention skills e.g writing

Trade off between: – innate behaviour (evolution)

• hard wired• fast• resistant to change

– learning (intra-life)• adaptable• slow• Maleable

Page 59: Wolpert Slides on Machine learning

Motor Learning

Actual behaviour

Page 60: Wolpert Slides on Machine learning

Predicted outcome can be compared to actual outcome to generate an error

Supervised learning is good for forward models

Page 61: Wolpert Slides on Machine learning

Weakly electric fish (Bell 2001)Produce electric pulses to • recognize objects in the dark or in murky habitats• for social communication.

The fish electric organ is composed of electrocytes, • modified muscle cells producing action potentials • EOD = electric organ discharges• Amplitude of the signal is between 30 mV and 7V • Driven by a pacemaker in medulla, which triggers each discharge

Page 62: Wolpert Slides on Machine learning

Sensory filteringSkin receptors are derived from the lateral line system

Removal of expected or predicted sensory input is one of the very general functions of sensory processing. Predictive/associative mechanisms for changing environments

Page 63: Wolpert Slides on Machine learning

Primary afferent terminate in cerebellar-like structures

Primary afferents terminate on principal cells either directly or via interneurons

Page 64: Wolpert Slides on Machine learning

Block EOD discharge with curare

Specific for Timing (120ms), Polarity, Amplitude & Spatial distribution

Page 65: Wolpert Slides on Machine learning

Proprioceptive Prediction

Tail bend affects feedbackPassive Bend phase locked to stimulus:

Bend

Page 66: Wolpert Slides on Machine learning

Learning ruleChanges in synaptic strength requires principal cell spike discharge

Change depends on timing of EPSP to spike

Anti-Hebbian learning

T1

T2

T2-T1

• Forward Model can be learned through self-supervised learning• Anti-hebbian rule in Cerebellar like structure of he electric fish

Page 67: Wolpert Slides on Machine learning

Motor planning (what is the goal of motor control)

Duration Hand Trajectory

Joint Muscles

• Tasks are usually specified at a symbolic level• Motor system works at a detailed level, specifying muscle activations• Gap between high and low-level specification• Any high level task can be achieved in infinitely many low-level ways

Page 68: Wolpert Slides on Machine learning

Eye-saccades Arm- movements

Motor evolution/learning results in stereotypyStereotypy between repetitions and individuals

Time (ms)

• Main sequence• Donder’s law• Listings Law

• 2/3 power law• Fitts’ law

Page 69: Wolpert Slides on Machine learning

Models

HOW models– Neurophysiological or black box models– Explain roles of brain areas/processing units in

generating behaviorWHY models

– Why did the How system get to be the way it is? – Unifying principles of movement production

• Evolutionary/Learning– Assume few neural constraints

Page 70: Wolpert Slides on Machine learning

The Assumption of OptimalityMovements have evolved to maximize fitness

– improve through evolution/learning– every possible movement which can achieve a task has a cost– we select movement with the lowest cost

Overall cost = cost1 + cost2 + cost3 ….

Page 71: Wolpert Slides on Machine learning

Optimality principles

• Parsimonious performance criteriaElaborate predictions

• Requires– Admissible control laws– Musculoskeletal & world model– Scalar quantitative definition of task

performance – usually time integral of f(state, action)

Page 72: Wolpert Slides on Machine learning

Open-loop

• What is the cost– Occasionally task specifies cost

• Jump as high as possible• Exert maximal force

– Usually task does not specify the cost directly• Locomotion well modelled by energy minimization• Energy alone is not good for eyes or arms

Page 73: Wolpert Slides on Machine learning

What is the cost?

Saccadic eye movements

• little vision over 4 deg/sec• frequent 2-3 /sec• deprives us of vision for 90

minutes/day

⇒Minimize time

Page 74: Wolpert Slides on Machine learning

Arm movementsMovements are smooth – Minimum jerk (rate of change of acceleration) of the hand

(Flash & Hogan 1985)

2 2

04 5 3

0 04 5 3

0 0

( ) ( )

( ) ( )(15 6 10 )

( ) ( )(15 6 10 )/

T

T

T

Cost x t y t dt

x t x x x

y t y y yt T

τ τ τ

τ τ ττ

= +

= + − − −

= + − − −=

Page 75: Wolpert Slides on Machine learning

Smoothness• Minimum Torque change (Uno et al, 1989)

Shouldertorque

Elbowtorque2 2

0( ) ( )

T

s eCost t t dtτ τ= +∫

Page 76: Wolpert Slides on Machine learning

The ideal cost for goal-directed movement

• Makes sense - some evolutionary/learning advantage• Simple for CNS to measure• Generalizes to different systems

– e.g. eye, head, arm

• Generalizes to different tasks – e.g. pointing, grasping, drawing

→ Reproduces & predicts behavior

Page 77: Wolpert Slides on Machine learning

Motor command noise

MotorSystem

Noise

Position

Error minimized byrapidity

Page 78: Wolpert Slides on Machine learning

Fundamental constraint=Signal-dependent noise

• Signal-dependent noise: – Constant coefficient of variation– SD (motor command) ~ Mean (motor command)

• Evidence from– Experiments: SD (Force) ~ Mean (Force)– Modelling

• Spikes drawn from a renewal process• Recruitment properties of motor units

(Jones, Hamilton & Wolpert , J. Neurophysiol., 2002)

Page 79: Wolpert Slides on Machine learning

Task optimization in the presence of SDN

Given SDN, Task Given SDN, Task ≡≡ optimizing optimizing f(statisticsf(statistics))

An average motor command ⇒ probability distribution (statistics) of movement.

Controlling the statistics of actionControlling the statistics of action

Page 80: Wolpert Slides on Machine learning

Finding optimal trajectories for linear systems

2 2 2( ) ( )u t k u tσ =

time

TM

x(t)0

[ ( )] ( ) ( )M

E x M u p M d Aτ τ τ= − =∫

2

0 0

2 2 2 2

0 0

[ ( )] [ ( ) ( )] [ ( )] ( )

( ) ( ) ( ) ( )

M M

M M

Var x T Var u p T d Var u p T d

k u p T d w u d

τ τ τ τ τ τ

τ τ τ τ τ τ

= − = −

= − =

∫ ∫∫ ∫

time

p(t)

Impulse responseSignal-dependent noise

( ) ( )

0[ ( )] ( ) ( ) 0

Mn nE x M u p M dτ τ τ= − =∫

Cost

Constraints

System

A

Linear constraints with quadratic cost:can use quadratic programming or isoperimetric optimization

Page 81: Wolpert Slides on Machine learning

Saccade predictions

SDN

Motor command

Jerk

3rd order linear system

Page 82: Wolpert Slides on Machine learning

0 100 2000

20

Time (ms)

Deg

rees

Prediction: very slow saccade

22 degree saccade in 270 ms (normally ~ 70 ms)

Page 83: Wolpert Slides on Machine learning

Head free saccade

0 100 200 300 400 500 600 700 8000

20

40

60

80

100

120

Deg

rees

Time (ms)

Head

Gaze

Eye

Τ1=0.3 Τ2=0.3

Τ1=0.15 Τ2=0.08 Free parameter, eye:head noise

0 100 200 300 400 500 600 700 8000

20

40

60

80

100

120

Time (ms)

Deg

rees

Eye

Head

Gaze

(Tomlinson & Bahra, 1986)

Page 84: Wolpert Slides on Machine learning

Coordination: Head and eyeFor a fixed duration (T), Var(A)=k A2

Var(A)=k A2 Var(A)= k (A/2)2 + k (A/2)2

= k A2 /2Var(A)=k A2

Eye only Head onlyEye &Head

Page 85: Wolpert Slides on Machine learning

Movement extent vs. target eccentricity

Gaze amplitude

Ang

ular

dev

iatio

n at

ac

quis

ition

Eye

Head

Gaze amplitude

0 20 40 60 80 100 120 1400

10

20

30

40

50

60

70

80

90

100

0 100 200 300 400 500 600 700 8000

20

40

60

80

100

120

} }Head

Eye

Page 86: Wolpert Slides on Machine learning

Arm movements

Drawing (⅔ power law) f=path errorObstacle avoidance f= limit probability of collision

Smoothness

Non smooth movement ⇒ requires abrupt change in velocity⇒ given low pass system⇒ large motor command⇒ increased noise

Smoothness ⇒ accuracy

Feedforward control•Ignores role of feedback•Generates desired movements•Cannot model trial-to-trial variability

Page 87: Wolpert Slides on Machine learning

Optimal feedback control (Todorov 2004)

– Optimize performance over all possible feedback control laws– Treats feedback law as fully programmable

• command=f(state)• Models based on reinforcement learning optimal cost-to-go functions• Requires a Bayesian state estimator

Page 88: Wolpert Slides on Machine learning

Minimal intervention principle

• Do not correct deviations from average behaviour unless they affect task performance– Acting is expensive

• energetically• noise

– Leads to • uncontrolled manifold• synergies

Uncontrolled manifold

Page 89: Wolpert Slides on Machine learning

Optimal control with SDN• Biologically plausible theoretical underpinning

for both eye, head, arm movements

• No need to construct highly derived signals to estimate the cost of the movement

• Controlling statistics in the presence of noise

Page 90: Wolpert Slides on Machine learning

What is being adapted?

• Possible to break down the control process:

• Visuomotor rearrangements• Dynamic perturbations• [timing, coordination , sequencing]

• Internal models captures the relationship between sensory and motor variables

Page 91: Wolpert Slides on Machine learning

Altering dynamics

Page 92: Wolpert Slides on Machine learning

Altering Kinematics

Page 93: Wolpert Slides on Machine learning

Representation of transformations

Look-upTable

Non-physicalParametersθ=f(x,ω)

PhysicalParametersθ=acos(x/L)

High storageHigh flexibility

Low Generalization

Low storageLow flexibility

High Generalization

θ

x

L ÷asin

x

θ

L

x

θ

x θ1 103 35. .. .

Page 94: Wolpert Slides on Machine learning

Generalization paradigm

• Baseline– Assess performance over

domain of interest – (e.g. workspace)

• Exposure– Perturbation: New task– Limitation: Limit the

exposure to a subdomain

• Test– Re-assess performance

over entire domain of interest

Page 95: Wolpert Slides on Machine learning

Difficulty of learning

(Cunningham 1989, JEPP-HPP)

• Rotations of the visual field from 0—180 degrees

Difficulty• increases from 0 to 90• decreases from 120 to 180

• What is the natural parameterization

Page 96: Wolpert Slides on Machine learning

Viscous curl field

(Shadmehr & Mussa-Ivaldi 1994, J. Neurosci.)

Page 97: Wolpert Slides on Machine learning

Representation from generalization: Dynamic

1. Test: Movements over entire workspace

2. Learning– Right-hand workspace – Viscous field

3. Test: Movements over left workspace

Two possible interpretations force = f(hand velocity) or torque=f(joint velocity)

Joint-based learning of dynamics

(Shadmehr & Mussa-Ivaldi 1994, J. Neurosci.)

Left hand workspaceBefore After with Cartesian field

Page 98: Wolpert Slides on Machine learning

Visuomotor coordinates

z

yx

(x,y,z)

Cartesian

z

y

x

Spherical Polar

θφ

r(r,φ,θ)

α1

α2

α3

(α1,α2,α3)

Joint angles

Page 99: Wolpert Slides on Machine learning

Representation- Visuomotor1. Test: Pointing accuracy to a set of targets

2. Learning– visuomotor remapping– feedback only at one target

3. Test: Pointing accuracy to a set of targets

-50

-40

-30

y = 22.2/16.2 cm y = 29.2/26.2 cm y = 36.2 cm

-20 -10 0 10 20

20

30

40z = -43.6 cm z = -35.6 cm

-20 -10 0 10 20

z = -27.6 cm

-20 -10 0 10 20

x (cm)

Prediction of eye-centred spherical coordinates

(Vetter et al, J. Neurophys, 1999)

Predictions of eye-centred spherical coordinates

• Generalization paradigms can be used to assess– Extent of generalization– Coordinate system of transformations

Page 100: Wolpert Slides on Machine learning

Altering dynamics: Viscous curl fieldBefore Early with force

Late with force Removal of force

Page 101: Wolpert Slides on Machine learning

A muscle activation levels sets the spring constant k (or resting length) of the muscle

Stiffness control

Equilibrium point

Page 102: Wolpert Slides on Machine learning

Equilibrium point control

• Set of muscle activations (k1,k2,k3…) defines a posture • CNS learns a spatial mapping

– e.g. hand positions muscle activations(x,y,z) (k1,k2,k3…)

Page 103: Wolpert Slides on Machine learning

Equilibrium control

The hand stiffness can vary with muscle activation levels.

Low stiffness High stiffness

Page 104: Wolpert Slides on Machine learning

Controlling stiffness

Burdet et al (Nature, 2002)

Page 105: Wolpert Slides on Machine learning

Stiffness ellipses

• Internal models to learn stable tasks• Stiffness for unpredictable tasks

Page 106: Wolpert Slides on Machine learning

Summary– Sensorimotor integration

• Static multi-sensory integration• Bayesian integration• Dynamic sensor fusion & the Kalman filter

– Action evaluation• Intrinsic loss function• Extrinsic loss functions

– Prediction• Internal model and likelihood estimation • Sensory filtering

– Control• Optimal feed forward control• Optimal feedback control

– Motor learning of predictable and stochastic environments

Wolpert-lab papers on www.wolpertlab.com

Page 107: Wolpert Slides on Machine learning

References• Bell, c.(2001) Memory-based expectations in electrosensory systemsCurrent Opinion in

Neurobiology 2001, 11:481–487• Burdet, E., R. Osu, et al. (2001). "The central nervous system stabilizes unstable dynamics by

learning optimal impedance." Nature 414(6862): 446-9.• Cunningham, H. A. (1989). "Aiming error under transformed spatial maps suggest a structure for

visual-motor maps." J. Exp. Psychol. 15:3: 493-506.• Ernst, M. O. and M. S. Banks (2002). "Humans integrate visual and haptic information in a

statistically optimal fashion." Nature 415(6870): 429-33.• Flash, T. and N. Hogan (1985). "The co-ordination of arm movements: An experimentally

confirmed mathematical model " J. Neurosci. 5: 1688-1703.• Shadmehr, R. and F. Mussa-Ivaldi (1994 ). "Adaptive representation of dynamics during learning

of a motor task." J. Neurosci. 14:5: 3208-3224.• Todorov, E. (2004). "Optimality principles in sensorimotor control." Nat Neurosci 7(9): 907-15.• Trommershauser, J., L. T. Maloney, et al. (2003). "Statistical decision theory and the selection of

rapid, goal-directed movements." J Opt Soc Am A Opt Image Sci Vis 20(7): 1419-33.• Uno, Y., M. Kawato, et al. (1989). "Formation and control of optimal trajectories in human

multijoint arm movements: Minimum torque-change model " Biological Cybernetics 61: 89-101.• van Beers, R. J., A. C. Sittig, et al. (1998). "The precision of proprioceptive position sense." Exp

Brain Res 122(4): 367-77.• Weiskrantz, L., J. Elliott, et al. (1971). "Preliminary observations on tickling oneself." Nature

230(5296): 598-9.