MURILow-Level

Control

Fabrication

High-LevelControl

How is Compliance used in Locomotion?

Berkeley & Stanford: Measurements of Cockroach Locomotion

What Compliance Strategies in Human-level Tasks?

Harvard & Johns Hopkins: Compliance Learning and Strategies for Unstructured Environments

Fabrication

MURILow-LevelControl High-Level

Control

What strategies are used in insect locomotion and what are their implications?Insect locomotion studies (Berkeley Bio)New measurement capabilities (Stanford)

What motor control adaptation strategies do people use and how can they be applied to robots?

Compliance Learning and Strategies for Unstructured Environments (Harvard & Johns Hopkins)Implications for biomimetic robots (Harvard, Stanford)

Guiding questionsGuiding questions

Yoky Matsuoka and Rob Howe

Harvard University

MURI

High-LevelControl

Impedance Adaptation in Unstructured Environments

MURI

High-LevelControl

• An example of manipulation with impedance

• Why is biology superior to current robots in an unstructured environment?

Manipulation with Impedance

MURI

High-LevelControl

Identify Impedance Learning Strategy in Human

• Two big questions:– What is the initial strategy used to cope with

unknown/unstructured environments?– After learning, what does the biology pick as the

good solution for impedance for a given environment?

How can these solutions help robot’s control strategy?

MURI

High-LevelControl

Comparison Between Analytical and Biological Solutions

• We can mathematically derive optimal impedance for a linear world.– Biological system converges to the analytical solution. ---

great!– Biological system converges to a different solution. --- what

and why: put the biological solution back in the equations and reverse engineer.

• What about a nonlinear varying world where it is difficult to derive the optimal impedance?– What does the biological system do? Can it be modeled as a

solution for robots?

• Goal: Find the “best” impedance.– For this case, find best

Khand.

• Uncertainty in the world– mball, kball, ball(0), and

khand

mhand

khand

mball

kballxball

xhand

x

MURI

High-LevelControl

Example: Linear World --- Catching a Ball

ball

hand

mball

mhand

kball

khand

• Cases:1. Hand stiffnes (khand) is too high

• hand < 0 bounces up

2. Hand stiffness (khand) is too low• xhand > Threshold bottoms out

3. Hand stiffness (khand) is just right• xball xhand until switch is pressed

x

khand 0 infinite

2 3 1

MURI

High-LevelControl

Example: Linear World --- Catching a Ball

mhand

khand

mball

kballxball

xhand

• Solve for xhand(t) and xball(t)– initial condition

• ball(0) > 0

• xball(0) = 0

• hand(0)= 0

• xhand(0) = 0xhand

x

x

m x k x x

m x k x x k xball ball ball hand ball

hand hand ball hand ball hand hand

( )

( )

0

0

MURI

High-LevelControl

Example: Linear World --- Catching a Ball

mhand

khand

mball

kballxball

MURI

High-LevelControl

Analytical Linear World to Biological Motor Control

• The example relates task performance to limb impedance and optimal solution.– Other examples: leg impedance, etc.

• Now measure human strategy….– “System identification” – Need a new technique

MURI

High-LevelControl

Existing System Identification Techniques

• Time invariant systems --- easy– assume constant m, b, and k over time.– apply external impulse perturbation force. – repeat the same condition and average.

)()()()( oxxktxbtxmtF

MURI

High-LevelControl

Existing System Identification Techniques

• Time varying systems• Cannot apply impulses close to each other.

• Need multiple impulses to solve for multiple unknowns.

– PRBS (Lacquaniti, et al. 1993)

)()()()( oxxktxbtxmtF

Setup

Handle

Accelerometer

Data Acquisition

System

Processor

HumanSubject

MURI

High-LevelControl

New System Identification Technique to Observe Learning

Robot

Force Sensor

Monitor

MURI

High-LevelControl

New System Identification Technique to Observe Learning

• Very short duration

• Very clean data from force and acc. sensors

F

m*a

b*v

k*x

m=F/a b= (F-ma)/v k= (F-ma-bv)/x

MURI

High-LevelControl

Testing the New Technique

MURI

High-LevelControl

Video of the task here

• Phantom robot is used as the perturbation/measurement tool.

• Task: balance the moving ball on paddle.

– ball moves at constant speed

– dies when the ball falls off the paddle

– perturbation applied every second

MURI

High-LevelControl

Testing the New Technique

MURI

High-LevelControl

Impedance Change with Learning

k change over time b change over time

m change over time

• Observe the impedance change within one catch

• Observe the impedance change between catches

** under development --- pilot studies underway

MURI

High-LevelControl

Contact Interaction Task -- Impedance Dependent Task

k

b

MURI

High-LevelControl

Current Understanding of the Structure of the Biological Controller

Inverse DynamicsModel

uu

usi

q

q

q q

Long-loopGain

q

uf

ui

q

Forward DynamicsModel 120 msec

30 msec

60 msec

InertialDynamics

From Shadmehr

MURI

High-LevelControl

• Developed a new impedance identification technique– Based on virtual environment --- extremely

versatile– Confirmed ability to measure instantaneous

impedance, characterized changes with learning.

Impedance AdaptationConclusions and Future Work

MURI

High-LevelControl

• Current experiments– Determine human interaction strategies

• initial impedance

• learning characteristics

• final impedance

• Next experiments– Determine human interaction strategies for

nonlinear varying tasks• e.g. plastic deformation (running in sand)

Impedance AdaptationConclusions and Future Work

Control of LocomotionMURI

High-LevelControl

Local controller (single limb):

Control of the limb based on local information:

- position and velocity of the limb

Is the limb far enough to the back so that I can start the return stroke?

- forces acting on the limb

Is the supporting load small enough for me to lift the limb?

Task controller (all limbs):

Coordination with other limbs:

- position of the other legs

Across species, control of limb based on local information appears similar (Cruse, TINS, 90). Coordination with other limbs appears highly species dependent.

MURI

High-LevelControl

Step cycle: generation of power (stance) and return (swing) strokes.

Return phase:

Move the limb from posterior to anterior position along a desired velocity profile.

- Maintain proper impedance to remain stable in case of perturbations

After hitting an obstacle, the limb should converge back to the desired path

- Adapt impedance to allow for generation of desired behavior in the face

of a persistent environment

limb is moving in highly viscous fluid, it must adapt its impedance to the characteristics of the environment.

Impedance control and adaptation in a position control task

Power phase:

Maintain contact, maintain height of load, move limb from front to rear.

Impedance control and adaptation in a contact/force control task

Control of a Limb Based on Local Information

MURI

High-LevelControl

Current Limb Local Control Models for Locomotion in Insects

Cruse et al (Neural Networks 1998): Stick insect model

- limb has little or no inertia

- no muscle like actuators

- controller output is velocity, feedback sensing via position and linear feedback

- no ability to adapt

Essentially a kinematic model of a limb only, with little or no dynamics

This kind of model tells us little about how to design good controllers

MURI

High-LevelControl

General Goals:

1. To understand what impedance strategies a biomechanical controller uses when it moves the limb in a position control task. Apply the results to control of the return phase.

2. To understand impedance strategies of the biomechanical controller in a force control task. Apply the results to the control of the stance phase.

Approach:

Study the human arm’s impedance adaptive control strategies in both position and force control tasks. Test validity of the strategies on a robotic system.

Designing a Single Limb Impedance Controller

MURI

High-LevelControl

Designing a Single Limb Impedance Controller

Task Division:

1. Impedance control at very short time intervals (<10 msec, preflexes)

Yoky Matsuoka and Rob Howe

2. Impedance control at intermediate and long time intervals (<300 msec)

Tie Wang and Reza Shadmehr

3. Test and implementation on a robotic system

Jay Dev Desai and Rob Howe

MURI

High-LevelControl

Challenges:

The biomechanics of the human arm are dominated by multiple feedback loops, with various time delays. Impedance measurements are done through imposition of perturbations and measurement of responses.

• How do time delays affect measures of arm impedance?

• Humans learn internal models when they learn control. How does a change in the internal model affect measures of arm impedance?

• Impedance measures require an estimation of where the system would have been if it had not been perturbed. How well can we do this with a non-stationary system like the human arm?

Quantifying Impedance Control Strategies of a

Biomechanical Controller

MURI

High-LevelControl

Current Understanding of the Structure of the Biological Controller

Inverse DynamicsModel

uu

usi

q

q

q q

Long-loopGain

q

uf

ui

q

Forward DynamicsModel 120 msec

30 msec

60 msec

InertialDynamics

Through modulation of input u to the muscles, impedance of the system is changed. The impedance depends on 3 feedback pathways:1. Near zero-delay mechanical stiffness/viscosity of the muscles (Yoky).2. Short delay sensory feedback through spinal structures.3. Long delay sensory feedback through cortical structures (forward model).

Are muscle “preflexes” enough?

1 cmdt=10ms dt=30msdt=10ms

Intact control systemHigh-level sensory feedback loop disrupted

MURI

High-LevelControl

MURI

High-LevelControl

Impedance of a biological arm: A definition

qd

dMB

dq

dMK

tuqqMqqq

tqtuut

tuqqMqqq

t

uqqM

qqqCqqHqqq

q

q

0

0

0000000

000

:Viscosity

:Stiffness

)(),,(),,(

).(y trajectorparticular a follows arm ,)( and )(when

)(),,(),,(

)( :tenvironmen from torquesExternal

),,( :musclesby produced Torques

),()(),,( : torquesInertial

MURI

High-LevelControl

Estimating Impedance: Theory

uqqqqq

uqquqqMuuqqqqM

uqqMuuqqqqMqqq

uuqqqqMqqqqqq

uuqq

uqqMqqq

du

dM

qd

dM

dq

dM

du

dM

qd

dM

dq

dM

uqq

qqq

00000

000000000

00000000

0000000

00

0000000

)()(

...),,(),,(

),,(),,()()( :)1()2(

(2) ),,(),,(

. and in resulting , :force perturbing a Impose

(1) ),,(),,( :onperturbati a without Dynamics

MURI

High-LevelControl

Estimating Impedance: Requirements

)( :onperturbatiour ofresult a asinput neural musclein change The 3.

)( :onsperturbati no were thereif follow wouldsystem thePath that 2.

),()(),,( :arm theof dynamics Inertial 1.

:estimatemust We

,, :measurecan We

0

tu

tq

qqqCqqHqqq

qq

uqtBqtKqqqdu

dM

u000 )()()()(

MURI

High-LevelControl

1.0 Estimating Inertial Dynamics of the Arm (Theory)

qqqCqqHqqq ),()(),,( : torquesInertial

FJ

qqa

qqqaqqaqqC

aqaa

qaaqaaaqH

T

0)sin(

))(sin()sin(),(

)cos(

)cos()cos(2)(

122

2122222

3223

2232231

),,(

0),,( musclesby produced Torques

qqq

uqqM

. , , parameters estimate :OBJECTIVE

hand on the acting force external :

321 aaa

F

Give a force pulse, use data for up to 14 ms after the pulse to estimate inertial parameters.

MURI

High-LevelControl

0 0.1 0.2 0.3 0.4-2

0

2

4

6

Time (s)

Sho

ulde

r T

orqu

e (N

m)

0 0.1 0.2 0.3 0.4-1

0

1

2

Time (s)

Elb

ow T

orqu

e (N

m)

0 0.1 0.2 0.3 0.41

1.05

1.1

1.15

1.2

Time (s)

Sho

ulde

r A

ngel

(ra

d)

0 0.1 0.2 0.3 0.41.6

1.605

1.61

1.615

Time (s)

Elb

ow A

ngel

(ra

d)

0 0.1 0.2 0.3 0.4-0.5

0

0.5

Time (s)

Sho

ulde

r V

eloc

ity (

rad/

s)

0 0.1 0.2 0.3 0.4-0.2

-0.1

0

0.1

0.2

Time (s)

Elb

ow V

eloc

ity (

rad/

s)

1.1 Estimating Inertial Dynamics of the Arm (Methods)

MURI

High-LevelControl

1.2 Estimating Inertial Dynamics of the Arm (Results)

0 10 20 30 40 50 60 70-4

-2

0

2

4

6S

ho

uld

er T

orq

ue

(Nm

) Experiment Model Error

0 10 20 30 40 50 60 70-4

-3

-2

-1

0

1

2

3

4

Elb

ow

To

rqu

e (N

m)

Index of Array

]0.1293 0.17270.3025 [][321

aaa

MURI

High-LevelControl

2.0 Predicting the Un-perturbed Trajectory (Theory)

][MovPerturbed_

][d_MovUnperturbe

2

)_:_(

1

)1_:1(timerial

2

)_:_(

1

)1_:1(timetrial

timemovtimeprbtrialtimeprbtrial

timemovtimeprbtrialtimeprbtrial

PP

SS

:ionapproximatfor bases estimate to(PCA) Analysis Components Principal Use

pertubed ismovement point when time:_

movementeach ofduration :

movements ofnumber :

matrices data raw ofmean : ,

matrices data raw fromPCA : ,

matrices data raw : ,

timeprb

time

trial

PSPS

PSPS

PS

meanmean

pcapca

Legend

MURI

High-LevelControl

2.1 Trajectory Prediction (Methods)

.predict todata raw theof components principal theuse

wemovements, of data raw theusing of instead that is idea The

))(())()(( :2 Method

movement.unknown epredict th to

function base theas movements of data raw theall use that is idea The

that Asume

predictionfor bases as movements dunperturbe all use :1 Method

methods) (2 predict to , , Use

'2

'1

22'2

11'1

21

222

111

2

21222111111112

2122

211

1

2121

KCK

PPK

PPK

KCK

SSK

SSK

S

SSSSSSSSPPP

KKPSKPSK

PPSS

pca

pca

pca

pca

mean

pcapcameanmeanpcapcamean

MURI

High-LevelControl

0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 0.55-0.1

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Sho

ulde

r V

eloc

ity(m

/s)

Measured Trajectory Predicted Trajectory

0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 0.55-2

-1.5

-1

-0.5

0

0.5

Elb

ow V

eloc

ity(m

/s)

Time (s)

Prediction error (%)

17.47516.89634.36491.9896

17.85197.0604 4.28551.9125

6.05.05.04.04.03.03.02.0 ssssssss

2.2 Trajectory Prediction (velocity)

MURI

High-LevelControl

2.3 Trajectory Prediction (force)

0 20 40 60-2

-1

0

1

0 20 40 60-2

-1

0

1

0 20 40 60-2

-1

0

1

0 20 40 60-2

-1

0

1

0 20 40 60-2

-1

0

1

0 20 40 60-2

-1

0

1

0 20 40 60-2

-1

0

1

0 20 40 60-2

-1

0

1

Conclusion: Position, velocity, and force can be reasonably well predicted for up to 300 msec after the last sampled data point.

Time (10 msec)

For

ce (

N)

MURI

High-LevelControl

3.0 Estimating the Effect of u(t) on Arm Impedance

uqtBqtKqqqdu

dM

u000 )()()()(

Inverse DynamicsModel

uu

usi

q

q

q q

Long-loopGain

q

uf

ui

q

Forward DynamicsModel 120 msec

30 msec

60 msec

InertialDynamics

u is the change in the input to the muscles as a result of our perturbation.

While u cannot be measured directly, we know that it depends on a number of time-delayed, possibly adaptive error feedback systems.

• Time-delayed error feedback from the spinal reflexes

• Time-delayed error feedback from the forward model based cortical pathways

• Input from inverse model based “open-loop” controller

MURI

High-LevelControl

In general, a time delay d in a feedback loop reduces apparent viscosity and adds apparent mass to a system.

Example:

)0( 0)()2/1(

:of impedanceapparent has system resulting The

)(2/1)()()(

:delay on theexpansion seriesTaylor Using

0)(

22

2

dkxxkdbxkdm

ddt

xdd

dtdx

txtx

dtkxxbxm

3.1 Time-delayed effect of u(t) on Arm Impedance

MURI

High-LevelControl

3.2 Estimating u(t) in Terms of Measurable Quantities

1. Effect of Spinal Reflexes

: thatfollowsit

on,perturbati aafter and seriesTaylor a of in termsdelay theRewrite

))(())(( :Assume

muscles theofeedback terror based spinal

muscles theinput to adaptive based, model forward

muscles theinput to adaptive based, model inverse

muscles theinput to

dssdsss

s

f

i

sfi

qdtqbqdtqku

u

u

u

u

uuuu

qdbdkqdkbqku sssssssss )2/1()( 2

MURI

High-LevelControl

3.2 Estimating u(t) in Terms of Measurable Quantities

2. Effect of the Inverse model

Inverse DynamicsModel

uu

usi

q

q

q q

Long-loopGain

q

uf

ui

q

Forward DynamicsModel 120 msec

30 msec

60 msec

InertialDynamics

0on perturbati aafter therefore,0 ii u

dq

du

MURI

High-LevelControl

3.2 Estimating u(t) in Terms of Measurable Quantities

3. Effect of the Forward Model

dfdff

f

dfdff

t

i

qdtqddqqbqdtqddqdqqku

qu

qE

qqbqqku

dtq

ddtqddtqdtqtq

dtdtqtq

qqEqqCuqqMqHq

)(ˆ)(ˆ2/3

: quantities measurable of in terms Rewrite

model. forward theofoutput is ˆ ,0ˆ Assume :1 Case

)ˆ()ˆ(

:assume and expansions seriesTaylor r with thei)( delays Replace

)(ˆ)(2)()(ˆ

:model forward theofon intergrati theeapproximatLinearly

ˆ)(ˆ

),(ˆ),(),,()(ˆ :model forward theofn Computatio

0022

0

2

1

MURI

High-LevelControl

3.2 Estimating u(t) in Terms of Measurable Quantities

4. Effect of Adaptation of the Forward Model

qWHdbWHdkqWdHbWHdku

u

qWHdqWdHqdtqddqqb

qWHdqWHddqdtqddqdqqku

qu

qWE

fffff

f

df

dff

f

1213112

1210

13120

22

:be willadaptation before response respect to

with response adapted in the change the,on perturbati a toresponseIn

)(ˆ

)(ˆ2/3

: quantities measurable of in terms Rewrite

.ˆ and present, ist environmen field force a Assume :2 Case

Predictable changes in impedance should occur as a function of the kind of model that the system learns as it practices movements in an unstructured environment.

If learning is via a forward model, the apparent viscosity must decrease as compared to values obtained before the controller had adapted.

MURI

High-LevelControl

-0.1 -0.05 0 0.05 0.1 0.15-0.05

0

0.05

0.1

0.15

X (m)

Y (

m)

Perturb the movement in different directions

Measuring Impedance of the Moving System

MURI

High-LevelControl

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5-8

-6

-4

-2

0

2

4

6

8

10

12

Time into the movement (s)

Interaction force

Inertial dynamics

Impedance Controlled force

acceleration (m/s2 )

velocityx25 (m/sec) positionx50 (m) Fo

rce

(N)

Per

turb

Impedance Early into the Movement

MURI

High-LevelControl

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5-8

-6

-4

-2

0

2

4

6

8

10

12

Dynamic force

Interaction force

Impedance controller’s force

Positionx50 (m)

Velocity x 25 (m/s)

Acceleration (m/s2)

Time into the movement (s)

Forc

e (N

)

Impedance in the Middle of the Movement

Per

turb

MURI

High-LevelControl

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5-8

-6

-4

-2

0

2

4

6

8

10

12

Time into the movement (s)

velocityx25 (m/sec)

acceleration (m/s2)

Inertial dynamics

Interaction force

Perturb

Impedance Controlled force

positionx50 (m)

Forc

e (N

)

Impedance Near End of Movement

MURI

High-LevelControl 0

-200

Join

t Sti

ffne

ss (

N.m

/rad

)

100 msec

Stiffness of the System: Results

System Characteristics:

- Initially a very stiff system (likely due to intrinsic muscle stiffness).

- The system yields as the perturbation persists, with stiffness dropping as the short- and long-loop reflex systems take control.

MURI

High-LevelControl

Immediate Plans

Position control task:

- Impose a force field, measure impedance changes as the system adapts

- Do we see evidence for formation of a forward model as indicated by reductions in the system’s viscosity?

Force control task:

- Measure impedance changes in a task that requires maintaining contact (pushing in order to roll a virtual conveyer belt).

- Push too hard, the belt breaks. Push too soft, arm slips on the belt.

- Measure impedance changes as the virtual belt’s dynamics changes, requiring adaptation in the controller.

- Virtual belt is produced by the robot manipulandum.