43
UTA Research Institute (UTARI) The University of Texas at Arlington, USA F.L. Lewis Optimal Control Introduction Supported by NSF, ARO, AFOSR He who exerts his mind to the utmost knows nature’s pattern. The way of learning is none other than finding the lost mind. Meng Tz 500 BC Man’s task is to understand patterns in nature and society. Mencius

optimal control intro for course 2012.ppt control intro for course 2012.pdfOptimal Control: Linear Quadratic Regulator System Cost function Optimal Control is Algebraic Riccati equation

  • Upload
    others

  • View
    30

  • Download
    0

Embed Size (px)

Citation preview

Page 1: optimal control intro for course 2012.ppt control intro for course 2012.pdfOptimal Control: Linear Quadratic Regulator System Cost function Optimal Control is Algebraic Riccati equation

UTA Research Institute (UTARI)The University of Texas at Arlington, USA

F.L. Lewis

Optimal Control Introduction

Supported byNSF, ARO, AFOSR

He who exerts his mind to the utmost knows nature’s pattern.

The way of learning is none other than finding the lost mind.

Meng Tz500 BC

Man’s task is to understand patterns innature and society.

Mencius

Page 2: optimal control intro for course 2012.ppt control intro for course 2012.pdfOptimal Control: Linear Quadratic Regulator System Cost function Optimal Control is Algebraic Riccati equation

Optimal Control

Reinforcement learning

Control of Robotic Devices

Adaptive Learning of Human-Robot Interfaces

Dynamic Games for HRI

Importance of Feedback Control

Darwin- FB and natural selectionVolterra- FB and fish population balanceAdam Smith- FB and international economyJames Watt- FB and the steam engineFB and cell homeostasis

The resources available to most species for their survival are meager and limited

Nature uses Optimal control

SYSTEMoutputsAlfred North Whitehead

Von BertalanffySystems Theory 1920s

inputs

processing

Page 3: optimal control intro for course 2012.ppt control intro for course 2012.pdfOptimal Control: Linear Quadratic Regulator System Cost function Optimal Control is Algebraic Riccati equation

Relevance- Machine Feedback Control

qd

qr1qr2

AzEl

barrel flexiblemodes qf

compliant coupling

moving tank platform

turret with backlashand compliant drive train

terrain andvehicle vibrationdisturbances d(t)

Barrel tipposition

qd

qr1qr2

AzEl

barrel flexiblemodes qf

compliant coupling

moving tank platform

turret with backlashand compliant drive train

terrain andvehicle vibrationdisturbances d(t)

Barrel tipposition

Vehicle mass m

ParallelDamper

mc

activedamping

uc

(if used)

kc cc

vibratory modesqf(t)

forward speedy(t)

vertical motionz(t)

surface roughness(t)

k c

w(t)

Series Damper+

suspension+

wheel

Single-Wheel/Terrain System with Nonlinearities

High-Speed Precision Motion Control with unmodeled dynamics, vibration suppression, disturbance rejection, friction compensation, deadzone/backlash control

VehicleSuspension

IndustrialMachines

Military LandSystems

Aerospace

plant

controlu(t)

outputy(t)

controller

systemidentifier estimated 

output 

)(ˆ ty

identificationerror

desiredoutput

)(tyd

plant

controlu(t)

outputy(t)

controllerdesiredoutput

)(tyd

trackingerror

plant

controlu(t)

outputy(t)

controller #1

controller #2

desiredoutput

)(tyd

trackingerror

Indirect Scheme

Fixed Controller Topologies

Direct Scheme

Feedback/FeedforwardScheme

Page 4: optimal control intro for course 2012.ppt control intro for course 2012.pdfOptimal Control: Linear Quadratic Regulator System Cost function Optimal Control is Algebraic Riccati equation

Cell Homeostasis The individual cell is a complex feedback control system. It pumps ions across the cell membrane to maintain homeostatis, and has only limited energy to do so.

Cellular Metabolism

Permeability control of the cell membrane

http://www.accessexcellence.org/RC/VL/GG/index.html

Optimality in Biological Systems

Optimality in Control Systems DesignR. Kalman 1960

Rocket Orbit Injection

http://microsat.sm.bmstu.ru/e-library/Launch/Dnepr_GEO.pdf

Fmmm

F

r

wvv

m

F

rr

vw

wr

cos

sin2

2

ObjectivesGet to orbit in minimum timeUse minimum fuel

Dynamics

Page 5: optimal control intro for course 2012.ppt control intro for course 2012.pdfOptimal Control: Linear Quadratic Regulator System Cost function Optimal Control is Algebraic Riccati equation

Optimal Control is Effective for:Aircraft AutopilotsVehicle engine controlAerospace VehiclesShip ControlIndustrial Process ControlRobot Control

Optimal Control:Minimum timeMinimum fuelMinimum energyConstrained control

Optimal control solutions are found byOffline solution of Matrix Design equationsA full dynamical model of the system is needed

Optimality and Games

PBPBRQPAPA TT 10

1 Tu R B Px Kx

BuAxx ( ( )) ( ) ( ) ( )T T T

t

V x t x Qx u Ru d x t Px t

Full system dynamics must be knownOff‐line solution

0 ( , , ) 2 2 ( )T

T T T T T T TV VH x u V x Qx u Ru x x Qx u Ru x P Ax Bu x Qx u Ru

x x

Optimal Control: Linear Quadratic Regulator (LQR)

System 

Performance Index

Leibniz’s formula‐

Optimal Control is SVFB

Algebraic Riccati equation

( , , ) 2( ) 0T T Td V dH x u Ax Bu Px x Qx u Ru

du x du

Stationarity Condition

2 0TRu B Px

( ) ( ) ( ) ( )T T T T T T T TdV x Qx u Ru x Px x Px x Px x Px Ax Bu Px x P Ax Bu

dt

( , , )V

H x ux

Hamiltonian function

Differential equivalent to PI is the Bellman equation

Page 6: optimal control intro for course 2012.ppt control intro for course 2012.pdfOptimal Control: Linear Quadratic Regulator System Cost function Optimal Control is Algebraic Riccati equation

PBPBRQPAPA TT 10

1 Tu R B Px Kx

BuAxx

( ( )) ( ) ( ) ( )T T T

t

V x t x Qx u Ru d x t Px t

Full system dynamics must be knownOff‐line solutionCannot change performance objectives

Optimal Control: Linear Quadratic Regulator

System model 

Performance Function

Optimal Control is

Algebraic Riccati equation

MATLAB Control Systems Toolbox

[K,P]=lqr(A,B,Q,R)

PBPBRQPAPA TT 10

1 Tu R B Px Kx

BuAxx

( ( )) ( ) ( ) ( )T T T

t

V x t x Qx u Ru d x t Px t

Full system dynamics must be knownOff‐line solution

0 ( , , ) 2 2 ( )T

T T T T T T TV VH x u V x Qx u Ru x x Qx u Ru x P Ax Bu x Qx u Ru

x x

Optimal Control: Linear Quadratic Regulator

0 ( ) ( )T TA BK P P A BK Q K RK

u Kx

System 

Cost

Leibniz’s formula‐ Differential equivalent is the Bellman equation

Given any stabilizing FB policy

The cost value is found by solving Lyapunov equation

Optimal Control is

Algebraic Riccati equation

( , , ) 2( ) 0T T Td V dH x u Ax Bu Px x Qx u Ru

du x du

Page 7: optimal control intro for course 2012.ppt control intro for course 2012.pdfOptimal Control: Linear Quadratic Regulator System Cost function Optimal Control is Algebraic Riccati equation

Many Successful Design Applications of Optimal Control

Page 8: optimal control intro for course 2012.ppt control intro for course 2012.pdfOptimal Control: Linear Quadratic Regulator System Cost function Optimal Control is Algebraic Riccati equation

PBPBRQPAPA TT 10

1 Tu R B Px Kx

BuAxx

( ( )) ( ) ( , ) ( ) ( )T T T

t t

V x t x Qx u Ru d r x u d x t Px t

Full system dynamics must be knownOff‐line solution

Optimal Control: Linear Quadratic Regulator

System 

Cost function

Optimal Control is

Algebraic Riccati equation

MATLAB Control Systems Toolbox

Chop off Tail of cost function 

( ( )) ( , ) ( ( ))t T

t

V x t r x u d V x t T

( ( )) ( , ) ( , )t T

t t T

V x t r x u d r x u d

Bellman Equation

1 Tu R B Px Kx Update Control using Hamiltonian Physics

Reinforcement Learning‐ Policy Iteration

Online Learning Feedback ControlSynthesis of Robot Control and Biologically Inspired Learning

Page 9: optimal control intro for course 2012.ppt control intro for course 2012.pdfOptimal Control: Linear Quadratic Regulator System Cost function Optimal Control is Algebraic Riccati equation

Neural Network Properties

Learning

Recall

Function approximation

Generalization

Classification

Association

Pattern recognition

Clustering

Robustness to single node failure

Repair and reconfiguration

Nervous system cell. http://www.sirinet.net/~jgjohnso/index.html

Dynamic Neural Networks and Control Learning

Robot System[I]

Robust ControlTerm

qqd

v(t)

e

Outer Tracking Loop

f(x)

rKv

Nonlinear Inner Learning Loop

^

Feedforward Loop

Tunable weights

dq

Page 10: optimal control intro for course 2012.ppt control intro for course 2012.pdfOptimal Control: Linear Quadratic Regulator System Cost function Optimal Control is Algebraic Riccati equation

Theorem 1 (NN Weight Tuning for Stability)

Let the desired trajectory )(tqd and its derivatives be bounded. Let the initial tracking error be

within a certain allowable set U . Let MZ be a known upper bound on the Frobenius norm of the

unknown ideal weights Z .

Take the control input as

vrKxVW vTT )ˆ(ˆ with rZZKtv MFZ )()( .

Let weight tuning be provided by

WrFxrVFrFW TTT ˆˆ'ˆˆˆ , VrGrWGxV TT ˆ)ˆ'ˆ(ˆ

with any constant matrices 0,0 TT GGFF , and scalar tuning parameter 0 . Initialize the weight estimates as randomVW ˆ,0ˆ .

Then the filtered tracking error )(tr and NN weight estimates VW ˆ,ˆ are uniformly ultimately bounded. Moreover, arbitrarily small tracking error may be achieved by selecting large control

gains vK .

NEW NN Tuning Laws for Control Learning

Force Control

Flexible pointing systemsSimis labs, Inc. and US Army

Vehicle active suspensionLeo Davis Technol.

SBIR Contracts

Learning NN Controller

Page 11: optimal control intro for course 2012.ppt control intro for course 2012.pdfOptimal Control: Linear Quadratic Regulator System Cost function Optimal Control is Algebraic Riccati equation

Applications at Boeing Defense Space & Security

Kevin Wise and Eugene Lavretsky

Highly reliable adaptive uncertainty approximation compensators for flight control applications:

unmanned aircraft – Phantom Ray

Tailkit adaptive control systems for Joint Direct Attack Munition (JDAM) munitions:

Mk-82, Mk-84, and Laser-guided variantsFielded for defense in Iraq and Afghanistan.

BUT-Nature is More than StableNature is OPTIMAL and conserves energy

Page 12: optimal control intro for course 2012.ppt control intro for course 2012.pdfOptimal Control: Linear Quadratic Regulator System Cost function Optimal Control is Algebraic Riccati equation

We want to Learn optimal control solutions Online in real-time Using adaptive control techniquesWithout knowing the full dynamics

For Adaptive Man/Machine systems-

Reinforcement Learning

Every living organism improves its control actions based on rewards received from the environment

The resources available to living organisms are usually meager.Nature uses optimal control.

1. Apply a control.  Evaluate the benefit of that control.2. Improve the control policy.

RL finds optimal policies by evaluating the effects of suboptimal policies

Optimality in Biological Systems

Page 13: optimal control intro for course 2012.ppt control intro for course 2012.pdfOptimal Control: Linear Quadratic Regulator System Cost function Optimal Control is Algebraic Riccati equation

Different methods of learning

SystemAdaptiveLearning system

ControlInputs

outputs

environmentTuneactor

Reinforcementsignal

Actor

Critic

Desiredperformance

Reinforcement learningIvan Pavlov 1890s

Actor-Critic Learning

We want OPTIMAL performance- ADP- Approximate Dynamic Programming

Sutton & Barto book

Adaptive Critic structure

Slow learning Improvement loop

Reinforcement learning

Fast inner control loops

Page 14: optimal control intro for course 2012.ppt control intro for course 2012.pdfOptimal Control: Linear Quadratic Regulator System Cost function Optimal Control is Algebraic Riccati equation

PBPBRQPAPA TT 10

1 Tu R B Px Kx

BuAxx

( ( )) ( ) ( , ) ( ) ( )T T T

t t

V x t x Qx u Ru d r x u d x t Px t

Full system dynamics must be knownOff‐line solution

Optimal Control: Linear Quadratic Regulator

System 

Cost function

Optimal Control is

Algebraic Riccati equation

MATLAB Control Systems Toolbox

Chop off Tail of cost function 

( ( )) ( , ) ( ( ))t T

t

V x t r x u d V x t T

( ( )) ( , ) ( , )t T

t t T

V x t r x u d r x u d

Bellman Equation

1 Tu R B Px Kx Update Control using Hamiltonian Physics

Reinforcement Learning‐ Policy Iteration

CT Policy Iteration – How to implement online?Linear Systems Quadratic Cost‐ LQR

( ) ( ) ( )( ) ( ) ( ) ( )t T

T T T Tk k k k

t

x t P x t x Q K RK x d x t T P x t T

Policy evaluation‐ solve IRL Bellman Equation

( ) ( ) ( ) ( ) ( )( ) ( )t T

T T T Tk k k k

t

x t P x t x t T P x t T x Q K RK x d

( ( )) ( ) ( )TV x t x t Px tValue function is quadratic

( ) ( ) ( ) ( ) ( ) ( )t T

T T T Tk k k k

t

p t p x t x t T x Q K RK x d

Reinforcement on time interval [t, t+T]

regression vector

Same form as standard System ID problems

Solve using RLS or batch LS

Union of Reinforcement Learning and Adaptive Control

Value Function Approximation

Page 15: optimal control intro for course 2012.ppt control intro for course 2012.pdfOptimal Control: Linear Quadratic Regulator System Cost function Optimal Control is Algebraic Riccati equation

1. Select initial control policy

2. Find associated cost

3. Improve control 11

Tk kK R B P

( ) ( ) ( ) ( ) ( ) ( , )t T

T T Tk k k

t

p x t x t T x Q K RK x d t t T

Bellman EquationSolves Lyapunov eq. without knowing dynamics

t t+T

observe x(t)

observe x(t+T)

apply uk(t)=Kkx(t)

observe cost integral

update P

do RLS until convergence to Pk

update control gain to Kk+1

A is not needed anywhere

This is a data-based approach that uses measurements of x(t), u(t)Instead of the plant dynamical model.

Integral Reinforcement Learning (IRL)

Data set at time [t,t+T)

( ), ( , ), ( )x t t t T x t T

Motor control 200 Hz

Oscillation is a fundamental property of neural tissue

Brain has multiple adaptive clocks with different timescales

theta rhythm, Hippocampus, Thalamus, 4-10 Hzsensory processing, memory and voluntary control of movement.

gamma rhythms 30-100 Hz, hippocampus and neocortex high cognitive activity.

• consolidation of memory• spatial mapping of the environment – place cells

The high frequency processing is due to the large amounts of sensorial data to be processed

Spinal cord

D. Vrabie, F. Lewis, and Dan Levine- RL for Continuous-Time Systems

Page 16: optimal control intro for course 2012.ppt control intro for course 2012.pdfOptimal Control: Linear Quadratic Regulator System Cost function Optimal Control is Algebraic Riccati equation

Doya, Kimura, Kawato 2001

Limbic system

Cerebral cortexMotor areas

ThalamusBasal

ganglia

Cerebellum

Brainstem

Spinal cord

Interoceptivereceptors

Exteroceptivereceptors

Muscle contraction and movement

Summary of Motor Control in the Human Nervous System

reflex

Supervisedlearning

ReinforcementLearning- dopamine

(eye movement)

inf. olive

Hippocampus

Unsupervisedlearning

Limbic System

Motor control 200 Hz

theta rhythms 4-10 Hz

picture by E. StinguD. Vrabie

Memoryfunctions

Long term

Short term

Hierarchy of multiple parallel loops

gamma rhythms 30-100 Hz,

Page 17: optimal control intro for course 2012.ppt control intro for course 2012.pdfOptimal Control: Linear Quadratic Regulator System Cost function Optimal Control is Algebraic Riccati equation

Adaptive Critic structure

Theta waves 4-8 Hz

Reinforcement learning

Motor control 200 Hz

F.L. Lewis and D. Vrabie,“Reinforcement learning andadaptive dynamic programmingfor feedback control,” IEEECircuits & Systems Magazine,Invited Feature Article, pp. 32-50, Third Quarter 2009.

IEEE Control Systems Magazine“Reinforcement learning andfeedback Control,” Dec. 2012

Page 18: optimal control intro for course 2012.ppt control intro for course 2012.pdfOptimal Control: Linear Quadratic Regulator System Cost function Optimal Control is Algebraic Riccati equation

D. Vrabie, K. Vamvoudakis, and F.L. Lewis, Optimal Adaptive Control and Differential Games by Reinforcement Learning Principles, IET Press, 2012, to appear.

Books

F.L. Lewis, D. Vrabie, and V. Syrmos, Optimal Control, third edition, John Wiley and Sons, New York, 2012.

New Chapters on:Reinforcement LearningDifferential Games

Direct Optimal Adaptive Controller

A hybrid continuous/discrete dynamic controller whose internal state is the observed cost over the interval

Draguna Vrabie

DynamicControlSystemw/ MEMORY

xu

V

ZOH T

x A x B u System

T Tx Q x u R u

Critic

Actor

L

T T

Run RLS or use batch L.S.To identify value of current control

Update FB gain afterCritic has converged

Reinforcement interval T can be selected on line on the fly – can change

Solves Riccati Equation Online without knowing A matrix

Page 19: optimal control intro for course 2012.ppt control intro for course 2012.pdfOptimal Control: Linear Quadratic Regulator System Cost function Optimal Control is Algebraic Riccati equation

Optimal Control Design Allows a Lot of Design Freedom

( ( )) ( , )t

V x t r x u d

Tailor controls design by choosing utility

Minimum EnergyMinimum FuelMinimum TimeControl Actuator Constraints

Optimal Control for Constrained Input Systems

This is a quasi-norm

2

0

2 ( )u

T

qu d

Weaker than a norm –homogeneity property is replaced by the weaker symmetry property

qqxx

(Used by Lyshevsky for H2 control)

Control constrained by saturation function tanh(p)

p

1

-1

0 0

( , ) ( ) 2 ( )u

TJ u d Q x d dt

Encode constraint into Value function

Then Is BOUNDED1 ( )T Vu R g x

x

Murad Abu-Khalaf

Page 20: optimal control intro for course 2012.ppt control intro for course 2012.pdfOptimal Control: Linear Quadratic Regulator System Cost function Optimal Control is Algebraic Riccati equation

-0.1 0 0.1 0.2 0.3 0.4 0.5 0.6-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6State Evolution for both controlllers

x1

x2

Switching Surface methodNonquadratic functionals method

1

0 0

tanh( ) 2 ( )u

TTV x Q x Rd dt

Near Minimum-Time ControlEncode into Value Function

Murad Abu-Khalaf

Page 21: optimal control intro for course 2012.ppt control intro for course 2012.pdfOptimal Control: Linear Quadratic Regulator System Cost function Optimal Control is Algebraic Riccati equation

Force ControlIntelligent Automation, Inc.

Flexible pointing systemsSimis labs, Inc. and US Army

Vehicle active suspensionLeo Davis Technol.

SBIR ContractsSBA Tibbets Award 1996 (led by TMAC)

Learning NN Controller

A. Yesildirek and F.L. Lewis, "Method for feedback linearization of neural networks and

neural network incorporating same," U.S. Patent 5,943,660, awarded 24 August 1999.

S. Jagannathan and F.L. Lewis, "Discrete-time tuning of neural network controllers for

nonlinear dynamical systems," U.S. Patent 6,064,997, awarded 16 May 2000.

R. Selmic, F.L. Lewis, A.J. Calise, and M.B. McFarland, "Backlash Compensation Using

Neural Network," U.S. Patent 6,611,823, awarded 26 Aug. 2003.

J. Campos and F.L. Lewis, "Method for Backlash Compensation Using Discrete-Time

Neural Networks," U.S. Patent 7,080,055, awarded July 2006.

Patents

K. Vamvoudakis, D. Vrabie, and F.L. Lewis, “Control methodology for online adaptation to optimal feedback controller using integral reinforcement learning,” provisional patent, filed March 2012.

Page 22: optimal control intro for course 2012.ppt control intro for course 2012.pdfOptimal Control: Linear Quadratic Regulator System Cost function Optimal Control is Algebraic Riccati equation

Applications of our Algorithms at Boeing Defense Space & Security

Kevin Wise and Eugene Lavretsky

Highly reliable adaptive uncertainty approximation compensators for flight control applications:

Unmanned aircraft – Phantom Ray

Tailkit adaptive control systems for Joint Direct Attack Munition (JDAM) munitions:

Mk-82, Mk-84, and Laser-guided variants

Fielded for defense in Iraq and Afghanistan.Saved Lives.

optimal engine controllers based on RL for Caterpillar, Ford (Zetec engine), and GM

Applications of Our Algorithms to Auto Engine Control

Student S. Jagannathan11 US patents

8-10% improvement in fuel efficiency and a drastic reduction in NOx (90%), HC (30%) and CO (15%) by operating with adaptive exhaust gas recirculation.

Savings to Caterpillar were over $65,000 per component.

Page 23: optimal control intro for course 2012.ppt control intro for course 2012.pdfOptimal Control: Linear Quadratic Regulator System Cost function Optimal Control is Algebraic Riccati equation

A Q-Learning Based Adaptive Optimal Controller Implementation for a Humanoid

Robot Arm

Said Ghani Khan1, Guido Herrmann1, Tony Pipe1, Chris Melhuish1

Bristol Robotics LaboratoryUniversity of Bristol and University of the West of England, Bristol, UK

Conference on Decision and Control (CDC) 2011, Orlando, 11 December 2011

The cost function is modified to include constraints

Introducing Constraints 

The new cost function....

qL is the joint limit.

Page 24: optimal control intro for course 2012.ppt control intro for course 2012.pdfOptimal Control: Linear Quadratic Regulator System Cost function Optimal Control is Algebraic Riccati equation

BRL BERT II ARM Bristol Robotics Lab

Natural Human-Like Motion

An Approximate Dynamic Programming Based Controllerfor an Underactuated 6DoF Quadrotor

Emanuel Stingu Frank Lewis

Automation & Robotics Research InstituteUniversity of Texas at ArlingtonARRI

Supported byARO grant W91NF-05-1-0314NSF grant ECCS-0801330

Page 25: optimal control intro for course 2012.ppt control intro for course 2012.pdfOptimal Control: Linear Quadratic Regulator System Cost function Optimal Control is Algebraic Riccati equation

3 control loops

The quadrotor has 17 states and only 4 control inputs, thus it is very under-actuated.Three control loops with dynamic inversion are used to generate the 4 control signals.

x

y

zamountof tilt

yaw

directionof tilt

Momentum theory applied to the propeller

Translationand yaw

Controller

AttitudeController

Motor andPropellerController

Referencetrajectory

Position and velocity errors

Desired attitude

Desired forces

Motor commands

Dynamic inversionaccelerations to attitude

Dynamic inversionrotation accelerations to forces

Altitude control: thrust force

Dynamic inversionthrust forces to motor voltages

Using standard controller

UsingReinforcement learningcontroller

Page 26: optimal control intro for course 2012.ppt control intro for course 2012.pdfOptimal Control: Linear Quadratic Regulator System Cost function Optimal Control is Algebraic Riccati equation

Potential New Applications in Prosthetics

With Sesh Commuri, Oklahoma UniversityAnd Muthu Wijesundara

Adaptive Reinforcement Learning of Human/Prosthetic Ankle Interactions

Current methods use stored gait information

Page 27: optimal control intro for course 2012.ppt control intro for course 2012.pdfOptimal Control: Linear Quadratic Regulator System Cost function Optimal Control is Algebraic Riccati equation
Page 28: optimal control intro for course 2012.ppt control intro for course 2012.pdfOptimal Control: Linear Quadratic Regulator System Cost function Optimal Control is Algebraic Riccati equation

What about Adaptive Human-Robot Interfaces (HRI)?

Adaptive Learning

HRI

Different usersDifferent robotic devices

Interface learns so as to improve Human/Robot team performance

Tune parameters such as stiffness, speed of response, dynamic motion

There are 3 componentsHumanRobotThe Interface

They should all be intelligent

We want verifiable performance and straightforward tuning algorithms

Page 29: optimal control intro for course 2012.ppt control intro for course 2012.pdfOptimal Control: Linear Quadratic Regulator System Cost function Optimal Control is Algebraic Riccati equation

Several Approaches to Adaptive HRI

1. Learning of Coordinate Transforms

2. Dynamic Motion Primitives to Modify Task Parameters

3. Adaptive Impedance Control

Current and Expected Funding

F.L. Lewis, “Adaptive Dynamic Programming for Real-Time Cooperative Multi-Player Games and Graphical Games,” NSF Grant, $272,000 for 3 years, July 2011.

D. Popa, Z. Celik-Butler, D. Butler, and F.L. Lewis, “NRI: Multi-Modal Skin and Garments for Healthcare and Home Robots,” NSF Grant, $1.3M for 4 years, Sept. 2012.

F.L. Lewis, “Games and Learning for Cooperative Nonlinear Systems and Internal Structure of Coalitions on Graphs, TARDEC/NAC grant, $81,000 for 1 year, Jan. 2013 expected.IF NO FISCAL CLIFF!

1d dq J y

Robot Inverse kinematics

( )dy t ( )dq t

( )dy tRobot

Inverse Jacobian

Hard to find AccuratelyNeed dynamics model of robotDoes not generalize to different robots

Learning of Coordinate Transforms

User coordinates Robot coordinates

User coordinates Robot coordinates

Standard robot controller

Page 30: optimal control intro for course 2012.ppt control intro for course 2012.pdfOptimal Control: Linear Quadratic Regulator System Cost function Optimal Control is Algebraic Riccati equation

( )dy tRobot

Inverse Jacobian

Hard to find accuratelyNeed dynamics model of robotDoes not generalize to different robots

1. Learning of Coordinate Transforms

Usercoordinates Robot

coordinates

Standard robot controller

Learning the Interface MappingWhat is it exactly?

u y

f u

What can we do to get f(u)?

The simplest way is to obtain a set of inputs and a set of outputs and calculate the relationship (Curve Fitting)

u

yCurve Fitting

Algorithm f x

0 0

( )M P

i i ij ii j

y f u w w u

Static nonlinear map

Dan PopaReinforcement Learning

Page 31: optimal control intro for course 2012.ppt control intro for course 2012.pdfOptimal Control: Linear Quadratic Regulator System Cost function Optimal Control is Algebraic Riccati equation

NN Training

Log sigmoid function is used as a neuron activation function . 1

1 ex

Train using MATLAB NN Toolbox very easily if input/output pairs are known.

Dan Popa

Static Mapping Approach

f u

f(u) is a coordinate transformation

Found by training neural network with known input/output pairsDan Popa

Page 32: optimal control intro for course 2012.ppt control intro for course 2012.pdfOptimal Control: Linear Quadratic Regulator System Cost function Optimal Control is Algebraic Riccati equation

Reinforcement Learning

u

yCurve Fitting

Algorithm f u

What if we cannot specify the desired output trajectory directly ? 

“Learning by interacting with an environment”Reinforcement LearningReinforcement Learning

With RL we do not have to specify a desired trajectory.

Instead, a Reward Function is used

Dan Popa

How to find the Reward Function?

Model of Task

A task is a profile in velocity/force spaceDifferent Curve for Different Tasks

2. Learn and Modify Task Parameters

Page 33: optimal control intro for course 2012.ppt control intro for course 2012.pdfOptimal Control: Linear Quadratic Regulator System Cost function Optimal Control is Algebraic Riccati equation

20( ) ( ) ( )T Tx Dx Ks g x K g x KW V s

s s

Dynamic Motion Primitives for Characterizing the Task

Learning the task:Kinesthetic demonstration to learn D, K, initial weightsReward-based learning to tune the weights

Phase variable s makes DMP independent of time

Tunable NN weights

Different NN weights give different task trajectories

• Build an adaptive interface system that allows a single operator to manipulate a robots with multiple degrees of freedom and/or multiple robots• The interface system should be intuitive,  easy to use and can be learned quickly , and should be able to apply with different robot configurations

Robot System

Operator

Performance Evaluation

Output

Interface System

input

Reinforcement Learning for Human-Robot InterfacesDan Popa

What to learn?How to evaluate Performance?

Adaptive Impedance Control

Page 34: optimal control intro for course 2012.ppt control intro for course 2012.pdfOptimal Control: Linear Quadratic Regulator System Cost function Optimal Control is Algebraic Riccati equation

3. Adaptive Impedance Control

ImpedanceModel

Inner loop FB controllers

Human intentRobot performance

Performanceevaluation

Tuning

Tune interface so robot learns to control the human model to improve task performance

Model of Human Behavior

For a given task, a skilled human operator adapts to learn-

1. An inverse model of the robot system to cancel nonlinearities

2. A feedforward predictive control based on the task

Frequency Crossover Theory -

Human operator adapts to make the overall closed-loop man-machine system

look like a high bandwidth frequency response

And remain invariant to a wide range of task variations and changes in operating condition

Suzuki & Furuta 2012

Page 35: optimal control intro for course 2012.ppt control intro for course 2012.pdfOptimal Control: Linear Quadratic Regulator System Cost function Optimal Control is Algebraic Riccati equation

For a given task, after learning, the human model has three parts1. Time delay due to vision processing and reaction time2. A first-order filter due to neuromuscular dynamics3. A PD controller in the cerebellum

Adaptive Impedance Control as a 2-Player Game

InterfaceParameters

Human intentRobot performance

Critic Agent forPerformance Evaluation

Tuning

A 2-Player Dynamic Game

Player 1- the human

Player 2- the Interface!

Page 36: optimal control intro for course 2012.ppt control intro for course 2012.pdfOptimal Control: Linear Quadratic Regulator System Cost function Optimal Control is Algebraic Riccati equation

Sun Tz bin fa孙子兵法

Graphical Coalitional Games

500 BC

Optimal Control is Effective for:Aircraft AutopilotsVehicle engine controlAerospace VehiclesShip ControlIndustrial Process ControlRobot Control

Multi-player Games Occur in:EconomicsControl Theory disturbance rejectionTeam gamesInternational politicsSports strategy

Optimality and Games

Page 37: optimal control intro for course 2012.ppt control intro for course 2012.pdfOptimal Control: Linear Quadratic Regulator System Cost function Optimal Control is Algebraic Riccati equation

Actor‐Critic Game structure ‐ three time scales

x

u2 1 0;x Ax B u B w x

System

Controller/Player 1

w( 1)T i kuu B P x 1

1T i

ww B P x

Interface/Player 2

CriticLearning procedure

ˆ ˆ, if =1

ˆ ˆ ˆ ˆ, if >1

T T T

T T

V x C Cx u u i

V w w u u i

T T

1 2

1i

uP ( 1) 1 ( 1)i k i i k

u u uP P Z

V

i

uPx

( ) 1 ( )i k i i k

u u uP P Z

The cloud-capped towers, the gorgeous palaces,The solemn temples, the great globe itself,Yea, all which it inherit, shall dissolve,And, like this insubstantial pageant faded,Leave not a rack behind.

We are such stuff as dreams are made on, and our little life is rounded with a sleep.

Our revels now are ended. These our actors, As I foretold you, were all spirits, and Are melted into air, into thin air.

Prospero, in The Tempest, act 4, sc. 1, l. 152-6, Shakespeare

Page 38: optimal control intro for course 2012.ppt control intro for course 2012.pdfOptimal Control: Linear Quadratic Regulator System Cost function Optimal Control is Algebraic Riccati equation
Page 39: optimal control intro for course 2012.ppt control intro for course 2012.pdfOptimal Control: Linear Quadratic Regulator System Cost function Optimal Control is Algebraic Riccati equation
Page 40: optimal control intro for course 2012.ppt control intro for course 2012.pdfOptimal Control: Linear Quadratic Regulator System Cost function Optimal Control is Algebraic Riccati equation
Page 41: optimal control intro for course 2012.ppt control intro for course 2012.pdfOptimal Control: Linear Quadratic Regulator System Cost function Optimal Control is Algebraic Riccati equation
Page 42: optimal control intro for course 2012.ppt control intro for course 2012.pdfOptimal Control: Linear Quadratic Regulator System Cost function Optimal Control is Algebraic Riccati equation
Page 43: optimal control intro for course 2012.ppt control intro for course 2012.pdfOptimal Control: Linear Quadratic Regulator System Cost function Optimal Control is Algebraic Riccati equation

The way that can be told is not the Constant WayThe name that can be named is not the Constant Name

For nameless is the true wayBeyond the myriad experiences of the world

To experience without intention is to sense the world

All experience is an archwherethrough gleams that untravelled landwhose margins fade forever as we move

Dao ke dao feichang daoMing ke ming feichang ming