Game Theoretic Approaches to Solve H Problems

Game Theoretic Approaches to Solve H Problems

Alex Feng

Joint work with Brian D. O. Anderson, Alexander Lanzon and Michael Rotkowitz

Global Motivation

• Some H problems give Riccati equations which cause standard solvers to break down

• We give a cure

• The cure is extendable to linear periodic H

problems and nonlinear game theory problems – This may be useful if there are numerical problems– Solution procedures are very few anyway, and

numerically not well understood.

Outline

• Global Motivation

• Solving Continuous-Time H Riccati equations

• Nonlinear Extension

• Solving Discrete-Time H Riccati equations

• Conclusions and Future Work

Outline


• Solving H Riccati equations

– Detailed Motivation– Solving Riccati Equations, Kleinman and its repair– Algorithm Convergence– Game Theoretic Interpretation

• Nonlinear Extension

• Solving Discrete-Time Riccati Equations


Detailed Motivation

• Software to solve Asymptotic Riccati Equations arising in H2 problems is standard

• The software can collapse on certain problems• The Kleinman algorithm will save the day:

– Recursive– Requires Lyapunov equation solutions– Requires stabilizing gain to initialize– Converges quadratically (it is a Newton algorithm)

• It does not extend to H equations (indefinite quadratic term)

• What can we do?

Solving Riccati Equations

Direct methods

Computational disadvantages

A numerical example

The example on the next transparency shows what can go wrong with an H-infinity Riccati equation if we use some direct methods.

Numerical Problem

Too Large !!Direct methods

Direct methods problems are an old difficulty! It is a motivation for using the Kleinman algorithm in the LQ case.

Direct methods


Iterative methods

Traditional Newton methods

Difficult to choose an initial condition

Kleinman algorithm

Solve AREs with R>0

LQ problem

A numerical example

H2 control problem : Definite R


Kleinman algorithm for H2

Question: Can we use Kleinman algorithm to solve H Riccati equations directly?

Kleinman algorithm for H

Diverge !!

The Kleinman algorithm cannot be used to solve H Riccati equations directly!


Direct methods


Iterative methods

Traditional Newton methods

Difficult to choose an initial condition

Kleinman algorithm

Solve AREs with R>0

LQ problem

New algorithm ??

H control problem : Indefinite R

A numerical example

H2 control problem : Definite R

Sign-indefinite Quadratic

Term

Problem setting

Sign indefinite quadratic

term

CARE Algorithm

Recursive algorithm using LQ Riccati equations , not Lyapunov equations!

Sign definite

quadratic term

Simple initial choice

Algorithm: A Summary

An H CARE A sequence of H2 CAREs

Our algorithm

Convert

A more difficult problemA sequence of less

difficult problem

Convert

Convergence

• Global convergence is guaranteed provided the H control problem is solvable

A monotone increasing matrix sequence is constructed to approximate the stabilizing solution of an H ARE.

• Local quadratic rate of convergence A typical feature of Newton’s method

Game theoretic interpretation

• Recall

• Player u: minimize J; player w: maximize J. • Strategies for player u and w:

uk+1 solves LQ, not game theoryproblem, when fixedwk is being used

is the monotone increasing matrix sequence in our algorithm

Comparison of results

Our algorithm• Reduce an ARE with an

indefinite quadratic term to a series of AREs with a negative quadratic term;

• Simple choice of the initial condition;

• A monotone non-decreasing matrix sequence.

Easy Newton method (Kleinman)

• Reduce an ARE with an positive definite quadratic term to a series of Lyapunov equations;

• Difficult to choose an initial condition;

• A monotone non-increasing matrix sequence.

For LQ H2 problems For LQ game problems

Periodic Equations--H2

• Some problems are periodic, e.g. satellite control

• H2 periodic Riccati equations potentially have stabilizing periodic solution

• Computational procedures are current research topic• Kleinman-like algorithm for time-varying Riccati equations

over a finite interval predates Kleinman algorithm• Kleinman-like algorithm for periodic time-varying Riccati

equations over infinite interval yields stabilizing periodic solution as limit of solution of periodic Lyapunov differential equations

Periodic Equations--H

• H periodic Riccati equations potentially have

stabilizing periodic solution

• Solution can be found by solving a sequence of H2 periodic Riccati equations (Kleinman-like algorithm will work for each one of these)

• Game theoretic interpretation exists• No surprises in relation to the time-invariant case:

– Local Quadratic Rate of Convergence.

The algorithm carries over for the periodic case!!

Outline


• Solving H Riccati equations

• Nonlinear Extension– Isaacs and HJB equations– Recursive solution of Isaacs via HJB equations– Quadratic convergence and game theoretic interpretation

• Solving DARE• Conclusions and Future Work

LQ and generalization

Disturbance input

One player game

LQ

HJB

Nonlinear optimal control

Isaacs

NonlinearLQ Game

Riccati equations

LQ problem Linear H-infinity control problem

Nonlinear H-infinity control problem

22

Summary of nonlinear result

H2 problem solution H problem solutionIteration

HJB problem solution Isaacs problem solutionIteration can be found

Linear-quadratic to Nonlinear-nonquadratic

Linear-quadratic to Nonlinear-nonquadratic

Problem Setting

Sign indefinite term

Nonlinear algorithm

HJB

Isaacs

Nice initial condition

Question: How to solve HJB?

Solving HJB

H2 problem solution

H problem solution

HJB problem solution

Isaacs problem solution

Linear PDE iteration to give HJB solution stems from 1967 approx, though not carefully done for infinite time problem.

Kleinmaniteration Iteration

Linear PDEIteration Iteration

exists

Other methods to solve HJB do exist

Convergence

• Local convergence is guaranteed provided the H infinity control problem is locally solvable

A monotone non-decreasing function sequence is constructed to approximate the stabilizing solution of an Isaacs equation.

• Local quadratic rate of convergence

Game theoretic interpretation

• Recall

• Player u: minimize J; player w: maximize J.

• Vk: The monotone increasing function sequence in algorithm

• Strategies for players u and w:

Comparison ResultsExisting methods

• Taylor Expansion Method– Stability not guaranteed– Converges slowly

• Galerkin Approximation Method – Difficult to initialize

• The Method of Characteristics– Converges slowly

• Other Methods

Our algorithm• Local quadratic rate of

convergence• Simple initial choice • A natural game

theoretic interpretation• Supposed to have a

higher numerical accuracy and reliability

Example (van der Schaft)

Figure compares exact solution, iterations from method of characteristics, and iterations from our method.

A numerical example (continued)

Outline

• Global Motivation• Solving H Riccati equations• Nonlinear Extension• Solving H Discrete-Time Riccati Equations

o Discrete game problemo Mappings between generalized DARE and generalized

CAREo The algorithm to solve DARE


Discrete game problem

Discrete game problem

Matrix inverse includes unknown variable P !!!

Mappings between DARE and CARE

• Why mapping??

• Stem from approximate 1960’s

• Mappings between generalized CARE and generalized DARE

Generalized DARE

Generalized CARE


Assumption: Matrix A has no eigenvalue at -1

LHS: parametersand solution of CARE

RHS: parametersand solution of DARE

Other mappings exist !!

A Summary for Mappings

Parameters of CARE Parameters of DARE

Solution of CARE Solution of DARE

Mapping


Mappings exist !!

DARE we want to solve recursively

CARE we have solved recursively

Algorithm to Solve DARE

Given a DARE

Mapping DARE into CARE

Obtain the monotone sequence of the CARE

Generate the monotone sequence of the DARE

Compute the stabilizing solution of the DARE

Algorithm to solve DARE

Monotone sequence in CARE Monotone sequence in DARE

Mapping

Game Theory Interpretation Game Theory Interpretation

Property of DARE Algorithm

• Recursive

• Simple initialization

• Global convergence

• Local quadratic rate of convergence

• Game Theoretic InterpretationProperties of CARE algorithmCarry over !!

Conclusions and future workConclusions • We developed a new algorithm to solve Riccati

equations (Isaacs equations) arising in H control• We proved the global (local) convergence and

local quadratic rate of convergence of our algorithm;

• Our algorithm has a natural game theoretic interpretation.

Conclusions and future workFuture work• More general Isaacs equations

– Time-varying, periodic, other system structure

• Viscosity case– Nonsmooth solutions

• Stochastic case– Random noise in system, modifies Isaacs

• Zero(Nonzero)-sum Multi-player game– Coupled HJB/Isaacs, but iteration style similar in

existing algorithms

• Discrete Isaacs equations

Acknowledgement

• Andras Varga

• Marco Lovera

• Matt James

• Minyi Huang

• Weitian Chen

Documents

Game Theoretic Approaches to Solve H Problems