How and How Not to Compute the

Exponential of a Matrix

Nick HighamSchool of Mathematics

The University of Manchester

highammamanacuk

httpwwwmamanacuk~higham

History amp Properties Applications Methods

Outline

1 History amp Properties

2 Applications

3 Methods

MIMS Nick Higham Matrix Exponential 2 41

History amp Properties Applications Methods

Cayley and Sylvester

Term ldquomatrixrdquo coined in 1850

by James Joseph Sylvester

FRS (1814ndash1897)

Matrix algebra developed by

Arthur Cayley FRS (1821ndash

1895)

Memoir on the Theory of Ma-trices (1858)

MIMS Nick Higham Matrix Exponential 3 41

History amp Properties Applications Methods

Cayley and Sylvester on Matrix Functions

Cayley considered matrix square

roots in his 1858 memoir

Tony Crilly Arthur Cayley Mathemati-

cian Laureate of the Victorian Age

2006

Sylvester (1883) gave first defini-

tion of f (A) for general f

Karen Hunger Parshall James Joseph

Sylvester Jewish Mathematician in a

Victorian World 2006

MIMS Nick Higham Matrix Exponential 4 41

Laguerre (1867)

Peano (1888)

History amp Properties Applications Methods

Matrices in Applied Mathematics

Frazer Duncan amp Collar Aerodynamics Division of

NPL aircraft flutter matrix structural analysis

Elementary Matrices amp Some Applications toDynamics and Differential Equations 1938

Emphasizes importance of eA

Arthur Roderick Collar FRS

(1908ndash1986) ldquoFirst book to treat

matrices as a branch of applied

mathematicsrdquo

MIMS Nick Higham Matrix Exponential 6 41

History amp Properties Applications Methods

Matrices in Applied Mathematics

Frazer Duncan amp Collar Aerodynamics Division of

NPL aircraft flutter matrix structural analysis

Elementary Matrices amp Some Applications toDynamics and Differential Equations 1938

Emphasizes importance of eA

Arthur Roderick Collar FRS

(1908ndash1986) ldquoFirst book to treat

matrices as a branch of applied

mathematicsrdquo

MIMS Nick Higham Matrix Exponential 6 41

History amp Properties Applications Methods

Formulae

A isin Cntimesn

Power series Limit Scaling and squaring

I + A +A2

2+

A3

3+ middot middot middot lim

srarrinfin

(I + As)s (eA2s)2s

Cauchy integral Jordan form Interpolation

1

2πi

int

Γez(zI minus A)minus1 dz Zdiag(eJk )Zminus1

nsum

i=1

f [λ1 λi ]

iminus1prod

j=1

(A minus λj I)

Differential system Schur form Padeacute approximation

Y prime(t) = AY (t) Y (0) = I Qdiag(eT )Qlowast pkm(A)qkm(A)minus1

MIMS Nick Higham Matrix Exponential 8 41

History amp Properties Applications Methods

Properties (1)

Theorem

For AB isin Cntimesn e(A+B)t = eAteBt for all t if and only if

AB = BA

Theorem (Wermuth)

Let AB isin Cntimesn have algebraic elements and let n ge 2

Then eAeB = eBeA if and only if AB = BA

MIMS Nick Higham Matrix Exponential 9 41

History amp Properties Applications Methods

Properties (1)

Theorem

For AB isin Cntimesn e(A+B)t = eAteBt for all t if and only if

AB = BA

Theorem (Wermuth)

Let AB isin Cntimesn have algebraic elements and let n ge 2

Then eAeB = eBeA if and only if AB = BA

Theorem

Let A isin Cntimesn and B isin C

mtimesm Then eAoplusB = eA otimes eB where

Aoplus B = Aotimes Im + In otimes B

MIMS Nick Higham Matrix Exponential 9 41

History amp Properties Applications Methods

Properties (2)

Theorem (Suzuki)

For A isin Cntimesn let

Tr s =

[

rsum

i=0

1

i

(

A

s

)i]s

Then

eA minus Tr s leAr+1

sr (r + 1)eA

and limrrarrinfin Tr s(A) = limsrarrinfin Tr s(A) = eA

MIMS Nick Higham Matrix Exponential 10 41

History amp Properties Applications Methods

Outline

1 History amp Properties

2 Applications

3 Methods

MIMS Nick Higham Matrix Exponential 11 41

History amp Properties Applications Methods

Application Control Theory

Convert continuous-time system

dx

dt= Fx(t) + Gu(t)

y = Hx(t) + Ju(t)

to discrete-time state-space system

xk+1 = Axk + Buk

yk = Hxk + Juk

Have

A = eFτ B =

(int τ

0

eFtdt

)

G

where τ is the sampling period

MATLAB Control System Toolbox c2d and d2c

MIMS Nick Higham Matrix Exponential 12 41

History amp Properties Applications Methods

Psi Functions Definition

ψ0(z) = ez ψ1(z) =ez minus 1

z ψ2(z) =

ez minus 1minus z

z2

ψk+1(z) =ψk(z)minus 1k

z

ψk(z) =infin

sum

j=0

z j

(j + k)

MIMS Nick Higham Matrix Exponential 13 41

History amp Properties Applications Methods

Psi Functions Solving DEs

y isin Cn A isin C

ntimesn

dy

dt= Ay y(0) = y0 rArr y(t) = eAty0

MIMS Nick Higham Matrix Exponential 14 41

History amp Properties Applications Methods

Psi Functions Solving DEs

y isin Cn A isin C

ntimesn

dy

dt= Ay y(0) = y0 rArr y(t) = eAty0

dy

dt= Ay + b y(0) = 0 rArr y(t) = t ψ1(tA)b

MIMS Nick Higham Matrix Exponential 14 41

History amp Properties Applications Methods

Psi Functions Solving DEs

y isin Cn A isin C

ntimesn

dy

dt= Ay y(0) = y0 rArr y(t) = eAty0

dy

dt= Ay + b y(0) = 0 rArr y(t) = t ψ1(tA)b

dy

dt= Ay + ct y(0) = 0 rArr y(t) = t2ψ2(tA)c

MIMS Nick Higham Matrix Exponential 14 41

History amp Properties Applications Methods

Exponential Integrators

Consider

y prime = Ly + N(y)

N(y(t)) asymp N(y(0)) implies

y(t) asymp etLy0 + tψ1(tL)N(y(0))

Exponential Euler method

yn+1 = ehLyn + hψ1(hL)N(yn)

Lawson (1967) recent resurgence

MIMS Nick Higham Matrix Exponential 15 41

History amp Properties Applications Methods

The Average Eye

First order character of optical system characterized by

transference matrix T =[

S0

δ1

]

isin R5times5 where S isin R

4times4 is

symplectic ST JS = J where J =[

0minusI2

I20

]

Average mminus1summ

i=1 Ti is not a transference matrix

Harris (2005) proposes the average exp(mminus1summ

i=1 log(Ti))

MIMS Nick Higham Matrix Exponential 16 41

History amp Properties Applications Methods

The Average Eye

First order character of optical system characterized by

transference matrix T =[

S0

δ1

]

isin R5times5 where S isin R

4times4 is

symplectic ST JS = J where J =[

0minusI2

I20

]

Average mminus1summ

i=1 Ti is not a transference matrix

Harris (2005) proposes the average exp(mminus1summ

i=1 log(Ti))

For Hermitian pos def A and B Arsigny et al (2007) define

the log-Euclidean mean

E(AB) = exp(12(log(A) + log(B)))

MIMS Nick Higham Matrix Exponential 16 41

History amp Properties Applications Methods

Beyond Matrices

GluCat library generic library of C++ templates for

universal Clifford algebras exp log square root trig

functions

httpglucatsourceforgenet

Group exponential of a diffeomorphism in

computational anatomy to study variability among

medical images (Bossa et al 2008)

MIMS Nick Higham Matrix Exponential 17 41

History amp Properties Applications Methods

Outline

1 History amp Properties

2 Applications

3 Methods

MIMS Nick Higham Matrix Exponential 18 41

History amp Properties Applications Methods

Sengupta (Adv Appl Prob 1989)

Note that eT is a power series in T This means that a wide

variety of methods in linear algebra can also be used to

evaluate eT brute force evaluation of the power series

matrix decomposition methods or polynomial methods

based on the Cayley-Hamilton theorem

MIMS Nick Higham Matrix Exponential 19 41

History amp Properties Applications Methods

Sengupta (Adv Appl Prob 1989)

Note that eT is a power series in T This means that a wide

variety of methods in linear algebra can also be used to

evaluate eT brute force evaluation of the power series

matrix decomposition methods or polynomial methods

based on the Cayley-Hamilton theorem

Since evaluation of functions of matrices may be fraught

with difficulties (such as roundoff and truncation errors ill

conditioning near confluence of eigenvalues etc) there is

a distinct advantage in having a rich class of solution

techniques available for finding eT If one method fails to

find an accurate answer one can always fall back on a

different method

MIMS Nick Higham Matrix Exponential 19 41

History amp Properties Applications Methods

CayleyndashHamilton Theorem

Theorem (Cayley 1857)

If AB isin Cntimesn AB = BA and f (x y) = det(xAminus yB) then

f (BA) = 0

p(t) = det(tI minus A) implies p(A) = 0

An =sumnminus1

k=0 cnAk

eA =sumnminus1

k=0 dnAk

MIMS Nick Higham Matrix Exponential 20 41

History amp Properties Applications Methods

Walzrsquos Method

Walz (1988) proposed computing

Ck = (I + 2minuskA)2k

with Richardson extrapolation to accelerate cgce of the Ck

Numerically unstable in practice (Parks 1994)

MIMS Nick Higham Matrix Exponential 21 41

History amp Properties Applications Methods

Diagonalization (1)

A = Zdiag(λi)Zminus1 implies f (A) = Zdiag(f (λi))Z

minus1

But

Z may be ill conditioned (κ(Z ) = ZZminus1 ≫ 1)

A may not be diagonalizable

MIMS Nick Higham Matrix Exponential 22 41

History amp Properties Applications Methods

Diagonalization (2)

gtgt A = [3 -1 1 1] X = funm_ev(Aexp)

X =

147781 -73891

73891 0

gtgt norm(X - expm(A))norm(expm(A))

ans = 13431e-009

gtgt expm_cond(A)

ans = 34676

gtgt [ZD]=eig(A)

Z = D =

07071 07071 20000 0

07071 07071 0 20000

MIMS Nick Higham Matrix Exponential 23 41

History amp Properties Applications Methods

Scaling and Squaring Method

B larr A2s so Binfin asymp 1

rm(B) = [mm] Padeacute approximant to eB

X = rm(B)2sasymp eA

Originates with Lawson (1967)

Ward (1977) algorithm with rounding error analysis

and a posteriori error bound

Moler amp Van Loan (1978) give backward error

analysis allowing choice of s and m

H (2005) sharper analysis giving optimal s and m

MATLABrsquos expm Mathematica NAG Library Mark 22

MIMS Nick Higham Matrix Exponential 24 41

History amp Properties Applications Methods

Padeacute Approximants rm to ex

rm(x) = pm(x)qm(x) known explicitly

pm(x) =m

sum

j=0

(2m minus j)m

(2m) (m minus j)

x j

j

and qm(x) = pm(minusx) Error satisfies

exminusrm(x) = (minus1)m (m)2

(2m)(2m + 1)x2m+1+O(x2m+2)

MIMS Nick Higham Matrix Exponential 25 41

History amp Properties Applications Methods

Scaling and Squaring Method

h2m+1(X ) = log(eminusX rm(X )) =infin

sum

k=2m+1

ck X k

Then rm(X ) = eX+h2m+1(X) Hence

rm(2minussA)2s= eA+2sh2m+1(2

minussA) = eA+∆A

Want ∆AA le u

Moler amp Van Loan (1978) a priori bound for h2m+1

m = 6 2minussA le 12 in MATLAB

H (2005) sharp normwise bound using symbolic

arithmetic and high precision Choose (sm) to

minimize computational cost

MIMS Nick Higham Matrix Exponential 26 41

History amp Properties Applications Methods

Scaling amp Squaring Algorithm (H 2005)

m 3 5 7 9 13

θm 0015 025 095 21 54

for m = [3 5 7 9 13]if A1 le θm X = rm(A) quit end

end

MIMS Nick Higham Matrix Exponential 27 41

History amp Properties Applications Methods

Scaling amp Squaring Algorithm (H 2005)

m 3 5 7 9 13

θm 0015 025 095 21 54

for m = [3 5 7 9 13]if A1 le θm X = rm(A) quit end

end

Alarr A2s with s ge 0 minimal st A2s1 le θ13 = 54

MIMS Nick Higham Matrix Exponential 27 41

History amp Properties Applications Methods

Scaling amp Squaring Algorithm (H 2005)

m 3 5 7 9 13

θm 0015 025 095 21 54

for m = [3 5 7 9 13]if A1 le θm X = rm(A) quit end

end

Alarr A2s with s ge 0 minimal st A2s1 le θ13 = 54A2 = A2 A4 = A2

2 A6 = A2A4

U = A[

A6(b13A6 + b11A4 + b9A2) + b7A6 + b5A4 + b3A2 + b1I]

V = A6(b12A6 + b10A4 + b8A2) + b6A6 + b4A4 + b2A2 + b0I

Solve (minusU + V )r13 = U + V for r13

X = r132s

by repeated squaring

MIMS Nick Higham Matrix Exponential 27 41

History amp Properties Applications Methods

Example

A =

[

1 b

0 minus1

]

eA =

[

e b2(e minus eminus1)

0 eminus1

]

b expm(A) s expm(A)dagger s funm(A)

103 17e-15 8 19e-16 0 19e-16

104 18e-13 11 76e-20 0 38e-20

105 75e-13 15 12e-16 0 12e-16

106 13e-11 18 20e-16 0 20e-16

107 72e-11 21 16e-16 0 16e-16

108 30e-12 25 13e-16 0 13e-16

For b = 108 rm(x)225asymp

(

(1 + 12x)(1minus 1

2x)

)225

with

x = plusmn2minus25 asymp plusmn10minus8

MIMS Nick Higham Matrix Exponential 28 41

History amp Properties Applications Methods

Overscaling

Kenney amp Laub (1998) Dieci amp Papini (2000)

A large A causes a larger than necessary s to be chosen

with a harmful effect on accuracy

exp

([

A11 A12

0 A22

])

=

eA11

int 1

0

eA11(1minuss)A12eA22s ds

0 eA22

MIMS Nick Higham Matrix Exponential 29 41

History amp Properties Applications Methods

Insight

Al-Mohy amp H (2009)Existing method based on analysis in terms of AWhy not instead use Ak1k

ρ(A) le Ak1k le A k = 1 infin

limkrarrinfinAk1k = ρ(A)

MIMS Nick Higham Matrix Exponential 30 41

A =

[

09 500

0 minus05

]

0 5 10 15 2010

0

102

104

106

108

1010

Ak2

Ak2

(A515

2)k

(A10110

2)k

Ak1k2

History amp Properties Applications Methods

Key Bounds (1)

Theorem

For any A isin Cntimesn

∥

∥

∥

infinsum

k=ℓ

ck Ak∥

∥

∥le

infinsum

k=ℓ

|ck |(

At1t)k

where At1t = maxAk1k k ge ℓ ck 6= 0

Proof Use Ak =(

Ak1k)kle

(

At1t)k

MIMS Nick Higham Matrix Exponential 32 41

History amp Properties Applications Methods

Key Bounds (1)

Theorem

For any A isin Cntimesn

∥

∥

∥

infinsum

k=ℓ

ck Ak∥

∥

∥le

infinsum

k=ℓ

|ck |(

At1t)k

where At1t = maxAk1k k ge ℓ ck 6= 0

Proof Use Ak =(

Ak1k)kle

(

At1t)k

MIMS Nick Higham Matrix Exponential 32 41

History amp Properties Applications Methods

Key Bounds (2)

Lemma

If k = pm1 + qm2 with pq isin N and m1m2 isin N cup 0

Ak1k le max(

Ap1p Aq1q)

Proof Let δ = max(Ap1p Aq1q) Then

Ak le Apm1Aqm2

le(

Ap1p)pm1

(

Aq1q)qm2

le δpm1δqm2 = δk

Take pq = r r + 1 for k ge r(r minus 1)

MIMS Nick Higham Matrix Exponential 33 41

History amp Properties Applications Methods

New Scaling and Squaring Algorithm

Truncation bounds use Ak1k instead of A

Roundoff considerations correction to chosen m

Use estimates of Ak where necessary (alg of H amp

Tisseur (2000))

Special treatment of triangular matrices to ensure

accurate diagonal

New alg no slower than expm potentially faster

potentially more accurate

MIMS Nick Higham Matrix Exponential 34 41

0 10 20 3010

minus20

10minus10

100

Relative error

expm

new

0 10 20 30minus20

minus15

minus10

minus5

0

5

log10

of ratio of errors newexpm

0 10 20 30 400

10

20

30

40

50

Values of s

expm

new

0 10 20 30

02

04

06

08

1

Ratio of cost newexpm

History amp Properties Applications Methods

Summary of New Alg

Major benefits in speed and accuracy through using

Ak1k in place of Ak

Overscaling problem ldquosolvedrdquo

Stability of squaring phase remains an open question

MIMS Nick Higham Matrix Exponential 36 41

History amp Properties Applications Methods

Summary of New Alg

Major benefits in speed and accuracy through using

Ak1k in place of Ak

Overscaling problem ldquosolvedrdquo

Stability of squaring phase remains an open question

MIMS Nick Higham Matrix Exponential 36 41

History amp Properties Applications Methods

Frecheacutet Derivative

f (A + E)minus f (A)minus L(AE) = o(E)

L(AE) =

int 1

0

eA(1minuss)EeAs ds

Method based on

f

([

X E

0 X

])

=

[

f (X ) L(X E)0 f (X )

]

Kenney amp Laub (1998) KroneckerndashSylvester alg

Padeacute of tanh(x)x 538n3 (complex) flops

Al-Mohy amp H (2009) eA and L(AE) in only 48n3 flops

MIMS Nick Higham Matrix Exponential 37 41

History amp Properties Applications Methods

In Conclusion

Many applications of f (A) eg control theory

Markov chains theoretical physics

Need better understanding of conditioning of

f (A)

How to exploit structure

Need ldquofactorization-freerdquo methods for large

sparse A

Specialize to f (A)b problem

MIMS Nick Higham Matrix Exponential 38 41

History amp Properties Applications Methods

References I

A H Al-Mohy and N J Higham

Computing the Freacutechet derivative of the matrix

exponential with an application to condition number

estimation

SIAM J Matrix Anal Appl 30(4)1639ndash1657 2009

A H Al-Mohy and N J Higham

A new scaling and squaring algorithm for the matrix

exponential

SIAM J Matrix Anal Appl 31(3)970ndash989 2009

MIMS Nick Higham Matrix Exponential 33 41

History amp Properties Applications Methods

References II

R Bellman

Introduction to Matrix Analysis

McGraw-Hill New York second edition 1970

xxiii+403 pp

Reprinted by Society for Industrial and Applied

Mathematics Philadelphia PA USA 1997 ISBN

0-89871-399-4

M Bossa E Zacur and S Olmos

Algorithms for computing the group exponential of

diffeomorphisms Performance evaluation

In Computer Vision and Pattern Recognition

Workshops 2008 (CVPRW rsquo08) pages 1ndash8 IEEE

Computer Society 2008

MIMS Nick Higham Matrix Exponential 34 41

History amp Properties Applications Methods

References III

T Crilly

Cayleyrsquos anticipation of a generalised CayleyndashHamilton

theorem

Historia Mathematica 5211ndash219 1978

T Crilly

Arthur Cayley Mathematician Laureate of the Victorian

Age

Johns Hopkins University Press Baltimore MD USA

2006

ISBN 0-8018-8011-4

xxi+610 pp

MIMS Nick Higham Matrix Exponential 35 41

History amp Properties Applications Methods

References IV

L Dieci and A Papini

Padeacute approximation for the exponential of a block

triangular matrix

Linear Algebra Appl 308183ndash202 2000

R A Frazer W J Duncan and A R Collar

Elementary Matrices and Some Applications to

Dynamics and Differential Equations

Cambridge University Press Cambridge UK 1938

xviii+416 pp

1963 printing

MIMS Nick Higham Matrix Exponential 36 41

History amp Properties Applications Methods

References V

Glucat Generic library of universal Clifford algebra

templates

|httpglucatsourceforgenet|

N J Higham

The Matrix Function Toolbox

http

wwwmamanacuk~highammftoolbox

MIMS Nick Higham Matrix Exponential 37 41

History amp Properties Applications Methods

References VI

N J Higham

Functions of Matrices Theory and Computation

Society for Industrial and Applied Mathematics

Philadelphia PA USA 2008

ISBN 978-0-898716-46-7

xx+425 pp

R A Horn and C R Johnson

Topics in Matrix Analysis

Cambridge University Press Cambridge UK 1991

ISBN 0-521-30587-X

viii+607 pp

MIMS Nick Higham Matrix Exponential 38 41

History amp Properties Applications Methods

References VII

C S Kenney and A J Laub

A SchurndashFreacutechet algorithm for computing the logarithm

and exponential of a matrix

SIAM J Matrix Anal Appl 19(3)640ndash663 1998

M J Parks

A Study of Algorithms to Compute the Matrix

Exponential

PhD thesis Mathematics Department University of

California Berkeley CA USA 1994

MIMS Nick Higham Matrix Exponential 39 41

History amp Properties Applications Methods

References VIII

K H Parshall

James Joseph Sylvester Jewish Mathematician in a

Victorian World

Johns Hopkins University Press Baltimore MD USA

2006

ISBN 0-8018-8291-5

xiii+461 pp

B Sengupta

Markov processes whose steady state distribution is

matrix-exponential with an application to the GIPH1queue

Adv Applied Prob 21(1)159ndash180 1989

MIMS Nick Higham Matrix Exponential 40 41

History amp Properties Applications Methods

References IX

G Walz

Computing the matrix exponential and other matrix

functions

J Comput Appl Math 21119ndash123 1988

MIMS Nick Higham Matrix Exponential 41 41