Krylov-Subspace Matrix Solution Methods...Mr Mp Mr Mp p rpprp Mp Mp Mp Mp ++ ++ ++ = =−∑ ⇒=− If M =⊥MrTkthen +1 Mpjj

Introduction to Simulation - Lecture 6

Krylov-Subspace Matrix Solution Methods

Jacob White

Thanks to Deepak Ramaswamy, Michal Rewienski, and Karen Veroy

Outline

• General Subspace Minimization Algorithm– Review orthogonalization and projection formulas

• Generalized Conjugate Residual Algorithm– Krylov-subspace – Simplification in the symmetric case.– Convergence properties

• Eigenvalue and Eigenvector Review– Norms and Spectral Radius– Spectral Mapping Theorem

SMA-HPC ©2003 MIT

Arbitrary Subspace methods

Approach to Approximately Solving Mx=b

{ }0 1Approximate as a weighted sum of ,...,kkx w w −

1

0

kk

i ii

x wα−

=

⇒ = ∑

{ }1 10 1

0 1

0 1

,..., ,...,

N N

k

k

k

w w

w ww w

−

−

−

⎧ ⎫⎡ ⎤ ⎡ ⎤⎪ ⎪⎢ ⎥ ⎢ ⎥

≡⎨ ⎬⎢ ⎥ ⎢ ⎥⎪ ⎪⎢ ⎥ ⎢ ⎥⎣ ⎦ ⎣ ⎦⎩ ⎭

Pick a k-dimensional

Subspace

SMA-HPC ©2003 MIT


Residual Minimization

1

0

kk k

i ii

r b Mx b Mwα−

=

= − =⇒ − ∑

1

0If

kk

i ii

x wα−

=

= ∑

Residual Minimizing idea: pick ' to minimizei sα

( ) ( )1 12

20 0

Tk kTk k ki i i i

i ir r r b Mw b Mwα α

− −

= =

⎛ ⎞ ⎛ ⎞≡ = − −⎜ ⎟ ⎜ ⎟

⎝ ⎠ ⎝ ⎠∑ ∑

The residual is de as fined k kr b Mx≡ −

SMA-HPC ©2003 MIT

Computational ApproachArbitrary Subspace methods


( ) ( ) ( ) ( )or orthogonal to0 , iT

i j jMwM Mw i j wM w = ≠

212

20 2

Minimizing is easy ifk

ki i

ir b Mwα

−

=

= − ∑

{ } { }0 1 0 1,..., ,...,k kspan p p span w w− −={ }0 1Create a set of vectors ,..., such thatkp p −

( ) ( )and 0,Ti jMp Mp i j= ≠

SMA-HPC ©2003 MIT

{ }0 1Given and a set of search directions , ,..., kM b w w −

1) Generate by orthogonalizing 's M 'j jp w s

( ) ( )( ) ( )

1

0

For 0 to 1 T

jj i

j j iTi i i

Mw Mpj k p w p

Mp Mp

−

=

= − = −∑

Algorithm StepsArbitrary Subspace methods


2) compute the minimizing solution kr x

( ) ( )( ) ( )

( ) ( )( ) ( )

01 1

0 0

T Tk k

i iki iT T

i ii i

i

i i

r Mp Mpx p p

Mp Mp Mp Mp

r− −

= =

= =∑ ∑

SMA-HPC ©2003 MIT

Algorithm Steps by PictureArbitrary Subspace methods


1) orthogonalize the 'iMw s

0w 1w 2w 3wp0 p1 p2 p3

0M p

1M p0r

2) compute the minimizing solution kr x

SMA-HPC ©2003 MIT

Arbitrary Subspace Solution Algorithm

Minimization Algorithm

For j = 0 to k-1

( ) ( )1T

j j

j j

Mp Mpp p← Normalize

j jp w=

( ) ( )jj iij

TMpp pMpp← −

OrthogonalizeSearch Direction

( ) ( )1 jjj

T

jj rx x pMp+ = + Update Solution

0 0r b Ax= −

( ) ( )1 jjj

T

jj r Mpr Mr p+ = −

For i = 0 to j-1

Update Residual

SMA-HPC ©2003 MIT


Subspace Selection

0 1Criteria for selecting ,..., kw w −

Criteria

{ }0 1All that matters is the ,..., kspan w w −

{ }10 1in the ,..., for kA b span w w k N−

−≈

{ }1One choice, unit vectors, ,...,kkx span e e∈

Generates the QR algorithm if k=N

1

0' such that is small

kk

i i ii

s b Mx b Mwα α−

=

∃ − = − ∑

Can be terrible if k < N

SMA-HPC ©2003 MIT


Subspace Selection Historical Development

( ) 1Consider minimizing 2

T Tf x x Mx x b= −

Assume (symmetric) and 0 (pos. def)T TM M x Mx= >

( ) 1 minimizesx f x Mx b x M b f−∇ = − ⇒ =

Steepest descent directions for f, but f is not residual

{ } ( ) ( ){ }0 10 1Pick ,..., ,..., k

k x xspan w w span f x f x −− = ∇ ∇

Does not extend to nonsymmetric, non pos def case

SMA-HPC ©2003 MIT


Subspace Selection Krylov Subspace

( ) ( ){ } { }0 1 0 1Note: ,..., ,...,k kx xspan f x f x span r r− −∇ ∇ =

10

0then

kk i

ii

r r Mrα−

=

= − ∑

{ } { }0 10 1If: ,..., ,..., k

kspan w w span r r −− =

{ } { }0 1 0 0 1 0and ,..., , ,...,k kspan r r span r Mr M r

Krylov Subspace

− −=

SMA-HPC ©2003 MIT

Krylov MethodsThe kth step of GCR

The Generalized Conjugate Residual Algorithm

( ) ( )( ) ( )

Tkk

k Tk k

r Mp

Mp Mpα =

Determine optimal stepsize in kth search direction

1k kk kx x pα+ = +

1k kk kr r Mpα+ = −

Update the solution and the residual

( ) ( )( ) ( )

11

10

Tkkjk

k jTj j j

Mr Mpp r p

Mp Mp

++

+=

= −∑Compute the new

orthogonalizedsearch direction

SMA-HPC ©2003 MIT

Krylov MethodsAlgorithm Cost for iter k


( ) ( )( ) ( )

Tkk

k Tk k

r Mp

Mp Mpα = Vector inner products, O(n)

Matrix-vector product, O(n) if sparse1k k

k kx x pα+ = +1k k

k kr r Mpα+ = − Vector Adds, O(n)

( ) ( )( ) ( )

11

10

Tkkjk

k jTj j j

Mr Mpp r p

Mp Mp

++

+=

= −∑ O(k) inner products, total cost O(nk)

If M is sparse, as k (# of iters) approaches n,3total cost ( ) (2 ) .... ( ) ( )O n O n O kn O n= + + + =

Better Converge Fast!

SMA-HPC ©2003 MIT

Krylov MethodsSymmetric Case


An Amazing fact that will not be derived

( ) ( )( ) ( )

( ) ( )( ) ( )

1 11 1

1 10

T Tk kkj kk k

k j k kT Tj k kj j

Mr Mp Mr Mpp r p p r p

Mp MpMp Mp

+ ++ +

+ +=

= − ⇒ = −∑

1If then T k jM M r Mp j k+= ⊥ <

Orthogonalization in one stepIf k (# of iters ) n, then symmetric,

sparse, GCR is O(n2 )Better Converge Fast!

SMA-HPC ©2003 MIT

Nodal FormulationKrylov Methods“No-leak Example”Insulated bar and Matrix

m2 11 2

11 2

M

−⎡ ⎤⎢ ⎥−⎢ ⎥⎢ ⎥−⎢ ⎥−⎣ ⎦

(0)T (1)TNear End

TemperatureFar End

Temperature

Incoming Heat

Discretization

Nodal Equation

Form

SMA-HPC ©2003 MIT

Nodal FormulationKrylov Methods“No-leak Example”

Circuit and Matrix

1 2 3 4 1m − m

m

2 11 2

11 2

M

−⎡ ⎤⎢ ⎥−⎢ ⎥⎢ ⎥−⎢ ⎥−⎣ ⎦

Nodal Equation

Form

SMA-HPC ©2003 MIT

Nodal FormulationKrylov Methods“leaky” Example

Conducting bar and Matrix

m2.01 1

1 2.011

1 2.01

M

−⎡ ⎤⎢ ⎥−⎢ ⎥⎢ ⎥−⎢ ⎥−⎣ ⎦

(0)T (1)TNear End

TemperatureFar End

TemperatureDiscretization

Nodal Equation

Form

SMA-HPC ©2003 MIT

Nodal FormulationKrylov Methods“leaky” Example

Circuit and Matrix

1 2 3 4 1m − m

m2.01 1

1 2.011

1 2.01

M

−⎡ ⎤⎢ ⎥−⎢ ⎥⎢ ⎥−⎢ ⎥−⎣ ⎦

Nodal Equation

Form

SMA-HPC ©2003 MIT

GCR Performance(Random Rhs)

0 10 20 30 40 50 6010-4

10-3

10-2

10-1

100

Iteration

Leaky

Insulating

Plot of log(residual) versus Iteration

RESIDUAL

SMA-HPC ©2003 MIT

0 5 10 15 20 25 30 35 40 45 5010-5

10-4

10-3

10-2

10-1

100

GCR Performance(Rhs = -1,+1,-1,+1….)

Iteration

RESIDUAL

Leaky

Insulating

Plot of log(residual) versus Iteration

SMA-HPC ©2003 MIT

Krylov Subspace Methods

( )1 0 0

0

kk i

i ki

x M r M rα ξ+

=

= =∑

0Note: for any 0α ≠

{ } { }0 1 0 0 0 00span , = span ,r r r Mr r Mrα= −

{ } { }0 0 00 span ,..., span ,f ,...,I k

kw w r Mr M r=

( )( )1 0 1 0 0

0

kk i

i ki

r r M r I M M rα ξ+ +

=

= − = −∑kth order polynomial

Polynomial Approach

Convergence Analysis

SMA-HPC ©2003 MIT

Krylov MethodsConvergence Analysis

Basic Properties

0 jIf for all in GCR, thenj kα ≠ ≤{ } { }0 0 0

0 11 span , ,..., span ,) ... , , kkp p p r Mr M r=

th1 02) is the k ( ) , o rderkk kx M rξ ξ+ =

21

2polynomial which minimizes kr +

1 1 0 03) ( )k kkr b Mx r M M rξ+ += − = −

( ) ( )0 01( )k kI M M r M rξ += − ≡℘

( ) ( )01where is the order poly 1 th

k M r k+℘ +( )21

12minimizing subject to 0 = 1 k

kr ++℘

SMA-HPC ©2003 MIT


Optimality of GCR poly

GCR Optimality Property

( )k+1 polynomial such that 0 =1℘

ThereforeAny polynomial which satisfies the zero constraint can be used

to get an upper bound on

21

2

kr +

2 21 0k+1 k+12 2

where is ( a) ny order k thr M r k+ ≤ ℘ ℘

SMA-HPC ©2003 MIT

Basic DefinitionsEigenvalues and Vectors Review

Eigenvalues and eigenvectors of a matrix M satisfy

eigenvector

eigenvalue

ii iu uM λ=

Or, is an eigenvalue of ifi Mλis singulariM Iλ−

( ) = 0ii uM Iλ−is an eigenvector of ifiu M

SMA-HPC ©2003 MIT

Basic DefinitionsEigenvalues and Vectors Review Examples

1.1 11 1.1

−⎡ ⎤⎢ ⎥−⎣ ⎦

1 1 0 01 1 0 0

0 0 1 10 0 1 2

−⎡ ⎤⎢ ⎥−⎢ ⎥⎢ ⎥−⎢ ⎥−⎣ ⎦

Eigenvalues?Eigenvectors?

11

21 22

1 1

0 0

0

N NN NN

MM M

M M M−

⎡ ⎤⎢ ⎥⎢ ⎥⎢ ⎥⎢ ⎥⎢ ⎥⎣ ⎦

What about a lower triangular matrix

SMA-HPC ©2003 MIT

A Simplifying Assumption

Eigenvalues and Vectors Review

Almost all NxN matrices have N linearly independent Eigenvectors

1 2 3 Nu u u u⎡ ⎤↑ ↑ ↑ ↑⎢ ⎥⎢ ⎥⎢ ⎥↓ ↓ ↓ ↓⎣ ⎦

M

1 1 2 2 3 3 N Nu u u uλ λ λ λ⎡ ⎤↑ ↑ ↑ ↑⎢ ⎥= ⎢ ⎥⎢ ⎥↓ ↓ ↓ ↓⎣ ⎦

The set of all eigenvalues of M is known as the Spectrum of M

SMA-HPC ©2003 MIT

A Simplifying Assumption Continued


Almost all NxN matrices have N linearly independent Eigenvectors

MU U=1 0 0

0 00 0 N

λ

λ

⎡ ⎤⎢ ⎥⎢ ⎥⎢ ⎥⎣ ⎦

1 1orM MU U U Uλ λ− −= =Does NOT imply distinct eigenvalues, can equal i jλ λDoes NOT imply is nonsingular M

SMA-HPC ©2003 MIT

Spectral RadiusEigenvalues and Vectors Review

( )Re λ

( )Im λ

iλ

The spectral Radius of M is the radius of the smallest circle, centered at the origin, which encloses all of

M’s eigenvalues

SMA-HPC ©2003 MIT

Heat Flow ExampleEigenvalues and Vectors Review

(1)T(0)T

Unit Length Rod

Incoming Heat

+-

+-

(0)sv T= (1)sv T=

1T NT

SMA-HPC ©2003 MIT

Heat Flow Example Continued


2 1 0 01 2 0

0 10 0 1 2

−⎡ ⎤⎢ ⎥−⎢ ⎥⎢ ⎥−⎢ ⎥−⎣ ⎦

Eigenvalues N=20

SMA-HPC ©2003 MIT

Heat Flow Example Continued


Four Eigenvectors – Which ones?

SMA-HPC ©2003 MIT

Spectral Mapping Theorem

Useful Eigenproperties

Given a polynomial( ) 0 1

ppf x a a x a x= + + +…

Apply the polynomial to a matrix

( ) 0 1p

pf M a a M a M= + + +…Then

( )( ) ( )( )spectrum f M f spectrum M=

SMA-HPC ©2003 MIT

Spectral Mapping Theorem Proof


Note a property of matrix powers1 1 2 1MM U U U U U Uλ λ λ− − −= =

Apply to the polynomial of the matrix( ) 1 1 1

0 1p

pf M a UU aU U a U Uλ λ− − −= + + +…

1p pM U Uλ −⇒ =

Factoring ( ) ( ) 10 1

pp

Diagonal

f M U a I a a Uλ λ −= + + +…

( ) ( )0 1p

pf M U U a I a aλ λ= + + +…

SMA-HPC ©2003 MIT

Spectral Decomposition


Decompose arbitrary x in eigencomponents1 1 2 2 N Nx u u uα α α= + + +…

11

N

U x U xα

αα

−

⎡ ⎤⎢ ⎥ = ⇒ =⎢ ⎥⎢ ⎥⎣ ⎦

Applying M to x yeilds(

Compute by solving

)1 1 2 2 N NMx M u u uα α α= + + +…1 1 1 2 2 2 N N Nu u uα λ α λ α λ= + + +…

SMA-HPC ©2003 MIT


Important Observations

1) The GCR Algorithm converges to the exact solution in at most n steps

( )where . i Mλ λ∈

( ) 0 0 and therefore 0nn M r r⇒ ℘ = =

2) If M has only q distinct eigenvalues, the GCR Algorithm converges in at most q steps

( ) ( )( ) ( )1 2Proof: Let = ... q qx x x xλ λ λ℘ − − −

( ) ( )( ) ( )1 2Proof: Let = ... n nx x x xλ λ λ℘ − − −

Summary

• Arbitrary Subspace Algorithm– Orthogonalization of Search Directions

• Generalized Conjugate Residual Algorithm– Krylov-subspace – Simplification in the symmetric case.– Leaky and insulating examples

• Eigenvalue and Eigenvector Review– Spectral Mapping Theorem

• GCR limiting Cases– Q-step guaranteed convergence

Documents

Krylov-Subspace Matrix Solution Methods...Mr Mp Mr Mp p rpprp Mp Mp Mp Mp ++ ++ ++ = =−∑ ⇒=− If M =⊥MrTkthen +1 Mpjj