An introduction to iterative projection methods

Eigenvalue problems

Luiza Bondar

the 23rd of November -2005

4th Seminar

Introduction (Erwin)

Perturbation analysis (Nico)

Direct (global) methods (Peter)

Introduction to projection methods (Luiza) (theoretical

background)

Krylov subspace methods 1 (Mark)

Krylov subspace methods 2 (Willem)

Outline

• Introduction

• The power method

• Projection Methods

• Subspace iteration

• Summary

Direct methods (Schur decomposition, QR iteration, Jacobi method,

method of Sturm sequences )

• compute all the eigenvalues and the corresponding eigenvectors

What if we DON’T need all the eigenvalues?

Example : compute the page rank of the www documents

Introduction

WEB: a graph (pages are nodes links are edges )

Introduction

Web graph: 1.4 bilion nodes (pages) 6.6 bilion edges (links)

page rank of page i : the probability that a surfer will visit the page i

The page rank is a dominant vector of a sparse 1.4 bilion X 1.4 bilionmatrix.

It makes little sense to compute all the eigenvectors.

page rank : vector with dimension N=1.4 bilion

Introduction

The power method

• computes the dominant eigenvalue and an associated eigenvector

Some background

consider that A has p distinct eigenvalues.

1 2n

pM M M

dim i iM

n niP :

kki i i iA P P I D 0i i

iD semi-simple i 0iD

iis the algebraic multiplicity of i

iPis the projection onto iM

The power method

consider that the dominant eigenvalue is unique and is semi-simple 1

initial vector such that

convergence ?NO YES

1 k

1 kx v

1 0P v 0

1Avk k 1

1k k

k v Av

0v

0

1v A vkk

k ( ) 1Avk compute an

dtake

The power method

initial vector 0vn , 1 2

npM M M 0 0

1

v vp

ii

P

0

1v A vkk

k

kki i i iA P P I D use

11 0 0

2 1

1v v v

k pk

k i i i ikik

P P D P

then 1 01 0

1v v

vk PP

and 1k ( )1Av vk k k

0

convergence of each term in given by

Σ1

i

The power method is used by to compute the page rank.

The power method

• the convergence of the method is given by

• the convergence might be very slow if are close from one another

• if the dominant eigenvalue is multiple but semi-simple, then the algorithm provides only one eigenvalue and a corresponding eigenvector

• does not converge if the dominant eigenvalue is complex and theoriginal matrix is real (2 eigenvalues with the same modulus)

2

1

1 2,

IMPROVEMENT : the shifted power method

LED TO : projection methods

The power method

Shifted power method A A I x x

Example

• let be the dominant eigenvalue of a matrix that has an egenvalue • then the power method does not converge when applied to • but the power method converges for a shift (e.g. )

1 1 1i

A IA

A

Other variants of the power method

• inverse power method (iterates with )

• inverse power method with shift

-1A smallest eigenvalue

eigenvalue closest to the shift

The power method

• inverse power method

A LU1 -1 -1

k-1v U L vkk

then converges to the smallest eigenvalue and converges to an

associated eigenvector k vk

• inverse power method with shift

-A I LU

1 1-

-1 -1 -1k-1 k-1v A I v U L vk

k k

then converges to and converges to an eigenvector associated with

k 1

vk

The power method

• does not converge if the dominant eigenvalue is complex and the original matrix is real (2 eigenvalues with the same modulus)

But after a certain k

1v vk k,

IDEA: extract the vectors by performing a projection into the subspace

contains approximations to the complex par of eigenvectors

0 1

1 0A=

1 i

1

i

2 i

1

i

power method

1

1

1

1

vk 1vk

Projection methods (Introduction)

1u vk

2 1u vk

ufind and such that Au u

1 1 2 2u u u

• impose 2 more constrains• one choice is to impose orthogonality conditions (Galerkin) i.e.,

1 , 2• introduce 2 degrees of freedom

1Au u u and

2Au u u

1u

2u

u

Auprojection method

Projection methods (Introduction)

Generalizationn nA

, nK L

dim K=dim L=m

find and such that nu Au u

A projection technique seeks an approximate eigenpar and such that

Ku

K uuA ~~~

L uuA ~~~

• orthogonal projection

or• oblique projection

K: the right subspace, L: the left subspace

A way to construct K is Krylov subspace

(inspired by the power method)

10 0 0, , ,m mK v Av A v

Projection methods (orthogonal)

Consider an orthonormal basis of K and

1 2, , , mv v v

1 2| | | ,mV v v v

Kuthe approximate can be written as u Vy

K uuA ~~~ , 0, Au u v K v , 0, 1, ,jv j m AVy vy

HV AVy y

n mV

:m HB V AV

i i

ix i iu Vx

• eigenvalue of then eigenvalue of

• eigenvector of then eigenvector of

mB

mB

A

A

Arnoldi’s method and the hermitian Lanczos algorithm are orthogonalprojection methods

Projection methods (oblique)

L uuA ~~~ KuSearch for and such that

1 2, , , mv v v 1 2| | | ,mV v v vn mV

the approximate can be written as u Vy

1 2| | | ,mW w w wn mW 1 2, , , mw w w

orthonormal basis of K

orthonormal basis of L

and are such that (biorthogonal) V W HW V I

Ku

The condition leads to the approximate eigenvalue

problem

L uuA ~~~

HW AVy y

The nonhermitian Lanczos alghoritm is an oblique projection method.

Projection methods (orthogonal)

How accurate can an orthogonal projection method be?

, uexact eigenpar

then 2 22

H Hk k kV AV I V u P A I P I P u

projection onto KkP

kP u

uK

u

ku-P u

Projection methods (orthogonal)

Hermitian case

kP u

uK

u

2( )

sin 1 sin ,,

k kP A I P

d

kuu, u

AP u

2

2sin ,A uI u

Subspace iteration

• generalization of the power method• start with an initial system of m vectors instead of

only one vector (power method)

• compute the matrix

0 1 2, , , mX x x x

0k

k X A X

If each of the m vectors is normalised in the same way as for the power method, then each of these vectors will converge to the SAMEeigenvector associated with the dominant eigenvalue (provided that )1 0, 1,Pxi i m

Note looses its linear independenceIDEA: restore the linear independence by performing aQR factorisation

0k

k X A X

Subspace iteration

0 1 2, , , mX x x x

0 0 0X Q R :0 0X Q

-1X AXk k

-1 -1 1 -1H H HH X AX Q AQk k k k k

start with

QR factorize 0X take

compute

convergence ?X Q Rk k k

:X Qk k

recover the first m eigenvalues

and corresponding eigenvectors

of A from Hk

NO YES

Subspace iteration

• the i-th column of converges to a Schur vector associated with the eigenvalue • the convergence of the column is given by the factor • the speed of convergence for an eigenvalue depends on how close isit to the next one

Variants of the subspace iteration method

• take the dimension of the subspace m larger than nev number of

eigenvalues wanted

• perform “locking” i.e., as soon as an eigenvalue has converged stop multiplying with A the corresponding vector in the subsequent

iterations

0Xi

1i

i

Subspace iteration

Some very theoretical result on residual norm

Xk kS 00 XS P

Pk kSprojection onto

projection onto the subspace spanned by the eigenvectors associated with the first m eigenvalues of

Then for any eigenvalue of there is an unique

such that and ui A

A

assume that are linearly independent , 1, ,Pxi i m

0si SPs ui i

122

u u sI P

k

mk i i i k

i

0

Summary

• The power method can be used to compute the dominant eigenvalue(real) and a corresponding eigenvector.

• Variants of the power method can compute the smallest eigenvalue orthe eigenvalue closest to a given number (shift).

• General projection methods consist in approximating the eigenvectors of a matrix with vectors belonging to a subspace of approximants with dimension smaller than the dimension of the matrix.

• Subspace iteration method is a generalization of the power method thatcomputes a given number of dominant eigenvalues and their corresponding eigenvectors.

Last minute questions answered by

Tycho van NoordenSorin Pop