Krylov Subspace Methods for Linear Systems and Matrix ...simoncin/siamala.pdf · Convergence...

0.5setgray0

0.5setgray1

Krylov Subspace Methods

for Linear Systems and Matrix Equations

V. Simoncini

Dipartimento di Matematica

Universita di Bolognavaleria@dm.unibo.it

http://www.dm.unibo.it/˜simoncin

Krylov subspace methods – p. 1

Projection methods for large-scale problems

Given the system of n equations

F(x) = 0 x ∈ Rn

- Construct approximation space Km (m = dim(Km) )

- Find x ∈ Km such that x ≈ x

? Projection onto a much smaller space m ¿ n

Approximation process:

residual: r := F(x)

Construct (left) space Lm of dimension m and impose

r ⊥ Lm

Projection methods for large-scale problems

Given the system of n equations

F(x) = 0 x ∈ Rn

- Construct approximation space Km (m = dim(Km) )

- Find x ∈ Km such that x ≈ x

? Projection onto a much smaller space m ¿ n

Approximation process:

residual: r := F(x)

Construct (left) space Lm of dimension m and impose

r ⊥ Lm

Challenges

How to select Km

How to derive good Lm so that it also carries other good properties

General idea:

Construct sequence of approximation spaces Km ⊂ Km+1 such that

xm ∈ Km and xm → x as m → ∞

(in some sense)

Analogously, Lm ⊂ Lm+1

Challenges

How to select Km

General idea:

(in some sense)

Challenges

How to select Km

General idea:

(in some sense)

More specific problems

F(x) = 0

A ∈ Rn×n

Ax− b = 0 (ϕk(A)x− b = 0) where ϕk polynomial of degree k

Ax − λMx = 0 (ϕk(λ, A0, A1, . . . , Ak)x = 0)

AX + XAT + Q = 0

Evaluation of Transfer and other matrix functions

F(x) = 0

A ∈ Rn×n

Ax − λMx = 0 (ϕk(λ, A0, A1, . . . , Ak)x = 0)

AX + XAT + Q = 0

F(x) = 0

A ∈ Rn×n

Ax − λMx = 0 (ϕk(λ, A0, A1, . . . , Ak)x = 0)

AX + XAT + Q = 0

F(x) = 0

A ∈ Rn×n

Ax − λMx = 0 (ϕk(λ, A0, A1, . . . , Ak)x = 0)

AX + XAT + Q = 0

Choosing Lm

Focus on linear system:

Ax = b, A nonsing.

xm ∈ Km:

? Lm = Km If A symmetric positive definite,

rm = b − Axm ⊥ Lm ⇔ ‖x − xm‖A = minex∈Km

? Lm = AKm

rm = b − Axm ⊥ Lm ⇔ ‖rm‖2 = minex∈Km

“Optimal” properties hold for any choice of Km Eiermann & Ernst A.N. ’01

Typically: Lm = Km ⇒ FOM, CG Lm = AKm ⇒ GMRES, MINRES

Choosing Lm

Ax = b, A nonsing.

xm ∈ Km:

? Lm = AKm

Choosing Lm

Ax = b, A nonsing.

xm ∈ Km:

? Lm = AKm

Choosing Lm

Ax = b, A nonsing.

xm ∈ Km:

? Lm = AKm

Relaxing optimality in Lm. Truncation

Example: A is non-normal, but “nice” Lm = AKm

0 10 20 30 40 50 60 70 8010−9

10−8

10−7

10−6

10−5

10−4

10−3

10−2

10−1

space dimension

k = ∞

(-): r ⊥ Lm (- -): r ⊥loc Lm k gives “locality”

Example: A is non-normal, but “nice” Lm = AKm

0 10 20 30 40 50 60 70 8010−9

10−8

10−7

10−6

10−5

10−4

10−3

10−2

10−1

space dimension

k = ∞

(-): r ⊥ Lm (- -): r ⊥loc Lm k gives “locality”

Example: A is non-normal, more “nasty” Lm = AKm

0 10 20 30 40 50 60 70 8010−9

10−8

10−7

10−6

10−5

10−4

10−3

10−2

10−1

space dimension

k = ∞

k = 30

k = 20

k = 10

(-): r ⊥ Lm (- -): r ⊥loc Lm Simoncini &Szyld, Num.Math.’05

Choosing Km

The “classical” approach

Km := Km(A, v) = span{v, Av, . . . , Am−1v}

(e.g., v = b in linear systems)

x ∈ Km, x = pm−1(A)v

Nice Properties:

For m sufficiently large, Km(A, v) invariant for A

Convergence (analysis) in terms of spectral properties of A

Variants of “basic” methods by acting on polynomial pm−1

This choice of Km is good, but no longer sufficiently good

Choosing Km

Nice Properties:

Choosing Km

Nice Properties:

Getting greedy

Km = Km−k(A, v) ∪ Sk New question: Sk?

? Faster approximation when spectral a-priori knowledge is available(even for A Hermitian)

? Main motivating problem:

Large m may be required for accurate approximation xm

Computational/Memory costs increase nonlinearly with m

(A non-normal)

Getting greedy

(A non-normal)

Getting greedy

(A non-normal)

Intermezzo. Restarting procedure

(A non-normal)

Restarting procedure: x0 initial approx, r0 = b − Ax0

Km(A, r0) → x(1)m , r(1)

Km(A, r(1)m ) → x(2)

m , r(2)m

... →...

Warning: Larger m not always implies faster convergence(Embree, Ernst, ...)

Restarted Methods

Convergence strongly depends on choice of m ...

0 100 200 300 400 500 600 700 80010−10

10−8

10−6

10−4

10−2

GMRES(15)

Number of Matrix−vector multiplies

lative r

GMRES(20)

Restarted Methods

Convergence strongly depends on choice of m ... true?

0 100 200 300 400 500 600 700 80010−10

10−8

10−6

10−4

10−2

GMRES(15)

FOM(15)

Global subspace generated

lative r

GMRES(20)

Restarted Methods

Switch to FOM residual vector only at the very first restart

0 50 100 150 200 250 300 350 400 45010−10

10−8

10−6

10−4

10−2

Number of Matrix−vector multiplies

lative

GMRES(15)

GMRES(15)+FOM

Pictures from Simoncini, SIMAX 2000. Intermezzo ends.

Augmented projection spaces

Km = Km−k(A, v) ∪ Sk

Sk from spectral information of A

Km−k(A, v) ∪ Sk = span{v, Av, . . . , Am−k−1v, y1, y2, . . . , yk}

y1, . . . , yk approximate eigenvectors of A associated to cluster

(Baglama, Calvetti, Erhel, Morgan, Nabben, Reichel, Saad, Sorensen, Vuik ...)

Sk important space from previous restartsRanking based on “error” or “effectiveness” of the past spaces

(De Sturler, Baker, Jessup, Manteuffel, ...)

Augmented projection spaces

Km = Km−k(A, v) ∪ Sk

Sk from spectral information of A

Km−k(A, v) ∪ Sk = span{v, Av, . . . , Am−k−1v, y1, y2, . . . , yk}

y1, . . . , yk approximate eigenvectors of A associated to cluster

(Baglama, Calvetti, Erhel, Morgan, Nabben, Reichel, Saad, Sorensen, Vuik ...)

Sk important space from previous restartsRanking based on “error” or “effectiveness” of the past spaces

(De Sturler, Baker, Jessup, Manteuffel, ...)

Augmented method. Spectral Information available.Structural Dynamics problem: (AB−1 + σI)x = b

A,B complex sym. n = 86, 000. Solve for σ in a wide interval

Note: at each iteration solve system with matrix B

B = B(κ) κ related to artificial stiff springs at ground boundaries

B numerically singular as κ → 0

Augmented method. Spectral Information available.Structural Dynamics problem: (AB−1 + σI)x = b

A,B complex sym. n = 86, 000. Solve for σ in a wide interval

Note: at each iteration solve system with matrix B

B = B(κ) κ related to artificial spring stiffness at ground boundaries

B numerically singular as κ → 0

Fill-in p=5 Fill-in p=15

E. Time # its E. Time # its

[s] (outer/ avg. inner) [s] (outer/ avg. inner)

κ = 0 ACG 14066 296/38 12790 281/33

κ = 100 14072 117/120 13739 121/102

κ = 1000 8694 88/96 8724 89/83

κ = 1000 unrealistic Perotti & Simoncini, ’02

ACG: Augmented CG, Saad & Yeung & Ehrel & Guyomarc’h, 2000

Example of Augmented method. Error SpaceTricky way to enhance approximation space

A from L(u) = −1000∆u + 2e4(x2+y2)ux − 2e4(x2+y2)uy n = 40 000

0 200 400 600 800 1000 120010−10

10−9

10−8

10−7

10−6

10−5

10−4

10−3

10−2

10−1

Iteration index

tive r

ual norm

GMRES(50)

GMRES(150)

Example of Augmented method. Error SpaceTricky way to enhance approximation space (E. De Sturler)

0 200 400 600 800 1000 120010−10

10−9

10−8

10−7

10−6

10−5

10−4

10−3

10−2

10−1

Iteration index

tive r

ual norm

GMRES(50)

GMRES(150)

GCROT(7,23,23,4,1,0)

Code: courtesy of Oliver Ernst see also Baker, Jessup, Manteuffel ’05

Changing Km

Modify Km instead of augmenting it!

Increase flexibility

Cope with “involved” operators

Changing Km. Flexible methods

Original problem

AP−1x = b P preconditioner

Km(AP−1, b) = span{b, AP−1b, . . . , (AP−1)m−1b}

at each iteration i: zi = P−1vi

Flexible variant: Saad, ’93

Iteration i: zi = P−1vi ⇒ zi = P−1i vi

xm ∈ span{b, z1, z2, . . . , zm−1} 6= Km(AP−1, b)

Changing Km. Flexible methods

Original problem

AP−1x = b P preconditioner

Km(AP−1, b) = span{b, AP−1b, . . . , (AP−1)m−1b}

at each iteration i: zi = P−1vi

Flexible variant: Saad, ’93

Iteration i: zi = P−1vi ⇒ zi = P−1i vi

xm ∈ span{b, z1, z2, . . . , zm−1} 6= Km(AP−1, b)

Flexible methods. An exampleFlexible method may be used as a Truncated/Augmented method.

z = P−1v ⇔ z ≈ A−1v span{b, z1, z2, . . . , zm−1}

A from L(u) = −∆u + 1000xux n = 900

0 200 400 600 800 1000 120010−10

10−9

10−8

10−7

10−6

10−5

10−4

10−3

10−2

10−1

F/G(20) FT/G(16−2) FT/G(12−4)

GM(20)

number of matrix−vector multiplies

tive r

Simoncini & Szyld, SINUM ’03

Flexible methods. An exampleFlexible method may be used as a Truncated/Augmented method.

z = P−1v ⇔ z ≈ A−1v span{b, z1, z2, . . . , zm−1}

A from L(u) = −∆u + 1000xux n = 900

0 200 400 600 800 1000 120010−10

10−9

10−8

10−7

10−6

10−5

10−4

10−3

10−2

10−1

F/G(20) FT/G(16−2) FT/G(12−4)

GM(20)

lative r

Flexible methods. A second exampleFlexible method may be used as a Truncated/Augmented method.z = P−1v ⇔ z ≈ A−1v span{b, z1, z2, . . . , zm−1}

0 500 1000 1500 2000 250010−7

10−6

10−5

10−4

10−3

10−2

10−1

lative r

GM(30)

F/G(30)FT/G(10−10)

FT/G(20−5)

Changing Km. Inexact methods.

Original problem

Ax = b A Possibly not available exactly

Km(A, b) = span{b, Ab, . . . , Am−1b}

Inexact (relaxed) variant:

Iteration i: zi = Avi ⇒ zi = Avi + fi

xm ∈ span{b, z1, z2, . . . , zm−1} 6= Km(A, b)

♣ “Worse” than for Flexible methods:

rm = b − Axm not available!

Available: rm computable residual

Original problem

Km(A, b) = span{b, Ab, . . . , Am−1b}

Original problem

Km(A, b) = span{b, Ab, . . . , Am−1b}

Inexact Methods. An example

if ‖fi‖ = O

‖ri−1‖

)∀i ⇒ ‖b − Axm‖ ≤ ε for m large enough

BT S−1B︸︷︷︸A

At each it. i solve:

Swi = Bvi

‖Swi − Bvi‖ ≤ εinner(‖fi‖)

δm = ‖rm − erm‖

0 20 40 60 80 100 12010−8

10−7

10−6

10−5

10−4

10−3

10−2

10−1

number of iterations

magnitude

εinner

Simoncini & Szyld, SISC ’03

Inexact Methods. An example

if ‖fi‖ = O

‖ri−1‖

)∀i ⇒ ‖b − Axm‖ ≤ ε for m large enough

BT S−1B︸︷︷︸A

At each it. i solve:

Swi = Bvi

‖Swi − Bvi‖ ≤ εinner(‖fi‖)

δm = ‖rm − erm‖

0 20 40 60 80 100 12010−8

10−7

10−6

10−5

10−4

10−3

10−2

10−1

number of iterations

magnitude

εinner

Simoncini & Szyld, SISC ’03

A different application. The Lyapunov equation

AX + XAT + Q = 0

with A dissipative, Q = BBT of low rank X ≈ X low rank

Standard Krylov approach: X ∈ Km(A, B) and

R = AX + XAT + Q ⊥ Lm = Km(A, B)

X = VmYmV Tm for some Ym Range(Vm) = Km(A, B) Saad, ’90

New “Enhanced” approach: X ∈ Kk(A, B) ∪ Kk(A−1, B) and

R = AX + XAT + Q ⊥ Lm = Kk(A, B) ∪ Kk(A−1, B)

? Very competitive w.r.to Cyclic ADI methodSimoncini, tr. 2006

AX + XAT + Q = 0

An example. Time-invariant linear system

x′ = xxx + xyy + xzz − 10xxx − 1000yxy − 10zxz + b(x, y)u(t)

A matrix 183 × 183

0 5 10 15 20 2510−11

10−10

10−9

10−8

10−7

10−6

10−5

10−4

10−3

10−2

CPU Time

Enhanced Krylov

Standard Krylov

approximation space dim.: 146 (Standard Krylov) 112 (Enhanced Krylov)

x′ = xxx + xyy + xzz − 10xxx − 1000yxy − 10zxz + b(x, y)u(t)

A matrix 183 × 183

0 5 10 15 20 2510−11

10−10

10−9

10−8

10−7

10−6

10−5

10−4

10−3

10−2

CPU Time

Enhanced Krylov

Standard Krylov

x′ = xxx + xyy + xzz − 10xxx − 1000yxy − 10xz + b(x, y)u(t)

A matrix 183 × 183

0 5 10 15 20 2510−11

10−10

10−9

10−8

10−7

10−6

10−5

10−4

10−3

10−2

CPU Time

Enhanced Krylov

Enhanced Krylov(reord)

Standard Krylov

Conclusions and Pointers

Projection is a versatile tool

Lots of room for improvements on hard problems

Survey

“ Recent computational developments in Krylov Subspace Methodsfor linear systems”

with Daniel Szyld, Temple University

To appear in J. Numerical Linear Algebra w/Appl. (352 refs)

This and other papers at

Conclusions and Pointers

Projection is a versatile tool

Lots of room for improvements on hard problems

Survey

“ Recent computational developments in Krylov Subspace Methodsfor linear systems”

with Daniel Szyld, Temple University

To appear in J. Numerical Linear Algebra w/Appl. (352 refs)

This and other papers at

Structural Dynamics Problem

Mq + Cq + Kq = f(t) n d.o.f.

Linearization under small deformations

Direct frequency analysis for damping modeling(as opposed to modal or time analysis)? ideal for mechanical properties depending on frequency- influence of deformation velocity on materials- presence of hysteretic damping, ...

In the frequency domain: (a = −(2πf)2q(f)):

−(2πf)2K∗ +

i(2πf)CV + M

)a = b

C = CV + 12πf

CH , ⇒ K∗ := K + iCH

Concrete Slab ElementsGround Elements

SIDE VIEW

PLAN VIEW

Concrete Slab

Elements

Ground

Elements

Model Problem: 3D Soil-Structure interaction

? viscous damping at the boundary ⇒ K∗ singular

⇒ FE discretization, K∗ almost singular

Algebraic Linearization:(σ2K∗ + σiCV − M

)x = b, x = x(σ)

equivalent to

[iCV −M

−M 0

︸︷︷︸A

[K∗ 0

︸︷︷︸B

with y = σx

Krylov Subspace Methods for Linear Systems and Matrix ...simoncin/siamala.pdf · Convergence...

Documents

Comparison of some Preconditioned Krylov Methods for

2003 Iterative Krylov Methods for Large Linear Systems

Rational Krylov Sequence Methods for Eigenvalue Computation · Rational Krylov Sequence Methods for Eigenvalue Computation Axe1 Ruhe ... of a rational Krylov subspace is found in

Krylov Methods for the Incompressible Navier-Stokes Equations...Krylov Methods for the Incompressible Navier-Stokes Equations ... our solution strategies should also be compatible

Krylov subspace methods for projected Lyapunov equations ...stykel/Publications/pr_10_735… · for Krylov subspace methods have been extended to projected Lyapunov equations [37,

Jens Zemke - Abstract Perturbed Krylov Methods

Krylov Subspace Methods on Supercomputers

Flexible Krylov Methods for p Regularization€¦ · Introduction Methods based on FGK Sparsity under transform Numerical experiments Conclusions Flexible Krylov Methods for ‘ p

PARALLEL LAGRANGE-NEWTON-KRYLOV-SCHUR METHODS FOR … · PARALLEL LAGRANGE-NEWTON-KRYLOV-SCHUR METHODS FOR PDE-CONSTRAINED OPTIMIZATION PART I: THE KRYLOV-SCHUR SOLVER GEORGE BIROS

Krylov space methods

Globalization techniques for Newton–Krylov methods and ...walker/Papers/globalization_techniques_repor… · Newton–Krylov methods and applications to the fully-coupled solution

Tensor-Krylov Methods for Solving Large-Scale …prod.sandia.gov/techlib/access-control.cgi/2004/041837.pdfTensor-Krylov Methods for Solving Large-Scale Systems of Nonlinear Equations

Krylov Subspace Methods for Nonlinear Model Reduction · Computational Methods in Systems and Control Theory ... Max-Planck-Institut Magdeburg P. Benner, Krylov Subspace Methods for

CME 345: MODEL REDUCTION - Stanford UniversityCME 345: MODEL REDUCTION - Moment Matching Krylov-based Moment Matching Methods Moment Matching by Krylov Methods Partial realization

Inexact Newton Methods, Newton-Krylov Methods, and ... · Inexact Newton Methods, Newton-Krylov Methods, and Extensions for Large-Scale Underdetermined Systems Homer Walker DOE O

Krylov-Subspace-Based Order Reduction Methods Applied to

Krylov subspace methods for eigenvalue problems · Krylov subspace methods for eigenvalue problems David S. Watkins watkins@math.wsu.edu Department of Mathematics Washington State

COMPUTING APPROXIMATE (BLOCK) RATIONAL ......Krylov subspace, as we have already shown for extended Krylov subspaces in [17]. Block Krylov subspace methods are an extension of Krylov

The Idea Behind Krylov Methods

Krylov Subspace Methods in Model Order Reduction