Iterative Solvers

Approaches range from simple (Jacobi) to complex (Newton-Krylov methods, Multigrid methods). • We will only look at the most simple iterative algorithms...

When to use: • When it is faster to solve iteratively than directly...

• Sparse systems of equations that are diagonally dominant (diagonal coefficient is larger in magnitude than the off-diagonal coefficients).

‣ If the system is not diagonally dominant, sometimes you can re-arrange it to help improve the diagonal dominance - PIVOTING.

‣ Lack of diagonal dominance may kill an iterative method.⇤

⇧0 1 30 2 14 1 2

⌅

⌃

�

⌥x1

x2

x3

⇥

� =

�

⌥b1

b2

b3

⇥

�

Concept: guess solution, and then iterate to improve guess until you get “close enough” to the true solution.

⇤

⇧4 1 20 2 10 1 3

⌅

⌃

�

⌥x1

x2

x3

⇥

� =

�

⌥b3

b2

b1

⇥

�

1 LinearSystems-iterative.key - September 3, 2014

Jacobi Iterative Approach

Example: 3 linear equations

If we have a guess for xi, then we can update that guess by solving each equation for the diagonal:

Now we have a new guess for xi. Repeat the above step to improve this guess again.

⇤

⇧a11 a12 a13

a21 a22 a23

a31 a32 a33

⌅

⌃

�

⌥x1

x2

x3

⇥

� =

�

⌥b1

b2

b3

⇥

�

xk+11 = 1

a11

�b1 � a12x

k2 � a13x

k3

⇥

xk+12 = 1

a22

�b2 � a21x

k1 � a23x

k3

⇥

xk+13 = 1

a33

�b3 � a31x

k1 � a32x

k2

⇥or,

equivalently:

This approach is simplifies implementation in a code.

Simplest (and potentially least robust) iterative solution method for linear systems.

Hoffman §1.7.1

x

k+1i = x

ki +

1

aii

0

@bi �

nX

j=1

aijxkj

1

A

x

k+11 = x

k1 + 1

a11

�b1 � a11x

k1 � a12x

k2 � a13x

k3

�

x

k+12 = x

k2 + 1

a22

�b2 � a21x

k1 � a22x

k2 � a23x

k3

�

x

k+13 = x

k3 + 1

a33

�b3 � a31x

k1 � a32x

k2 � a33x

k3

�

2 LinearSystems-iterative.key - September 3, 2014

Example: Jacobi Iterationxk+1

i = xki +

1ai,i

�

⇤bi �n⇧

j=1

ai,jxkj

⇥

⌅x0 =

�

⇤111

⇥

⌅⇤

⇧5 1 12 3 03 0 4

⌅

⌃

�

⌥x1

x2

x3

⇥

� =

�

⌥101112

⇥

�

First iteration:

x =

�

⇤0.9302325581395353.0465116279069772.302325581395349

⇥

⌅

What kinds of error are present?

• Iteration error? • Roundoff error? • Approximation error?

x11 = x0

1 +15

�10� 5x0

1 � 1x02 � 1x0

3

⇥=

85

x12 = x0

2 +13

�11� 2x0

1 � 3x02

⇥= 3

x13 = x0

3 +14(12� 3x0

1 � 4x03) =

94

k indicates the iteration number.

k x1 x2 x3

0 1 1 11 1.6 3.0 2.252 0.95 2.60 1.83 1.12 3.03 2.294 0.936 2.92 2.1610 0.93036 3.04363 2.299120 0.93023 3.04650 2.3023225 0.9302327 3.046511615 2.3023255730 0.93023256 3.0465116276 2.302325581135 0.9302325581 3.0465116277 2.302325581140 0.9302325581 3.0465116279 2.3023255814

1 0.9302325581 3.0465116279 2.3023255814

3 LinearSystems-iterative.key - September 3, 2014

Convergence - When to Stop Iterating?

Error Measures

Absolute error:

or

Relative error:

Hoffman §1.7.2

⇥a⇥1 =�n

i=1 |ai| L-1 norm⇥a⇥2 =

⇥�ni=1 a2

i L-2 norm⇥a⇥� = max(|ai|) L-infinity norm

In MATLAB, norm(a,1), norm(a,2), norm(a,inf),

Vector “norms” measure the “magnitude” of a vector.

or

Iterate until: , where is the user-specified tolerance.

� � ⇥ �

ε ⇒ absolute error (and error tolerance)

ε ⇒ relative error (and error tolerance)

When writing an iterative solver, be sure to clearly specify whether you expect a relative or absolute tolerance!

Convergence

or

� =����

bi �⇥n

i=1 aijxj

bi

���� , i = 1 . . . n

� =

�����xk

i � xk�1i

xki

�����

� = kb�Axk =

������bi �

nX

j=1

aijxj

������, i = 1 . . . n

✏ = kx� x

exact

k

✏ =��x

k � x

k�1��

4 LinearSystems-iterative.key - September 3, 2014

Jacobi Solution AlgorithmAx = b

n�

j=1

ai,jxj = bi

function x = jacobi( A, b, xguess, tol )! ![nr,nc]=size(A);!!x = xguess;!r = abs(b-A*xguess); % initial residual!ad = diag(A); % diagonal of A! !niter=0;! !while max(abs(r))>tol! x = xguess + 1./ad .* (b-A*xguess);! xguess = x;! r = b - A*x;! niter = niter+1;!end

1. Guess xi.2. Update guess for each xi, starting with i=1.

3. Repeat step 2 until convergence is obtained.

xk+1i = xk

i +1

ai,i

�

⇤bi �n⇧

j=1

ai,jxkj

⇥

⌅

xk+1 = xk + diag(A)�1�b�Axk

⇥or

Algorithm

function x = jacobi( A, b, xguess, tol )!![nr,nc] = size(A);! !x = xguess;!r = b-A*xguess; % initial residual!!niter=0;!while norm(r,inf)>tol! for i=1:nr! x(i) = xguess(i) + 1./A(i,i) * ...! (b(i)-A(i,:)*xguess);! end! r = b - A*x; % update the residual! xguess = x;! niter = niter+1;!end

5 LinearSystems-iterative.key - September 3, 2014

Gauss-Seidel Iterative SolverObservation: Jacobi iteration does not take advantage of the most recent solution information! • x1 is updated first, but the updated value is not

used in solution for x2...

Idea: Always use most recent information. • Called the “Gauss-Seidel” method.

Jacobi update:

xk+1i = xk

i +1

ai,i

�

⇤bi �n⇧

j=1

ai,jxkj

⇥

⌅

Gauss-Seidel update:

1. Guess xi.2. Update guess for each xi, starting with i=1. Use

most recent information when calculating each xi.

3. Repeat step 2 until convergence is obtained.

xi = xi +1

ai,i

�

⇤bi �n⇧

j=1

ai,jxj

⇥

⌅

Algorithm

Hoffman §1.7.3

6 LinearSystems-iterative.key - September 3, 2014

Example: Gauss-Seidelx0 =

�

⇤111

⇥

⌅⇤

⇧5 1 12 3 03 0 4

⌅

⌃

�

⌥x1

x2

x3

⇥

� =

�

⌥101112

⇥

�

First iteration:

x =

�

⇤0.9302325581395353.0465116279069772.302325581395349

⇥

⌅

k x1 x2 x3

0 1 1 11 1.6 2.6 1.82 1.12 2.92 2.163 0.984 3.011 2.2624 0.945 3.036 2.29110 0.93024044 3.04650637 2.3023196720 0.93023255817 3.04651162789 2.3023255813825 0.93023255814 3.04651162791 2.30232558140� 0.9302325581 3.0465116279 2.3023255814

xi = xi +1

ai,i

�

⇤bi �n⇧

j=1

ai,jxj

⇥

⌅

For this system, Gauss-Seidel requires about 1/2 the iterations of Jacobi to achieve the same tolerance.

x1 = x1 +15

(10� 5x1 � 1x2 � 1x3) =85

x12 = x2 +

13

(11� 2x1 � 3x2) =3915

x13 = x3 +

14(12� 3x1 � 4x3) =

95

7 LinearSystems-iterative.key - September 3, 2014