Solving Systems of Linear Equations

● Motivation● The idea behind Googles pagerank● An example from economics

● Gaussian elimination● LU Decomposition● Iterative Methods – The Jacobi method● Summary

Motivation

● Systems of linear equations of the form

or in components

● Are found in many contexts. For instance we need them to find roots of systems of non-linear equations using the Newton method.

A x=b

a11 x1+a12 x2+...+a1n xn=b1

a21 x1+a22 x2+...+a2n xn=b2

...

am1 x1+am2 x2+ ...+amn xn=bm

A=(a11 a12 ... a1n

a21 a22 ... a2n

... ... ... ...am1 am2 ... amn

) , b=(b1

b2

...bm

)

Short Reminder (1)

● Matrix-vector multiplication

● Matrix-matrix multiplication

A x=[a11 a12 a13

a21 a22 a23

a31 a32 a33] [

x1

x2

x3]=[

a11 x1+a12 x2+a13 x3

a21 x1+a22 x2+a23 x3

a31 x1+a32 x2+a33 x3]

[a11 a12 a13

a21 a22 a23

a31 a32 a33][b11 b12 b13

b21 b22 b23

b31 b32 b33]=[

... ... ...

... ∑kaik bkj ...

... ... ...]

Short Reminder (2)● Graphical illustration of 2d systems

● Every equation represents a line

● The following situations are possible– Lines cross at exactly one point -> unique solution (non-

singular)– Lines are parallel (singular) and don't cross -> no solution– Lines overlap completely -> infinitely many solutions

● In higher d the same situations are possible

non-singular singular

Short Reminder (3)

● Focus on square matrices● Some criteria that an n*n matrix is non-singular:

● The inverse of A (denoted A-1) exists● det (A) !=0● rank (A) = n● For any vector z!=0 it holds Az!=0

Example: How Google Search Works (sort of)

● In a particular group of people, who is the most popular?

● Say there are three people: Jane, Charlie, Fred● We ask them to list their friends

● J lists C, C lists J,F, F lists J● Could write this as a matrix

J C F

J 0 1 1

C 1 0 0

F 0 1 0

#total 1 2 1

Example (2)

● Some people list more than others as friends. To compensate we normalise each column by the sum

● Now want to find a vector p=(j,c,f) which assigns a popularity to each person

● Idea: a person's popularity is the weighted sum of the popularities of the people who mention it

→ p=Lp

(0 1/2 11 0 00 1/2 0)=L

J C F

J

C

F

“linking matrix”

Example (3)

1

1/2

1/2

1/31/3

1/3

A node (person) is the more popular the● more people list it as friends● less friends these people list● more popular these friends are (i.e. Having one very popular good friend can make you more popular than having 10 not so popular friends)

Example (4)● This defines a system of linear equations!

● E.g.: j=1/2 c+ f, c=j, f=1/2 c or● In this case this is easy to solve● System is under-determined: (e.g.) set j=1 as a

scale, -> c=1, f=1/2, i.e.● Charlie and Jane are most popular, Fred is less so.

(0 1/2 11 0 00 1/2 0)=L

J C F

J

C

F

Lp=p

Remark

● We could use this to assign a popularity to web-pages● The friend lists from before would then just refer to

the lists of links in web-pages, i.e. How many web-pages link to a given page (normalized)?

● The popularity p could be interpreted as page rank

(or we might wish to add contextual information (c) for search queries and not base them purely on popularity … e.g. p=Lp + c)

● We might wonder if the system Lp=p always has positive solutions p● -> This is ensured by the Frobenius-Perron theorem

Example: Setting Prices in an Economy

● This is a problem that goes back to Wassily Leontief (who got the Nobel prize for economics in 1973)

● Suppose in a far away land of Eigenbazistan, in a small country town called Matrixville, there lived a Farmer, a Tailor, a Carpenter, a Coal Miner and Slacker Bob. The Farmer produced food; the Tailor, clothes; the Carpenter, housing; the Coal Miner supplied energy; and Slacker Bob made High Quality 100% Proof Moonshine, half of which he drank himself.

● Let's assume that:

● No outside supply or demand● Everything produced is consumed.

● Need to specify what fraction of each good is consumed by each person

● Problem: What are the incomes of each person?

Consumption

Food Clothes Housing Energy High Quality 100 Proof Moonshine

Farmer 0.25 0.15 0.25 0.18 0.20Tailor 0.15 0.28 0.18 0.17 0.05Carpenter 0.22 0.19 0.22 0.22 0.10Coal Miner 0.20 0.15 0.20 0.28 0.15Slacker Bob 0.18 0.23 0.15 0.15 0.50

● I.e. The farmer eats 25% of all food, 15% of all clothes, 25% of all housing, 18% of all energy

● Let's say price levels are pF,p

T,p

CA,p

CO,p

S

● Since this is a closed economy, we require:

pF=0.25 pF+0.15 pT+0.25 pCA+0.18 pCO+0.20pS

Again ... a System of Linear Equations

● Again this is a system of the type Ap=p (i.e. Under-determined, can set one price arbitrarily)

● Not so easy to solve by hand ... computer methods would be useful!

pF=0.25 pF+0.15 pT+0.25 pCA+0.18 pCO+0.20pS

pT=0.15 pF+0.28 pT+0.18 pCA+0.17 pCO+0.05pS

pCA=0.22 pF+0.19 pT+0.22 pCA+0.22 pCO+0.10pS

pCO=0.20 pF+0.15 pT+0.20 pCA+0.28 pCO+0.15pS

pS=0.18 pF+0.23 pT +0.15 pCA+0.15 pCO+0.50pS

Gaussian Elimination

● One idea to “mechanise” solving such systems is Gaussian elimination

● Consider the system:

● Observations:● Can multiply equations by constants, swap

equations, or add or subtract equations without changing solution

● Apply such operations to transform system into convenient form?

● What is a convenient form? -> triangular

2x+ y−z=8−3x− y+2z=−11−2x+ y+2z=−3

Gaussian Elimination (2)

● Triangular form:

● If our system had the above form we could just solve it by successive substitution, i.e.

2x+ y−z=81/3 y+1/3 z=2 /3

−z=1 (3)

(2)(1)

(3) z=−1

(2)(substitute into) 1/3 y−1 /3=2/3→ y=3

(3)(substitute into) 2 x+3−(−1)=8→ x=2

How to Transform into Triangular Form?

● Start with:

● Pivot (1). Multiply (2) by 2/3 and add to (1)

● Multiply (3) by 1 and add to (1)

2x+ y−z=8−3x− y+2z=−11−2x+ y+2z=−3

(1)(2)(3)

2x+ y−z=8y3+

z3=2/3

−2x+ y+2z=−3

(1)

(2')

(3)

2x+ y−z=8y3+

z3=1

2y+z=5

(1)

(2')

(3')

How to Transform into Triangular Form? (2)

● Now multiply (2') by -6 and add to (3')

● Matrix is now in triangular form. Can diagonalise it as follows:

2x+ y−z=8y3+

z3=2/3

2y+z=5

(1)

(2')

(3')

2x+ y−z=8y3+

z3=2/3

−z=1

(1)

(2')

(3'')

Diagonalising ...

● Multiply (2') by 3 and add to (3''), (3'') by -1 add to (1):

● And finally: Multiply (2'') by -1 and add to (1). Finally, scale coefficients:

2 x+ y =7

y =3

−z=1

(1)

(2'')

(3'')

2 x =4

y =3

z=1

(1')

(2'')

(3'')

Gaussian Elimination

● Usually, this is done using only coefficient schemes the “augmented” matrix

● Can be used to diagonalise matrices, just append identity matrix and perform all operations on it as well

[ 2 1 −1 1 0 0−3 −1 2 0 1 0−2 1 2 0 0 1 ]

Coefficient matrix Identity matrix

Example ...

[ 2 1 −1 1 0 0−3 −1 2 0 1 0−2 1 2 0 0 1 ][2 1 −1 1 0 00 1/2 1 /2 3/2 1 00 2 1 1 0 1] 3/2*(1)+(2)

(1)+(2)

[2 1 −1 1 0 00 1/2 1 /2 3/2 1 00 0 −1 −5 −4 1] -4*(2)+(3)

Example (2)

[2 1 −1 1 0 00 1/2 1 /2 3/2 1 00 0 −1 −5 −4 1]

[2 1 0 6 4 −10 1 0 −2 −2 10 0 −1 −5 −4 1 ] 2*(2)+(3)

(1)-(3)

[2 0 0 8 6 −20 1 0 −2 −2 10 0 −1 −5 −4 1 ]

(1)-(2)

[1 0 0 4 3 −10 1 0 −2 −2 10 0 1 5 4 −1]

1/2*(1)1*(2)-1*(3)

This is A-1

Example (3)

● OK, we found A-1, how to check if this is correct?

● Suppose we now have A-1 … how is this useful to find x?

Some Remarks

● Pivot may be zero or (“inconvenient” for numerical stability) -> reorder rows

● Possibly divide by very small numbers (if pivot is small), better reorder to use largest possible pivot

● Computational complexity is of order O(n3) (roughly: n-1 rows need to be dealt with, this process involves order n operations and needs to be repeated less than n times)

Pseudocode

for k = 1 ... min(m,n): Find the k-th pivot: i_max := argmax (i = k ... m, abs(A[i, k])) if A[i_max, k] = 0 error "Matrix is singular!" swap rows(k, i_max) Do for all rows below pivot: for i = k + 1 ... m: Do for all remaining elements in current row: for j = k + 1 ... n: A[i, j] := A[i, j] - A[k, j] * (A[i, k] / A[k, k]) Fill lower triangular matrix with zeros: A[i, k] := 0

Turns m*n matrix into “echelon” formation which can then be solved by back-substitution

LU Decomposition

● A slightly different strategy is LU factorization● Write A as a product of two triangular matrices,

one upper (U) and one lower triangular A=LU

● How does this help to solve Ax=b?

● Can first solve Ly=b and then Ux=y

[a11 a12 a13

a21 a22 a23

a31 a32 a33]=[

l11 0 0l21 l22 0l31 l32 l 33

] [u11 u12 u13

0 u22 u23

0 0 u33]

A x=LU x=b

LU Decomposition (2)

● Ly=b is “easy” to solve because

we can solve it by back-substitution● Similarly, Ux=y is “easy”:

[l11 0 0l21 l22 0l31 l32 l33

][y1

y2

y3]=[

l11 y1

l21 y1+ l22 y2

l 31 y1+l32 y2+l33 y3]=[

b1

b2

b3]

[u11 u12 u13

0 u22 u23

0 0 u33] [

x1

x2

x3]=[

u11 x1+u12 x2+u13 x3

u22 x2+u23 x3

u33 x3]=[

y1

y2

y3]

LU Decomposition (3)● Essentially variant of Gaussian elimination● Like in Gaussian elimination we might have to

exchange● Rows (“partial pivoting”) PA=LU ● Rows and columns (“full pivoting”) PAQ=LU● Where P is a permutation matrix that keeps accounts for

row permutations and Q for column permutations

● Any square matrix allows an LUP factorization● For positive definite symmetric (or Hermitian)

matrices there is a similar decomposition, the Cholesky decomposition A=L L* (often used in statistics)

Doolittle Algorithm (1)● One algorithm to generate LUP● Similarly to GE we aim to remove entries of A

under the diagonal which is achieved by multiplying A with an appropriate matrix L' from the left, e.g.:

● This is equivalent to multiplying A from the left by

[a11 a12 a13

a21 a22 a23

a31 a32 a33] Multiply by -a21/a11 and add to first row

Multiply by -a31/a11 and add to first row

[1 0 0

−a21/ a11 1 0−a31/ a11 0 1 ][

a11 a12 a13

a21 a22 a23

a31 a32 a33]=[

a11 a12 a13

0 −a21 / a11 a12+a22 −a21/ a11 a13+a23

0 −a31 / a11 a12+a32 −a31/ a11 a13+a33]

Doolittle Algorithm (2)

● Start with matrix A(0)=A

● At stage n-1 left multiply A(n-1) with a matrix Ln that has

diagonal entries 1 and only entries in the nth column under the diagonal. These entries are

● In N-1 steps obtain an upper triangular matrix

● We have:

● Since products and inverses of lower triangular matrices are lower triangular we have (*)

and A=LU

li ,n(n )

=−ai , n

(n−1)

ann(n−1)

U=A (N−1)

A=L1−1 L2

−1... LN−1−1 A (N−1)

L=L1−1 L2

−1... LN−1−1

Doolittle (3)● Still need to determine L from (*)

Ln=[1 0 0 0 0 ... 00 1 0 0 0 ... 00 ... 0 1 0 ... 00 ... 0 li , n 0 ... 00 ... 0 lN ,n 0 ... 1

]Ln

−1=[

1 0 0 0 0 ... 00 1 0 0 0 ... 00 ... 0 1 0 ... 00 ... 0 −li , n 0 ... 00 ... 0 −lN ,n 0 ... 1

]

Doolittle (4)

● And hence:

L=[1 0 0 0 0 ... 0

−l21 1 0 0 0 ... 0−l31 −l 32 ... 1 0 ... 0−l41 −l 42 −l43 −li n 0 ... 0

... ... ... ... ... ... ...−lN1 −lN2 −lN3 ... ... −lN N−1 1

]

Beyond Doolittle ...

● LU decomposition has some slight advantages over Gaussian elimination● i.e. once we have L and U we can solve systems for

every r.h.s. B● Small modifications can also be used to calculate

the matrix inverse

● Generalizations are e.g. the Crout algorithm or LUP decomposition

(the wiki pages are very good on this topic and give many more “advanced” references:

http://en.wikipedia.org/wiki/LU_decomposition)

Iteration Methods

● Time complexity of LU decomposition or Gaussian elimination is O(n3), i.e. dealing with millions of equations (which arise when solving certain PDEs) can take a long time ...

● There are faster methods that only come up with approximate solutions, these are iteration methods

● The trade-off of these methods is that they might not converge for general types of coefficient matrices A

● For diagonally dominant methods a widely used method is the Jacobi method

Jacobi Method

● Want to solve Ax=b (*)● Write A=D+R

● Where D are the diagonal elements of A and R is the rest

● Diagonally dominant means that “D is large compared to R”, e.g.

● Write (*) as:

● Iterate this:

∣aii∣>∑ j∣aij∣

A x=(D+R) x= b x=D−1( b−R x )

xn+1=D−1( b−R xn) Iteration #

Example Jacobi

● Consider the system:

● Jacobi: say:

2x+ y=115x+7y=13

A=[2 15 7 ] , b=[11

13]xn+1=D−1

( b−R xn)

D=[2 00 7 ] , R=[0 1

5 0]→D−1=[1/2 0

0 1 /7 ]

x0=[1,1]T

c=D−1 b=[11/2,13 /7]T

T=D−1 R=[ 0 1/25/7 0 ]

xn+1= c−T xn

Example Jacobi (2)

● Hence we have to iterate:

● With x0=[1,1]

● Iterating 25 times one obtains: x=[7.111, -3.222]● Exact -> x=64/9=7.1111...., y=-29/9=-3.22222....

xn+1= c−T xn

xn+1=[11 /213/7]−[ 0 −1/2

−5/7 0 ] xn

x1=[11 /213/7 ]−[ 0 −1/2

−5 /7 0 ] [11]=[ 58/7 ]

The Gauss-Seidel Method

● Jacobi method only applicable to diagonally dominant matrices

● Gauss-Seidel can be used for general matrices, but convergence only guaranteed if● Diagonally dominant or● Symmetric and positive definite

● Idea: A=L+U

Lower triangular Strictly upper triangular

xn+1=L−1(b−U xn)

Software Packages

● LINPACK ● Package for solving wide variety of linear systems

and special systems (e.g. symmetric, banded, etc.)● Has become standard benchmark for comparing

performance of computers

● LAPACK● More recent replacement of LINPACK – higher

performance on modern computer architectures including some parallel computers

● Available from Netlib -- http://www.netlib.org/

Summary

● Exact methods to solve linear systems of equations● Gaussian elimination ● LU decomposition – Doolittle algorithm● With full/partial pivoting -> in practice stable● O(n3)

● Iterative methods – e.g. Jacobi method● Faster – suited to millions of equations that might

arise when solving certain PDEs● Limited convergence● More sophisticated: weighted Jacobi, Gauss-Seidel

successive over-relaxation, ...

References

● The English wiki pages are quite comprehensive on this topic (and I used them quite a bit for preparing the lecture)

● More comprehensive:● David M. Young, Jr. Iterative Solution of Large

Linear Systems, Academic Press, 1971. (reprinted by Dover, 2003)

● Richard S. Varga 2002 Matrix Iterative Analysis, Second ed. (of 1962 Prentice Hall edition), Springer-Verlag.