1

Chapters: 3, 4 and 6

Solving Linear Systems

A system of n simultaneous, non-homogenous, algebraic, linear equations is given in

the form:

A : is an nn coefficient matrix. b : is an 1n right-hand side vector.

x : is an 1n vector of unknowns.

nnnn

n

n

AAA

AAA

AAA

21

22221

11211

nx

x

x

2

1

=

nb

b

b

2

1

nnA

1nx

1nb

First: Exact Methods

1) Cramer's Rule: This method is one of the least efficient for solving a large number of non-homogeneous linear equations. In general, given:

ijij

bxA : nji ,,3,2,1

A

A

A

Ax

jj

i

det

det

where,

A : is the determinant of the coefficient matrix A .

jA : is the determinant of the coefficient matrix A with its j th column replaced by

the i

b vector.

bAx

2

Example: Use Cramer's Rule to solve the following system of equations:

53221 xx

521 xx

Solution:

bAx

5

5

11

32

2

1

x

x

4

11

32

15

35

1

1

A

Ax

1

11

32

51

52

2

2

A

Ax

1

4

2

1

x

x

Elementary Row Operations:

(1) Interchange rows k and j .

(2) Multiply row k by 0 . (3) Add times row j to row k .

3

2) Gauss-Jordan Elimination Method:

We can solve bAx using Gauss-Jordan elimination method. Algorithm:

1. Form the augmented matrix: )1(, nnbAC

2. For 1j to n do

{

a) Compute the pivot index "p": jpn such that:

ij

n

jipj

CC

max

b) If 0pj

C , exit (no solution).

c) If jp , interchange rows p and j .

d) Divide row j by the pivot jj

C .

e) For each ji , subtract ij

C times row j from row i . Ri = jiji

RCR

}

3. Partition C as xIC , .

Example: Solve the following system of equations using Gauss-Jordan Elimination method:

6

1

2

210

122

011

3

2

1

x

x

x

Solution:

436210

1122

2011

C

1j , 13 p 2max3

ijji

Cp

0221

p

C

12 jp interchange rows 2 and 1

6210

2011

1122

2

1R

6210

2011

5.05.011

4

6210

5.15.000

5.05.011

2j 23 p , 3p

132C

23 interchange rows 3 and 2

5.15.000

6210

5.05.011

1

2R

5.15.000

6210

5.05.011

5.15.000

6210

5.65.101

3j 33 p , 3p

05.033

C

5.0

3R

3100

6210

5.65.101

3100

6210

5.65.101

3232

3131

RCR

RCR

3100

0010

2001

I x

3

2

1

3

0

2

x

x

x

x

row2 - C21.row1

row3 - C31.row1

row1 - C12.row2 row3 - C32.row2

5

3) Matrix Inverse Method:

bAx bAx 1

- First, find 1

nxnA

- Then, bAx 1 Algorithm:

1. Form the augmented matrix: nnIAC 2,

2. For 1j to n do

{ a) Compute the pivot index jpn such that:

ij

n

jipj

CC

max

b) If 0pj

C , exit (no inverse).

c) If jp , interchange rows p and j .

d) Divide row j by the pivot jj

C .

e) For each ji , subtract ij

C times row j from row i .

}

3. Partition C as 1, AIC .

Example: Consider the following 33 matrix:

120

402

011

A Find 1A

Solution:

63100120

010402

001011

,

IAC

1,

2.02.04.0100

4.01.02.0010

4.01.08.0001

AI

2.02.04.0

4.01.02.0

4.01.08.01A

6

4) Gaussian Elimination Method:

bAx

)1(, nnbAC )1(, nneD

D : is an upper-triangular matrix (matrix with zeros below the main diagonal) Algorithm:

1. Form the augmented matrix: )1(, nnbAC .

2. For 1j to n do

{ a) Compute the pivot index jpn such that:

ij

n

jipj

CC

max

b) If 0pj

C , exit (no solution).

c) If jp , interchange rows p and j .

d) For each ji , subtract

jj

ij

C

Ctimes row j from row i .

Ri

j

jj

ij

iR

c

cR

}

3. Partition C as eDC , , where D is an nn and e is 1n .

4. For nj down to 1, compute:

{

n

jiijij

jj

jxDe

Dx

1

1

}

Example: Solve the following linear algebraic system using Gaussian Elimination method:

814

312

201

2

1

1

3

2

1

x

x

x

7

Solution:

2814

1312

1201

C

1j :

431C , 3p

1201

1312

2814

1

11

21

2R

c

cR

1

11

31

3R

c

cR

5.0025.00

215.10

2814

2j

3j

8333.01667.000

215.10

2814

C

3j

58333.01667.0

1 3

43

i

ijixDx

25.1

35)1(2

5.1

12

5.1

12

5.1

1323

3

32

xDxDxi

iji

9)58()21(24

12

4

12

4

1313212

3

21

xDxDxDxi

iji

5

2

9

3

2

1

x

x

x

0

8

LU Decomposition (Triangular Factorization)

Definition: The nonsingular matrix A has a triangular factorization if it can be expressed as the

product of a lower triangular matrix L and an upper triangular matrix U.

A = LU

L: it has 1’s along the diagonal.

U: it has nonzero diagonal elements.

L =

1

01

001

0001

21

3231

21

nn mm

mm

m U =

nn

n

u

u

uuu

000

00

0 22

11211

e.g

Given,

A =

621

542

134

Convert the matrix A to the form A=LU.

Solution:

621

542

134

=

1

01

001

3231

21

mm

m

33

2322

131211

00

0

u

uu

uuu

4 = u11 , 3 = u12 , -1 = u13

m21 u11 = -2 → m 21 = - 0.5

m21u12 + u22 = - 4 → m21 = - 4 – (- 0.5*3) = -2.5

m21u13 + u23 = 5 → u23 = 5 – (- 0.5*-1) = 4.5

m31u11 = 1 → m31 = 0.25

m31u12 +m32u22 = 2 → m32 = 5.2

3*4

12

= -0.5

m31u13 + m32u23 + u33 = 6

u33 = 5.8)5.4*2

1(1*

4

16

621

542

134

=

5.800

5.45.20

134

15.025.0

015.0

001

A = L U

9

5) Solution of a Linear System: AX = b

Suppose that the coefficient matrix A of the linear system: AX = b has a triangular

factorization A = LU, then the solution to: AX = b → LUX = b can be obtained by

defining:

Y = UX and then solving two systems:

First, solve LY = b for Y using forward substitution.

Then, solve UX = Y for X using back substitution.

e.g.

Solve the following system of equations using the triangular factorization method.

82

79

52

21

610124

88103

4682

1421

4

3

2

1

x

x

x

x

Solution:

1. A = L U

44

3433

242322

14131211

434241

3231

21

000

00

0

1

01

001

0001

610124

88103

4682

1421

u

uu

uuu

uuuu

mmm

mm

m

Find the values of the m’s and u’s. Therefore,

L =

1214

0113

0012

0001

, U =

6000

3200

2240

1421

2. solve LY = b for Y using forward substitution:

1214

0113

0012

0001

4

3

2

1

y

y

y

y

=

82

79

52

21

10

y1 = 21

2y1 + y2 = 52 → y2 = 10

3y1 + y2 + y3 = 79 → y3 = 6

4y1 + y2 + 2y3 + y4 = 82 → y4 = -24

Y =

24

6

10

21

3. solve UX = Y for X using back substitution:

6000

3200

2240

1421

4

3

2

1

x

x

x

x

=

24

6

10

21

x1 + 2x2 + 4x3 + x4 = 21 → x1 = 1

4x2 – 2x3 +2x4 = 10 → x2 = 2

-2x3 +3x4 = 6 → x3 = 3

- 6x4 = -24 → x4 = 4

X =

4

3

2

1

is the solution of the system of the equations.

e.g.

Show that the following matrix can not be factored directly as: A = LU.

A =

532

184

621

11

ILL-Conditional Systems:

Consider the following system:

100

100

401200

200100

2

1

x

x

2011x , 100

2x

Now, consider the system:

100

100

400200

200101

2

1

x

x

501x , 75.24

2x

Clearly the solution is very sensitive to the values of A in this case. When the solution is highly sensitive to the values of the coefficient matrix A or the right-hand side vector b , the equations are said to be ill-conditioned.

Vector and Matrix Norms: A norm of a vector is a generalization of the absolute value of a scalar. That is, a norm is a measure of the length or magnitude of the vector. There are several ways to define a norm of an 1n vector x .

1)

l norm i

n

i

xx max1

x : is called the infinity norm of x .

2) 2

l norm

n

ii

xx1

2

2

2l norm: is called the Euclidean norm (distance from the origin).

In order to develop a measure of the sensitivity of a system, we need to consider the notion of a matrix norm:

n

jij

n

i

AA11

max

12

Example:

13

12

1

.

137

285

964

3

2

1

x

x

x

19}11,15,19max{3

1

3

1max

j

iji

AA

The solution of this system is:

1

0

2

3

2

1

x

x

x

2}1,0,2max{

X

5142

X

Properties of norms:

1. 00 xforx .

2. 00 xforx .

3. xx . : is a scalar.

4. yxyx .

5. xAAx . .

Condition Number:

AX = b

Consider the effect of changes in the right-hand side vector b, say from b to b+∆b.

Let X denotes the solution to the original system, and let X + ∆X denotes the

solution to the perturbed system. Then, A(X + ∆X) = b+∆b which reduces to A ∆X

= ∆b because AX = b

∆X = A-1

∆b

13

Using:

AX ≤ XA .

X ≤ bA .1

Using the properties of norms, the relative change in the solution can be expressed in

terms of the perturbation as follows:

b

bAK

X

X

)(

K(A): is a scalar called the condition number of the matrix A and is defined as follows,

assuming A is nonsingular.

K(A) = 1. AA

Note that K(A) is a measure of the relative sensitivity of the solution to changes in b .

Properties of the condition number:

1. When K(A) becomes large, the system is regarded as being an ill-conditioned.

e.g

A =

401200

200100 determine K(A).

Solution :

A-1

=

12

201.4

valuelarge01.361201.6*601)(

01.6}3,01.6max{

601}601,300max{1

AK

A

A

Our system is ill-conditional system.

2. ∞> K(A) ≥ 1

3. K(I) = 1

4. K(CA) = K(A) C is a scalar.

5. If K(A) is large , then A is close to singular matrix.

14

e.g.

Given, b =

7.9

1.4 , x=

0

1

if b is changed to bnew =

7.9

11.4 and x is changed to xnew =

97.0

34.0, determine K(A).

K(A) ≥

b

b

X

X

∆X =

97.0

66.0 ∆b =

0

01.0

7.9,01.0

1,97.0

bb

XX

K(A) ≥ 9.9407.9/01.0

1

97.0

K(A) ≥940.9

Second: Iterative (Numerical) Methods

1) Jacobi method. 2) Gauss-Seidel method. 3) Relaxation methods:

a. Successive Under-Relaxation.

b. Successive Over-Relaxation.

These are methods by which an approximation to the solution of a system of linear algebraic equations may be obtained. The iterative procedure may not always yield a solution. A sufficient condition for a solution to be found is that the absolute value of the diagonal coefficient in any equation must be greater than the sum of the absolute values of all other coefficients appearing in that equation.

1:

knAAkj

kjkk

15

Example:

3333232131

2323222121

1313212111

bxaxaxa

bxaxaxa

bxaxaxa

Test:

131211aaa

232122aaa

323133aaa

To start the iterative scheme, an initial guess 0x must be chosen. For example, if

nothing is known about the solution, the initial guess 00 x is as good as any. Typically, the iterations are repeated until the norm of the difference between successive estimates is sufficiently small, that is:

)()1( kk xx " : tolerance"

1) Jacobi Method:

Jacobi iteration can be written as a set of scalar equations expressed directly in

terms of the components of A and b ,

1:1 )()1(

inxAbA

xij

k

jiji

ii

k

i

Example: Given the following system of equations:

4

2

7

.

430

153

205

3

2

1

x

x

x

Use Jacobi iterative method to find 21

, xx and 3

x .

Do two iterations, let the initial guess 00 x .

205

135 304

16

)27(5

1 )(

3

)1(

1

kk xx

)32(5

1 )(

3

)(

1

)1(

2

kkk xxx

)34(4

1 )(

2

)1(

3

kk xx

1st iteration (k=0)

4.15

7)27(

5

1 )0(

3

)1(

1 xx : )1(

1x is the value of

1x in the first iteration.

4.05

2)32(

5

1 )0(

3

)0(

1

)1(

2 xxx

0.14

4)34(

4

1 )0(

2

)1(

3 xx

?)0()1( xx

0.1

4.0

4.1

0

0

0

0.1

4.0

4.1)0()1( xx

4.1)0()1( xx

2nd

iteration (k=1)

0.1)]0.12(7[5

1)27(

5

1 )1(

3

)2(

1 xx

24.0)]0.1()4.13(2[5

1)32(

5

1 )1(

3

)1(

1

)2(

2 xxx

7.0)]4.03(4[4

1)34(

4

1 )1(

2

)2(

3 xx

?)1()2( xx

0

0 0

17

3.0

64.0

4.0

0.1

4.0

4.1

7.0

24.0

0.1)1()2( xx

64.0)1()2( xx

Jacobi Estimates:

k 1x 2

x 3x

0 0.0000000 0.0000000 0.0000000

1 1.4000000 0.4000000 - 1.0000000

2 1.0000000 - 0.2400000 - 0.7000000

3 1.1200000 - 0.0600000 - 1.1799999

4 0.9280000 - 0.0360000 - 1.0450000

5 0.9820000 0.0522000 - 1.0270000

6 0.9892000 0.0162000 - 0.9608500

7 1.0156599 - 0.0013500 - 0.9878500

8 1.0048599 - 0.0118259 - 1.0010124

9 0.9995950 - 0.0027135 - 1.0088694

2) Gauss-Seidel Method:

The convergence rate of the Jacobi method can be improved using Gauss-Seidel method. In general, it is better to take advantage of the most recent information when updating an estimate of a solution as follows:

1:1 1

1 1

)()1()1(

inxAxAbA

xi

j

n

ij

k

jij

k

jiji

ii

k

i

18

Example:

Use Gauss-Seidel iterative method to find 21

, xx and 3

x for the same system in the

previous example:

4

2

7

.

430

153

205

3

2

1

x

x

x

Do two iterations, and let the initial guess 00 x .

Solution:

)27(5

1 )(

3

)1(

1

kk xx

)32(5

1 )(

3

)1(

1

)1(

2

kkk xxx

)34(4

1 )1(

2

)1(

3

kk xx

1st iteration (k=0)

4.15

7)27(

5

1 )0(

3

)1(

1 xx

44.0]0)4.13(2[5

1)32(

5

1 )0(

3

)1(

1

)1(

2 xxx

33.1)]44.03(4[4

1)34(

4

1 )1(

2

)1(

3 xx

?)0()1( xx

33.1

44.0

4.1

0

0

0

33.1

44.0

4.1)0()1( xx

4.1)0()1( xx

2nd

iteration (k=1)

868.0)]33.12(7[5

1)27(

5

1 )1(

3

)2(

1 xx

1452.0)]33.1()868.03(2[5

1)32(

5

1 )1(

3

)2(

1

)2(

2 xxx

8911.0)]1452.03(4[4

1)34(

4

1 )2(

2

)2(

3 xx

19

?)1()2( xx

4389.0

5852.0

532.0

33.1

44.0

4.1

8911.0

1452.0

868.0)1()2( xx

5852.0)1()2( xx

Gauss-Seidel Estimates:

k 1x 2

x 3x

0 0.0000000 0.0000000 0.0000000

1 1.4000000 - 0.4400000 - 1.3299999

2 0.8680000 0.1452000 - 0.8911000

3 1.0435599 - 0.0479159 - 1.0359370

4 0.9856252 0.0158123 - 0.9881408

5 1.0047437 - 0.0052180 - 1.0039135

6 0.9984345 0.0017220 - 0.9987085

7 1.0005165 - 0.0005682 - 1.0004262

8 0.9998295 0.0001875 - 0.9998593

9 1.0000563 - 0.0000619 - 1.0000464

20

Jacobi vs Gauss-Seidel Estimates:

k 1

x 2x 3

x

Jacobi Gauss-Seidel Jacobi Gauss-Seidel Jacobi Gauss-Seidel

0 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000

1 1.4000000 1.4000000 0.4000000 - 0.4400000 - 1.0000000 - 1.3299999

2 1.0000000 0.8680000 - 0.2400000 0.1452000 - 0.7000000 - 0.8911000

3 1.1200000 1.0435599 - 0.0600000 - 0.0479159 - 1.1799999 - 1.0359370

4 0.9280000 0.9856252 - 0.0360000 0.0158123 - 1.0450000 - 0.9881408

5 0.9820000 1.0047437 0.0522000 - 0.0052180 - 1.0270000 - 1.0039135

6 0.9892000 0.9984345 0.0162000 0.0017220 - 0.9608500 - 0.9987085

7 1.0156599 1.0005165 - 0.0013500 - 0.0005682 - 0.9878500 - 1.0004262

8 1.0048599 0.9998295 - 0.0118259 0.0001875 - 1.0010124 - 0.9998593

9 0.9995950 1.0000563 - 0.0027135 - 0.0000619 - 1.0088694 - 1.0000464

3) Successive Relaxation (SR) Method: The convergence rate can sometimes be increased by introducing a relaxation factor

0 ,

1:)1(1

1 1

)()1()()1(

inxAxAbA

xxi

j

n

ij

k

jij

k

jiji

ii

k

i

k

i

If: 1 , SR method reduces to the Gauss-Seidel method. If: 01 Successive under relaxation (SUR). If: 12 Successive over relaxation (SOR).

Example:

Use Successive Relaxation method to find 21

, xx and 3

x in the previous example

4

2

7

.

430

153

205

3

2

1

x

x

x

Do two iterations, let the initial guess 00 x and 1.1 .

21

Solution:

)1(

1

kx )07(5

1.11.0

3

2

)()(

1

j

k

jij

k xAx

)27(22.01.0 )(

3

)(

1

kk xx

)(

3

)(

144.054.11.0 kk xx

)1(

2

kx )2(5

1.11.0

3

3

)(

2

1

1

)1(

2

)(

2

j

k

jjj

k

jj

k xAxAx

)32(22.01.0 )(

3

)1(

1

)(

2

kkk xxx

)(

3

)1(

1

)(

222.066.044.01.0 kkk xxx

)1(

3

kx )04(4

1.11.0

2

1

)1(

3

)(

3

j

k

jj

k xAx

)304(275.01.0 )1(

2

)(

3

kk xx

)1(

2

)(

3825.01.11.0 kk xx

)1(

1

kx 54.144.01.0 )(

3

)(

1 kk xx

)1(

2

kx 44.022.066.01.0 )(

3

)1(

1

)(

2 kkk xxx

)1(

3

kx 1.1825.01.0 )1(

2

)(

3 kk xx

1st iteration (k=0)

54.154.100)1(

1x

5764.044.0)54.166.0(44.0066.00 )1(

1

)1(

2 xx

5755.11.1)5764.0825.0(1.1825.00 )1(

2

)1(

3 xx

5755.1

5764.0

54.1)0()1( xx

5755.1)0()1( xx

22

2nd

iteration (k=1)

6928.054.144.01.0 )1(

3

)1(

1

)2(

1 xxx

387.044.022.066.01.0 )1(

3

)2(

1

)1(

2

)2(

2 xxxx

6232.01.1825.01.0 )2(

2

)1(

3

)2(

3 xxx

9523.0

9634.0

8472.0

5755.1

5764.0

54.1

6232.0

387.0

6928.0)1()2( xx

9634.0)1()2( xx

Solving a system of m-equations in n-unknowns with m > n

(more equations than unknowns) in the least square sence

(Normal equations)

Solve: Am×n Xn×1 = bm×1

Such systems can be solved using Normal equations as follows:

AT AX = AT b

Note:

A T

mn Am×n → results in a square matrix of size n×n

A T

mn bm×1 → results in a vector of size n×1

Now, ATAX = AT b

Can be solved using the previous exact methods (Gauss-Jordan

elimination, inverse method, Gaussian method, etc…)

23

e.g.

Find the least squares solution to the inconsistent system (more

equations than unknowns):

x1 + 4x2 = -2

x1 + 2x2 = 6

2x1 + 3x2 = 1

Solution:

A =

1

6

2

,

32

21

41

b

ATA =

7

6,

2912

126bAT

7

6

2912

126

2

1

x

x

Solving the system above,

1

3

2

1

X

XX → least squares solution

This solution is NOT an exact solution but rather an approximate

that minimizes the distance between AX and b (i.e. it minimizes

the length of the error vector: e = AX – b).