Upload
ngoque
View
232
Download
5
Embed Size (px)
Citation preview
Computational Linear Algebra Syllabus
NUMERICAL ANALYSIS
Linear Systems of Equations and Matrix Computations
Module 1: Direct methods for solving linear system of equation
Simple Gaussian elimination method, gauss elimination method with partial pivoting, determinant evaluation, gauss Jordan method, L U decompositions Doolittle’s lu decomposition, Doolittle’s method with row interchange.
Module 2: Iterative methods for solving linear systems of equations
Iterative methods for the solution of systems equation, Jacobin iteration, gauss – seidel method, successive over relaxation method (sort method). Module 3: Eigenvalues and Eigenvectors
An introduction, eigenvalues and eigenvectors, similar matrices, hermitian matrices, gramm – Schmidt orthonormalization, vector and matrix norms. Module 4: Computations of eigenvaues
Computation of eigenvalues of a real symmetric matrix, determination of the eigenvalues of a real symmetric tridiagonal matrix, tridiagonalization of a real symmetric matrix, Jacobin iteration for finding eigenvalues of a real symmetric matrix, the q r decomposition, the Q-R algorithm.
Vittal Rao/IISc, Bangalore V1/1-4-04/1
Computational Linear Algebra Syllabus
Lecture Plan Modules Learning Units Hours per
Topics Total Hours
1. Simple Gaussian elimination method 1
2. Gauss elimination method with partial
pivoting.
2
3. Determinant evaluation 1
4. Gauss Jordan method 1
5. L U decompositions 1
6. Doolittle’s LU Decomposition 2
1. Direct methods for solving linear system of equation.
7. Doolittle’s method with row interchange. 2
10
8. Iterative methods for the solution of systems equation
3
9. Jacobi iteration. 2
10. Gauss – Seidel method 2
2. Iterative methods for solving linear systems of equations.
11. Successive over relaxation method (sort method).
2
9
12. An introduction. 1
13. Eigenvalues and eigenvectors, 2
14. Similar matrices, 2
15. Hermitian matrices. 1
16. Gramm – Schmidt orthonormalization, 2
3. Eigenvalues and Eigenvectors
17. Vector and matrix norms. 1
9
18. Computation of eigenvalues 2
19. Computation of eigenvalues of a real symmetric matrix.
2
20. Determination of the eigenvalues of a real symmetric tridiagonal matrix,
2
21. Tridiagonalization of a real symmetric matrix
1
22. Jacobian iteration for finding eigenvalues of a real symmetric matrix
1
4. Computations of eigenvalues.
23. The Q R decomposition 1
11
Vittal Rao/IISc, Bangalore V1/1-4-04/2
Computational Linear Algebra Syllabus
24. The Q-R algorithm. 2
Vittal Rao/IISc, Bangalore V1/1-4-04/3
Numerical Analysis/ Direct methods for solving linear Lecture Notes
system of equation
1. DIRECT METHODS FOR SOLVING LINEAR SYSTEMS OF EQUATIONS
1.1. SIMPLE GAUSSIAN ELIMINATION METHOD Consider a system of n equations in n unknowns,
a11x1 + a12x2 + …. + a1nxn = y1
a21x1 + a22x2 + …. + a2nxn = y2
… … … … …
an1x1 + an2x2 + …. + annxn = yn
We shall assume that this system has a unique solution and proceed to describe the simple Gaussian elimination method for finding the solution. The method reduces the system to an upper triangular system using elementary row operations (ERO). Let A(1) denote the coefficient matrix A.
A(1) = Where a
⎟⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜⎜
⎝
⎛
nnnn
n
n
aaa
aaaaaa
)1(2
)1(1
)1(
2)1(
22)1(
21)1(
1)1(
12)1(
11)1(
...................................................
.....
.....
(1)ij = aij
Let
y(1) = Where y
(1 )1
(1 )2
(1 )n
yy
y
⎛ ⎞⎜ ⎟⎜ ⎟⎜⎜ ⎟⎜ ⎟⎝ ⎠
M ⎟ (1)i = yi
We assume a(1)11 ≠ 0
Then by ERO of type applied to A(1) reduce all entries below a(1)11 to zero. Let the
resulting matrix be denoted by A(2).
A(1) ⎯⎯⎯ →⎯ + 11)1( RmR ii
A(2) Where ;11
)1(1
)1(
1)1(
aam i
i −= i > 1.
Note A(2) is of the form
VittalRao/IISc, Bangalore M1/L1and L2/V1/May2004/1
Numerical Analysis/ Direct methods for solving linear Lecture Notes
system of equation
M1/L1and L2/V1/May2004/2
A(2) =
⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜
⎝
⎛
nnn
n
n
n
aa
aaaaaaa
)2(2
)2(
3)2(
32)2(
2)2(
22)2(
1)1(
12)1(
11)1(
......0...............
......0
......0
......
Notice that the above row operations on A(1) can be effected by premultiplying A(1) by M(1) where
M(1) =
⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜
⎝
⎛
−
)1(1
1)1(
31
)1(21
00001
n
n
m
Imm
M
i.e.
M(1) A(1) = A(2)
Let
y(2) = M(1) y(1) i.e )2()1( 11 yy RmR ii ⎯⎯⎯ →⎯ +
Then the system Ax = y is equivalent to
A(2)x = y(2)
Next we assume
a(2)22 ≠ 0
and reduce all entries below this to zero by ERO
A(2) ⎯⎯⎯ →⎯ + 2)2(ii mR
A(3) ; ;22
)2(2
)2(
2)2(
aam i
i −= i > 2.
Here
1 0 0 . . . 0
0 1 0 . . . 0
0 m(2)32
VittalRao/IISc, Bangalore
M(2) = 0 m(2)
42 In-2
Numerical Analysis/ Direct methods for solving linear Lecture Notes
system of equation
.
0 m(2)n2
and M(2) A(2) = A(3) ; M(2) y(2) = y(3) ;
and A(3) is of the form
⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜
⎝
⎛
=
nnn
n
n
n
aa
aaaaaaaa
A
)3(3
)3(
3)3(
33)3(
2)2(
23)2(
22)2(
1)1(
12)1(
11)1(
)3(
...00
...
...00
...0
......
MMMM
We next assume a(3)33 ≠ 0 and proceed to make entries below this as zero. We thus get
M(1), M(2), …. , M(r) where
1 0 …. 0
0 1 …. 0
0 …… 1
m(r)r+1r
M(r) = rxr m(r)r+2r In-r
.
.
.
m(r)nr
⎟⎟⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜⎜⎜
⎝
⎛
==
++
+
++
+++
+
nnr
nrr
nrr
rrr
rnr
rrr
n
n
rrr
aa
aaaaaaaa
AAM
)1(1
)1(
1)1(
11)1(
)()(
2)2(
22)2(
1)1(
11)1(
)1()()(
...000.........
...0
......0
.........0
............
MMM
MM
M
VittalRao/IISc, Bangalore M1/L1and L2/V1/May2004/3
Numerical Analysis/ Direct methods for solving linear Lecture Notes
system of equation
M(r) y(r) = y(r+1)
At each stage we assume a(r)rr ≠ 0.
Proceeding thus we get, M(1), M(2), …. , M(n-1) such that
M(n-1) M(n-2) …. M(1) A(1) = A(n) ; M(n-1) M(n-2) …. M(1) y(1) = y(n)
a(1)11 a(1)
12 . . . . . a(1)1n
a(2)22 . . . . . a(2)
2n
where A(n) = .
.
a(n)nn
which is an upper triangular matrix and the given system is equivalent to
A(n)x = y(n)
and since this is an upper triangular, this can be solved by backward substitution; and hence the system can be solved easily
Note further that each M(r) is a lower triangular matrix with all diagonal entries as 1. Thus let M(r) is 1 for every r. Now,
A(n) = M(n-1) …. M(1) A(1)
Thus
det A(n) = det M(n-1) det M(n-2) …. det M(1) det A(1)
det A(n) = det A(1) = det A since A = A(1)
Now A(n) is an upper triangular matrix and hence its determinant is
a(1)11 a(2)
22 …. a(n)
nn. Thus det A is given by
det A = a(1)11 a(2)
22 …. a(n)
nn
Thus the simple GEM can be used to solve the system Ax = y and also to evaluate det A provided a(i)
ii ≠ 0 for each i. Further note that M(1), M(2), …. , M(n-1) are lower triangular, and nonsingular as their det = 1 ≠ 0. They are all therefore invertible and their inverses are all lower triangular, i.e. if L = M(n-1) M(n-2) …. M(1) then L is lower triangular, and nonsingular and L-1 is also lower triangular.
Now LA = LA(1) = M(n-1) M(n-2) …. M(1) A(1) = A(n)
VittalRao/IISc, Bangalore M1/L1and L2/V1/May2004/4
Numerical Analysis/ Direct methods for solving linear Lecture Notes
system of equation
Therefore A = L(-1) A(n)
Now L(-1) is lower triangular which we denote by α and A(n) is upper triangular which we denote by u, and we thus get the so called αu decomposition
A = αu of a given matrix A – as a product of a lower triangular matrix with an upper triangular matrix. This is another application of the simple GEM. REMEMBER IF AT ANY STAGE WE GET a(1)
ii = 0 WE CANNOT PROCEED FURTHER WITH THE SIMPLE GSM.
EXAMPLE:
Consider the system
x1 + x2 + 2x3 = 4
2x1 - x2 + x3 = 2
x1 + 2x2 = 3
Here
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−=
021112211
A ⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛=
324
y
)2(2
)1(
210330
211
021112211
12
13
AARR
RR=
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
−−−
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−= →
−
−
a(1)11 = 1 ≠ 0 m(1)
21 = -2 a(2)22 = -3 ≠ 0
m(1)31 = -1
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
−−=
101012001
)1(M )1()1()2()1(
16
4
324
yMyy ==⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
−−→
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛=
VittalRao/IISc, Bangalore M1/L1and L2/V1/May2004/5
Numerical Analysis/ Direct methods for solving linear Lecture Notes
system of equation
)3(31
)2(
210330
21123
AARR
=⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
−−−→
+
a(3)33 = -3
M(2)31 = 1/3
⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜
⎝
⎛
=
1310
010001
)2(M ⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
−−==
36
4)2()2()3( yMy
Therefore the given system is equivalent to A(3)x = y(3)
x1 + x2 + 2x3 = 4
-3x2 - 3x3 = -6
- 3x3 = -3
Backward Substitution
x3 = 1
-3x2 - 3 = - 6 ⇒ -3x2 = -3 ⇒ x2 = 1
x1 + 1 + 2 = 4 ⇒ x1 = 1
Thus the solution of the given system is,
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛=
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛=
111
3
2
1
xxx
x
The determinant of the given matrix A is
a(1)11 a(2)
22 a(3)33 = (1) (-3) (-3) = 9.
Now
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛=−
101012001
)1(1M
⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜
⎝
⎛
−
=−
1310
010001
)1(2M
VittalRao/IISc, Bangalore M1/L1and L2/V1/May2004/6
Numerical Analysis/ Direct methods for solving linear Lecture Notes
system of equation
L = M(2) M(-1)
L-1 = (M(2) M(1))-1 = (M(1))-1 (M(2))-1
= ⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
101012001
⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜
⎝
⎛
− 1310
010001
L = L(-1) = ⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜
⎝
⎛
− 1311
012001
u = A(n) = A(3) = ⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
−−−
300330
211
Therefore A = lu i.e.,
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−
021112211
= ⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜
⎝
⎛
− 1311
012001
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
−−−
300330
211
is the lu decomposition of the given matrix.
We observed that in order to apply simple GEM we need a(r)rr ≠ 0 for each stage r. This
may not be satisfied always. So we have to modify the simple GEM in order to overcome this situation. Further, even if the condition a(r)
rr ≠ 0 is satisfied at each stage, simple GEM may not be a very accurate method to use. What do we mean by this? Consider, as an example, the following system:
(0.000003) x1 + (0.213472) x2 + (0.332147) x3 = 0.235262
(0.215512) x1 + (0.375623) x2 + (0.476625) x3 = 0.127653
(0.173257) x1 + (0.663257) x2 + (0.625675) x3 = 0.285321
Let us do the computations to 6 significant digits.
Here,
A(1) = ⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
625675.0663257.0173257.0476625.0375623.0215512.0332147.0213472.0000003.0
VittalRao/IISc, Bangalore M1/L1and L2/V1/May2004/7
Numerical Analysis/ Direct methods for solving linear Lecture Notes
system of equation
y(1) = a⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
285321.0127653.0235262.0
(1)11 = 0.000003 ≠ 0
3.71837000003.0215512.0
11)1(21
)1(
21)1( −=−=−=
aaM
3.57752000003.0173257.0
11)1(31
)1(
31)1( −=−=−=
aaM
M(1) = ; y⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
−−
103.57752013.71837001
(2) = M(1) y(1) = ⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
−−
6.135865.16900
235262.0
A(2) = M(1) A(1) = ⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
−−−−
7.191818.1232700.238609.153340
332147.0213472.0000003.0
a(2)22 = - 15334.9 ≠ 0
803905.09.153348.12327
22)2(32
)2(
32)2( −=
−−
−=−=aaM
M(2) = ⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
− 1803905.00010001
y(3) = M(2) y(2) = ⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
−−
20000.05.16900
235262.0
A(3) = M(2) A(2) = ⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
−−−
50000.0000.238609.153340
332147.0213472.0000003.0
Thus the given system is equivalent to the upper triangular system
A(3)x = y(3)
VittalRao/IISc, Bangalore M1/L1and L2/V1/May2004/8
Numerical Analysis/ Direct methods for solving linear Lecture Notes
system of equation
Back substitution yields,
x1 = 0.40 00 00
x2 = 0.47 97 23
x3 = -1.33 33 3
This compares poorly with the correct answers (to 10 digits) given by
x1 = 0.67 41 21 46 9
x2 = 0.05 32 03 93 39.1
x3 = -0.99 12 89 42 52
Thus we see that the simple Gaussian Elimination method needs modification in order to handle the situations that may lead to a(r)
rr = 0 for some r or situations as arising in the above example. In order to do this we introduce the idea of Partial Pivoting. The idea of partial pivoting is the following:
At the r th stage we shall be trying to reduce all the entries below the r th diagonal as zero. Before we do this we look at the entries in the r th diagonal and below it and then pick the one that has the largest absolute value and we bring it to the r th diagonal position by a row interchange, and then reduce the entries below the r th diagonal as zero. When we incorporate this idea at each stage of the Gaussian elimination process we get the GAUSS ELIMINATION METHOD WITH PARTIAL PIVOTING. We now illustrate this with a few examples:
Example:
x1 + x2 + 2 x3 = 4
2x1 – x2 + x3 = 2
x1 + 2x2 = 3
We have
Aavg = 324
021112211
−
1st Stage: The pivot has to be chosen as 2 as this is the largest absolute valued entry in the first column. Therefore we do
VittalRao/IISc, Bangalore M1/L1and L2/V1/May2004/9
Numerical Analysis/ Direct methods for solving linear Lecture Notes
system of equation
Aavg ⎯→⎯12R
342
021211112 −
Therefore we have
M(1) = and M⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
100001010
(1) A(1) = A(2) = ⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛ −
021211112
M(1) A(1) = y(2) = ⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
342
Next we have
R2 – ½ R1 2 -1 1 2
A(2)avg 0 3/2 3/2 3
R3 – ½ R1 0 5/2 -1/2 3
Here
M(2) =
⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜
⎝
⎛
−
−
1021
0121
001 ; M(2) A(2) = A(3) =
⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜
⎝
⎛
−
−
21
250
23
230
112
M2 y(2) = y(3) = ⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
232
Now at the next stage the pivot is 25 since this is the entry with the largest absolute value
in the 1st column of the next sub matrix. So we have to do another row interchange.
Therefore
2 -1 1 2
A(3)avg 0 5/2 -1/2 2 ⎯→⎯ 23R
VittalRao/IISc, Bangalore M1/L1and L2/V1/May2004/10
Numerical Analysis/ Direct methods for solving linear Lecture Notes
system of equation
0 3/2 3/2 3
M(3) = M⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
010100001
(3) A(3) = A(4) = ⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜
⎝
⎛
−−
23
230
21
250
112
M(3) y(3) = y(4) = ⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
322
Next we have
2 -1 1 2
A(4)avg ⎯⎯⎯ →⎯
− 23 53
RR 0 5/2 -1/2 2
0 0 9/5 9/5
Here
M(4) = ⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜
⎝
⎛
− 1530
010001
M(4) A(4) = A(5) =
⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜
⎝
⎛
−
−
590021
250
112
M(4) y(4) = y(5) = ⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜
⎝
⎛
5922
This completes the reduction and we have that the given system is equivalent to the system
A(5)x = y(5)
i.e.
2x1 – x2 + x3 = 2
25 x2 -
21 x3 = 2
59 x3 =
59
We now get the solution by back substitution:
VittalRao/IISc, Bangalore M1/L1and L2/V1/May2004/11
Numerical Analysis/ Direct methods for solving linear Lecture Notes
system of equation
The 3rd equation gives,
x3 = 1
using this in second equation we get
25 x2 -
21 = 2
giving 25 x2 =
25
and hence x2 = 1.
Using the values of x1 and x2 in the first equation we get
2x1 – 1 + 1 = 2 giving x1 = 1
Thus we get the solution of the system as x1 = 1, x2 = 1, x3 = 1; the same as we had obtained with the simple Gaussian elimination method earlier.
Example 2:
Let us now apply the Gaussian elimination method with partial pivoting to the following example:
(0.000003)x1 + (0.213472)x2 + (0.332147) x3 = 0.235262
(0.215512)x1 + (0.375623)x2 + (0.476625) x3 = 0.127653
(0.173257)x1 + (0.663257)x2 + (0.625675) x3 = 0.285321,
the system to which we had earlier applied the simple GEM and had obtained solutions which were for away from the correct solutions.
Note that
A = ⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
625675.0663257.0173257.0476625.0375623.0215512.0332147.0213472.0000003.0
y = ⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
285321.0127653.0235262.0
We observe that at the first stage we must choose 0.215512 as the pivot. So we have
A(1) = A A⎯→⎯ 12R (2) = ⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
625675.0663257.0173257.0332147.0213472.0000003.0476625.0375623.0215512.0
VittalRao/IISc, Bangalore M1/L1and L2/V1/May2004/12
Numerical Analysis/ Direct methods for solving linear Lecture Notes
system of equation
y(1) = y y⎯→⎯ 12R (2) = M⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
285321.0235262.0127653.0
(1) = ⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
100001010
Next stage we make all entries below 1st diagonal as zero
R2 + m21R1 0.215512 0.375623 0.476625
A(2) A(3) 0 0.213467 0.332140
R3 + m21R1 0 0.361282 0.242501
Where
m21 = - 11
21
aa
= - 215512.0000003.0 = - 0.000014
m31 = - 11
31
aa
= - 215512.0173257.0 = - 0.803932
M(2) = ; y⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
−−
10803932.001000014.0001
(2) = M(2) y(2) = ⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
182697.0235260.0127653.0
In the next stage we observe that we must choose 0.361282 as the pivot. Thus we have to interchange 2nd and 3rd row. We get,
A(3) A⎯→⎯ 23R (4) = ⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
332140.0213467.00242501.0361282.00476625.0375623.0215512.0
M(3) = y⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
010100001
(4) = M(3) y(3) = ⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
235260.0182697.0127653.0
Now reduce the entry below 2nd diagonal as zero
A(4) A⎯⎯⎯ →⎯ + 2323 RMR 5 = ⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
188856.000242501.0361282.00476625.0375623.0215512.0
VittalRao/IISc, Bangalore M1/L1and L2/V1/May2004/13
Numerical Analysis/ Direct methods for solving linear Lecture Notes
system of equation
M32 = - 361282.0213467.0 = - 0.590860
M(4) = y⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
− 159086.0010001
(5) = M(4) y(4) = ⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
127312.0182697.0127653.0
Thus the given system is equivalent to
A(5) x = y(5)
which is an upper triangular system and can be solved by back substitution to get
x3 = 0.674122
x2 = 0.053205 ,
x1 = 0.991291
which compares well with the 10 decimal accurate solution given at the end of page 9. Notice that while we got very bad errors in the solutions while using simple GEM whereas we have come around this difficulty by using partial pivoting.
VittalRao/IISc, Bangalore M1/L1and L2/V1/May2004/14
Numerical analysis/Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L3/V1/May 2004/1
DETERMINANT EVALUATION
Notice that even in the partial pivoting method we get Matrices
M(k), M(k-1) …. M(1) such that
M(k), M(k-1) …. M(1) A is upper triangular and therefore
det M(k), det M(k-1) …. det M(1) det A = Product of the diagonal entries in the final upper triangular matrix.
Now det M(i) = 1 if it refers to the process of nullifying entries below a diagonal to zero; and
det M(i) = 1 if it refers to a row interchange necessary for a partial pivoting.
Therefore det M(k) …. det M(1) = (-1)m where m is the number of row inverses effected in the reduction.
Therefore det A = (-1)m product of the diagonals in the final upper triangular matrix.
In our example 1 above, we had M(1), M(2), M(3), M(4) of which M(1) and M(3) referred to row interchanges. Thus therefore there were to row interchanges and hence
det A = (-1)2 (2)( 25 )(
59 ) = 9.
In example 2 also we had M(1), M(3) as row interchange matrices and
therefore det A = (-1)2 (0.215512) (0.361282) (0.188856) = 0.013608
LU decomposition:
Notice that the M matrices corresponding to row interchanges are no longer lower triangular. (See M(1) & M(3) in the two examples.) Thus,
M(k) M(k-1) . . . . . M(1)
is not a lower triangular matrix in general and hence using partial pivoting we cannot get LU decomposition in general.
Numerical Analysis/Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L3/V1/May 2004/1
GAUSS JORDAN METHOD
This is just the method of reducing Aavg to (AR / yR ) where AR = In is the Row Reduced Echelon Form of A (in the case A is nonsingular). We could also do the reduction here by partial pivoting.
Remark:
In case in the reduction process at some stage if we get arr = ar+1r = . . . . = ar+1n = 0, then even partial pivoting does not being any nonzero entry to rth diagonal because there is no nonzero entry available. In such a case A is singular matrix and we proceed to the RRE form to get the general solution of the system. As observed earlier, in the case A is singular, Gauss-Jordan Method leads to AR = In and the product of corresponding M(i) give us A-1.
Numerical Analysis / Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L5/V1/May 2004/1
LU decompositions
We shall now consider the LU decomposition of matrices. Suppose A is an nxn matrix. If L and U are lower and upper triangular nxn matrices respectively such that A = LU. We say that this is a LU decomposition of A. Note that LU decomposition is not unique. For example if A = LU is a decomposition then A = Lα Uα
is also a LU decomposition where α ≠ 0 is any scalar and Lα = α L and Uα = 1/α U.
Suppose we have a LU decomposition A = LU. Then, the system, Ax = y, can be solved as follows:
Set Ux = z …………… (1)
Then the system Ax = y can be written as,
LUx = y,
i.e.,
Lz = y ……………..(2)
Now (2) is a triangular system – infact lower triangular and hence we can solve it by forward substitution to get z.
Substituting this z in (1) we get an upper triangular system for x and this can be solved by back substitution.
Further if A = LU is a LU decomposition then det. A can be calculated as det. A = det. L . det. U = l11 l22 ….lnn u11u22 …..unn
Where lii are the diagonal entries of L and uii are the diagonal entries of U.
Also A-1 can be obtained from an LU decomposition as A-1 = U-1 L-1.
Thus an LU decomposition helps to break a system into Triangular system; to find the determinant; and to find the inverse of a matrix.
We shall now give methods to find LU decomposition of a matrix. Basically, we shall be considering three cases. First, we shall consider the decomposition Tridiagonal matrix; secondly the Doolittles’s method for a general matrix, and thirdly the Cholesky’s method for a symmetric matrix.
Numerical Analysis / Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L5/V1/May 2004/2
1 TRIDIAGONAL MATRIX
Let
⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜
⎝
⎛
=
−
−−
bcabc
abcabc
ab
A
i
iii
1
12
432
321
21
0........00....0
........................
........................0....00....00....00
be an nxn tridiagonal matrix. We seek a LU decomposition for this. First we shall give some preliminaries.
Let δi denote the determinant of the ith principal minor of A
ii
ii
i
bcabc
abcab
1
112
321
21
0....0....0
....................
....................0....0....0
−
−−
=δ
Expanding by the last row we get,
δi = bi δi-1 – ci-1 ai δi-2 ; I = 2,3,4, ……..
……..(I)
δi = b1
We define δi = 1
From (I) assuming that δi are all nonzero we get
1
21
1 −
−−
−
−=i
iiii
i
i acbδδ
δδ
setting ii
i k=−1δδ
this can be written as
).(..............................1
1 IIkackb
i
iiii
−−+=
Numerical Analysis / Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L5/V1/May 2004/3
Now we seek a decomposition of the form A = LU where,
⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜
⎝
⎛
=
− 1........00........................0....0100........010............01
1
2
1
nw
ww
L ;
⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜
⎝
⎛
=
−
n
nn
uu
uu
U
0....00....00
....................0....00....0
1
32
21
α
αα
i.e. we need the lower triangular and upper triangular parts also to be ‘tridiagonal’ triangular.
Note that if A = (Aij) then because A is tridiagonal, Aij is nonzero only when i and j differ by 1. i.e. only Ai-1i, Aii, Aii+1 are nonzero. In fact,
Ai-1i = ai
Aii = bi …………….. (III)
Ai+1i = ci
In the case of L and U we have
Li + 1i = wi
Lii = 1 …………….. (IV)
Lij = 0 if j>i or j<i and i-j ≥ 2.
Uii+1 = αi+1
Uii = ui ……………………. (V)
Uij = 0 if i>j or i<j and j-I ≥ 2.
Now A = LU is what is needed.
Therefore,
∑=
=n
kkjikij VIULA
1).........(..........
Therefore
∑=
−− =n
kkikiii ULA
111
Using (III), (IV) AND (V) we get
Numerical Analysis / Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L5/V1/May 2004/4
Therefore
iiiiii ULa α== −−− 111
Therefore
αi = ai ………………….. (VII)
This straight away gives us the off diagonal entries of U. From (VI) we also get
∑=
=n
kkiikii ULA
1
iiiiiiii ULUL += −− 11
Therefore
)......(..........1 VIIIuwb iiii += − α
From (VI) we get further,
∑=
++ =n
kkikiii ULA
111
iiiiiiii ULUL 1111 ++++ +=
iii uWc =
Thus ………………… (IX) iii uWc =
Using (IX) in (VIII) we get (also using αI = ai)
ii
iii u
uac
b +=−
−
1
1
Therefore
)..(....................1
1 Xu
acubi
iiii
−
−+=
Comparing (X) with (II) we get
).(....................1
XIkui
iii
−
==δδ
Numerical Analysis / Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L5/V1/May 2004/5
using this in (IX) we get
).......(....................1 XIIc
uc
wi
ii
i
ii δ
δ −==
From (VII) we get
αI = ai ……………….(XIII) (XI), (XII) and (XIII) completely determine the matrices L and U and hence we get the LU decomposition.
Note : We can apply this method only when δI are all nonzero. i.e. all the principal minors have nonzero determinant.
Example:
Let
⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜
⎝
⎛
−−−−
−−
=
1300013900025200011200022
A
Let us now find the LU decomposition as above.
We have
b1 = 2 b2 = 1 b3 = 5 b4 = -3 b5 = -1
c1 = -2 c2 = -2 c3 = 9 c4 = 3
a2 = -2 a3 = 1 a4 = -2 a5 = 1
We have
δ0 = 1
δ1 = 2
δ2 = b2 δ1 – a2 c1 δ0 = 2-4 = -2
δ3 = b3 δ2 – a3 c2 δ1 = (-10) – (-2) (2) = -6
δ4 = b4 δ3 – a4 c3 δ2= (-3) (-6) – (-18) (-2) = -18
δ5 = b5 δ4 – a5 c4 δ3 = (-1) (-18) – (3) (-6)= 36.
Note δ1,δ2,δ3,δ4,δ5 are all nonzero. So we can apply the above method.
Therefore by (XI) we get
Numerical Analysis / Direct methods for solving linear system of equation Lecture notes
M1/L5/V1/May 2004/6
326;1
22;2
2
33
1
22
0
11 =
−−
==−=−
====δδ
δδ
δδ uuu
36
18
3
44 =
−−
==δδ
u ; and 218
36
4
55 −=
−==
δδu
From (XII) we get From (XIII) we get
122
1
11 −=
−==
ucw 222 −== aα
212
2
22 =
−−
==ucw 133 == aα
339
3
33 ===
ucw 244 −== aα
133
4
44 ===
ucw 155 == aα
Thus;
⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜
⎝
⎛−
=
1100001300001200001100001
L ;
⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜
⎝
⎛
−
−−−
=
2000013000023000011000022
U
In the above method we had made all the diagonal entries of L as 1. This will facilitate solving the triangular system LZ = y (equation (2)) in page 17. However by choosing these diagonals as 1 it may be that the ui, the diagonal entries in U are small thus creating problems in backward substitution for the system Ux = z (equation (1) on page 17). In order to avoid this situation Wilkinson suggests that in any triangular decomposition choose the diagonal entries of L and U to be of the same magnitude. This can be achieved as follows:
We seek
A = LU
where
VittalRao/IISc, Bangalore
Numerical Analysis / Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L5/V1/May 2004/7
l1
L = w1 l2
;
wn-1ln
⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜
⎝
⎛
=
−
n
nn
uu
uu
U
0....00....00
....................
........00....0
1
32
21
α
αα
Lii = li
Now Li+1i = wi
Lij = 0 if j>i or j<i and i-j ≥ 2
Uii = ui
Uii+1 = αi+1
Uij = 0 if i>j; or j>i and j-i ≥ 2
Now (VII), (VIII) and (IX) change as follows:
iii Aa 1−= ki
n
kki UL∑
−−=
11 iiii UL 111 −−−= iil α1−=
Therefore
ai = li-1 αI ………………. (VII`)
iii Ab = ki
n
kikUL∑
−
=1
iiiiiiii ULUL += −− 11 iiii ulW += − α1
`).......(..........1 VIIIulWb iiiii +=∴ − α
iii Ac 1+= ki
n
kki UL∑
−+=
11 iiii UL 1+= iiuw=
iii uwc = ……………….. (IX`)
From (VIII`) we get using (VII`) and (IX`)
Numerical Analysis / Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L5/V1/May 2004/8
iii
i
i
ii ul
la
uc
b +=−−
−
11
1 .
iiii
ii ululca
+=−−
−
11
1
`)..(....................1
1 Xppca
b ii
iii +=
−
−
where
pi = li ui
Comparing (X`) with (II) we get
1−==
i
iii kp δ
δ
therefore
1−=
i
iiiul δ
δ
we choose 1−
=i
iil δ
δ ………………. (XIV)
⎟⎠⎞⎜
⎝⎛=
−1sgn
i
iiu δ
δ
1−i
iδ
δ ………… (XV)
Thus li and ui have same magnitude. These then can be need to get wi and αi from (VII`) and (IX`). We get finally,
1−=
i
iil δ
δ ;
11
.sgn−−
⎟⎟⎠
⎞⎜⎜⎝
⎛=
i
i
i
iiu δ
δδδ
. . . . . . . . . . . .(XI`)
i
ii u
Cw = . . . . . . . . . . . . .. . . . . . . . . (XII`)
1−=
i
ii l
aα . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . .(XIII`)
These are the generalizations of formulae (XI), (XII) and (XIII).
Let us apply this to our example matrix (on page 21).
Numerical Analysis / Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L5/V1/May 2004/9
We get;
δ0 = 1 δ1 = 2 δ2 = -2 δ3 = -6 δ4 = -18 δ5 = 36
b1 = 2 b2 = 1 b3 = 5 b4 = -3 b5 = -1
c1 = -2 c2 = -2 c3 = 9 c4 = 3
a1 = -2 a3 = 1 a4 = -2 a5 = 1
We get δ1/δ0 = 2 ; δ2/δ1 = -1 ; δ3/δ2 = 3 ; δ4/δ3 = 3 ; δ5/δ4 = -2
Thus from (XI`) we get
l1 = √2 u1 = √2
l2 = 1 u2 = -1
l3 = √3 u3 = √3
l4 = √3 u4 = √3
l5 = √2 u5 = -√2
From (XII`) we get
222
1
11 −=
−==
uCw ; 2
12
2
22 =
−−
==uCw ;
333
9
3
33 ===
uCw ; 3
33
4
44 ===
uCw
From (XIII`) we get
222
1
22 −=
−==
laα ; 1
11
2
33 ===
laα ;
32
3
44
−==
laα ;
31
4
55 ==
laα
Thus, we have LU decomposition,
Numerical Analysis / Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L5/V1/May 2004/10
⎟⎟⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜⎜⎜
⎝
⎛
−
−−−
⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜
⎝
⎛
−=
⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜
⎝
⎛
−−−−
−−
=
200003
13000
032300
0011000022
23000033300003200001200002
1300013900025200011200022
A
L U
in which the L and U have corresponding diagonal elements having the same magnitude.
Numerical Analysis/Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L6/V1/May 2004/1
DOOLITTLE’S LU DECOMPOSITION
We shall now consider the LU decomposition of a general matrix. The method we describe is due to Doolittle.
Let A = (aij). We seek as in the case of a tridiagonal matrix, an LU decomposition in which the diagonal entries lii of L are all 1. Let L = (lii) ; U = (uij). Since L is a lower triangular matrix, we have
lij = 0 if j > i ; and by our choice, lij =1.
Similarly, since U is an upper triangular matrix, we have
uij = 0 if i > j.
We determine L and U as follows : The 1st row of U and 1st column of L are determined as follows :
∑=
=n
kkk ula
11111
= l11 u11 Since l1k = 0 for k>1
= u11 Since l11 = 1.
.111 =∴u In general,
∑=
=n
kkjkj ula
111
= l11 u11 Since l1k = 0 for k>1
= u1j Since l11 = 1.
Numerical Analysis/Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L6/V1/May 2004/2
⇒ u1j = a1j . . . . . . . . . (I) Thus the first row of U is the same as the first row of A. The first column of L is determined as follows:
∑=
=n
kkjkj ula
111
= lj1 u11 Since uk1 = 0 if k>1
⇒ lj1 = aj1/u11 . . . . . . . . . (II)
Note : u11 is already obtained from (I).
Thus (I) and (II) determine respectively the first row of U and first column of L. The other rows of U and columns of L are determined recursively as given below: Suppose we have determined the first i-1 rows of U and the first i-1 columns of L. Now we proceed to describe how one then determines the ith row of U and ith column of L. Since first i-1 rows of U have been determined, this means, ukj ; are all known for 1 ≤ k ≤ i-1 ; 1 ≤ j ≤ n. Similarly, since first i-1 columns are known for L, this means, lik are all known for 1 ≤ i ≤ n ; 1 ≤ k ≤ i-1.
Now
∑=
=n
kkjikij ula
1
Since l∑=
=i
kkjik ul
1ik = 0 for k>i
ijii
i
kkjik ulul += ∑
−
=
1
1
Numerical Analysis/Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L6/V1/May 2004/3
since lij
i
kkjik uul += ∑
−
=
1
1ii = 1.
∑−
=
−=⇒1
1
i
kkjikijij ulau
. . . . . ... . . . .(III)
Note that on the RHS we have aij which is known from the given matrix. Also the sum on the RHS involves lik for 1 ≤ k ≤ i-1 which are all known because they involve entries in the first i-1 columns of L ; and they also involve ukj ; 1 ≤ k ≤ i-1 which are also known since they involve only the entries in the first i-1 rows of U. Thus (III) determines the ith row of U in terms of the known given matrix and quantities determined upto the previous stage. Now we describe how to get the ith column of L :
∑=
=n
kkijkji ula
1
Since u∑=
=i
kkijk ul
1ki = 0 if k>i
iiji
i
kkijk ulul += ∑
−
=
1
1
⎥⎦
⎤⎢⎣
⎡−=⇒ ∑
−
=
1
1
1 i
kkijkji
iiji ula
ul
…..(IV)
Once again we note the RHS involves uii, which has been determined using (III); aij which is from the given matrix; ljk; 1 ≤ k ≤ i-1 and hence only entries in the first i-1 columns of L; and uki, 1 ≤ k ≤ i-1 and hence only entries in the first i-1 rows of U. Thus RHS in (IV) is completely known and hence lji, the entries in the ith column of L are completely determined by (IV).
Numerical Analysis/Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L6/V1/May 2004/4
Summarizing, Doolittle’s procedure is as follows:
lii = 1; 1st row U = 1st row of A ; Step 1 determining 1st row of U and
lj1 = aj1/u11 1st column of L.
For i ≥ 2; we determine
∑−
=
−=1
1
i
kkjikijij ulau
; j = i, i+1, i+2, …….,n
(Note for j<i we have uij = 0)
⎥⎦
⎤⎢⎣
⎡−= ∑
−
=
1
1
1 i
kkijkji
iiji ula
ul
; j = i, i+1, i+2,…..,n
(Note for j<i we have ljj = 0)
We observe that the method fails if uii = 0 for some i.
Example:
Let
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
−
−−−
=
126064191444622
3112
A
Let us determine the Doolittle decomposition for this matrix.
First step:
Numerical Analysis/Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L6/V1/May 2004/5
1st row of U : Same as 1st row of A.
∴u11 = 2 ; u12 = 1 ; u13 = -1 ; u14 = 3
1st column of L:
l11 = 1;
l21 = a21/u11 = -2/2 = -1.
l31 = a31/u11 = 4/2 = 2.
l41 = a41/u11 = 6/2 = 3.
Second step:
2nd row of U : u12 = 0 (Because upper triangular)
u22 = a22 – l21 u12 = 2 – (-1) (1) = 3.
u23 = a23 – l21 u13 = 6 – (-1) (-1) = 5.
u24 = a24 – l21 u14 = - 4 – (-1) (3) = -1.
2nd column of L : l12 = 0 (Because lower triangular)
l22 = 1.
l32 = (a32 – l31 u12) /u22
= [14 – (2)(1)]/3 = 4.
L42 = (a42 – l41 u12) /u22
Numerical Analysis/Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L6/V1/May 2004/6
= [0 – (3)(1)]/3 = -1.
Third Step:
3rd row of U: u31 = 0 Because upper triangular
u32 = 0
u33 = a33 – l31 u13 – l32 u23
= 19 – (2) (-1) – (4)(5) = 1.
u34 = a34 – l31 u14 – l32 u24
= 4 – (2) (3) – (4)(-1) = 2.
3rd column of L : l13 = 0 Because lower triangular
l23 = 0
l33 =1
l43 = (a43 – l41 u13 – l42 u23)/ u33
= [-6 – (3) (-1) – (-1) (5)]/1
= 2.
Fourth Step:
4th row of U: u41 = 0
u42 = 0 Because upper triangular
u43 = 0
u44 = a44 – l41 u14 – l42 u24 – l43 u34
= 12 – (3) (3) – (-1) (-1) – (2) (2)
= -2.
Numerical Analysis/Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L6/V1/May 2004/7
4th column of L : l14 = 0 = l24 = l34 Because lower triangular
l44 = 1.
Thus.
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
−
−=
1213014200110001
L ; . . . . . . . . . . .(V)
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
−
−−
=
200021001530
3112
U
and
A = LU.
This gives us the LU decomposition by Doolittle’s method for the given A.
As we observed in the case of the LU decomposition of a tridiagonal matrix; it is not advisable to choose the lii as 1; but to choose in such a way that the diagonal entries of L and the corresponding diagonal entries of U are of the same magnitude. We describe this procedure as follows:
Once again 1st row and 1st column of U & L respectively is our first concern:
Step 1: a11 = l11 u11
Choose ( ) 1111111111 .sgn; aaual ==
Next 1011111
1 >=== ∑=
forkaslulula kj
n
kkjkij
Numerical Analysis/Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L6/V1/May 2004/8
11
1l
au jij⇒
Thus note that u1j have been scaled now as compared to what we did earlier.
Similarly,
11
11 u
al jj
These determine the first row of U and first column of L. Suppose we have determined the first I-1 rows of U and first I-1 columns of L. We determine now the ith row of U and ith column of L as follows:
∑=
=n
kkiikii ula
1
for l∑=
=n
kkiik ul
1ik = 0 if k>i
ii
i
kiikiik ulul∑
−
=
+=1
1
saypulaul i
i
kkiikiiiiii ,
1
1=−=∴ ∑
−
=
Choose ki
i
kikiiiii ulapl ∑
−
=
−==1
1
Numerical Analysis/Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L6/V1/May 2004/9
iiii ppu sgn−=
iforklulula ikkj
i
kikkj
n
kikij >=== ∑∑
=−
011
Q
ijiikj
i
kik ulul += ∑
−
=
1
1
⎟⎠
⎞⎜⎝
⎛−=⇒ ∑
−
=kj
i
kikijij ulau
1
1 lii
determining the ith row of U.
ki
n
kjkji ula ∑
=
=1
iifkuul kiki
i
kjk >== ∑
=
01
Q
iijiki
i
kjk ulul += ∑
−
=
1
1
⎟⎠
⎞⎜⎝
⎛−=⇒ ∑
−
=
1
1
i
kkijkjiji ulal uii ,
thus determining the ith column of L.
Let us now apply this to matrix A in the example in page 30.
Numerical Analysis/Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L6/V1/May 2004/10
First Step:
2;22 1111111111 ==∴== ulaul
23;
21
21
11
1414
11
1313
11
1212 ==−====
lau
la
ulau
23;
21;
21;2 14131211 =−=== uuuu
22
2
11
2121 −=−==
ua
l
222
4
11
3131 ===
ual
232
6
11
4141 ===
ual
therefore
23
22
2
2
41
31
21
11
=
=
−=
=
l
l
l
l
Second Step:
1221222222 ulaul −=
Numerical Analysis/Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L6/V1/May 2004/11
( ) 32
122 =⎟⎠
⎞⎜⎝
⎛−−=
3;3 2222 ==∴ ul
( )13212323 ulau −= l22
( )3
53/2
126 =⎥⎦
⎤⎢⎣
⎡⎟⎠
⎞⎜⎝
⎛−−−=
[ ]14212424 ulau −= l22
( ) ( )3
13/2
324 −=⎥⎦
⎤⎢⎣
⎡⎟⎠
⎞⎜⎝
⎛−−−=
therefore
31;
35;3;0 24232221 −====∴ uuuu
( ) 2212313232 /uulal −=
( ) 3/2
12214 ⎥⎦
⎤⎢⎣
⎡⎟⎠
⎞⎜⎝
⎛−=
34=
Numerical Analysis/Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L6/V1/May 2004/12
( ) 2212414242 /uulal −=
( ) 3/2
1230 ⎟⎟⎠
⎞⎜⎜⎝
⎛⎟⎠
⎞⎜⎝
⎛−=
3−=
therefore
3
34
3
0
42
32
22
12
=
=
=
=
l
l
l
l
Third Step:
23321331333333 ululaul −−=
( ) ( ) ⎟⎟⎠
⎞⎜⎜⎝
⎛−⎟
⎠
⎞⎜⎝
⎛−−=
3534
212219
= 1
1;1 3333 ==∴ ul
( ) 33243214313434 / lululau −−=
Numerical Analysis/Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L6/V1/May 2004/13
( ) ( ) 1/3
1342
3224⎥⎥⎦
⎤
⎢⎢⎣
⎡⎟⎟⎠
⎞⎜⎜⎝
⎛⎟⎟⎠
⎞⎜⎜⎝
⎛−−⎟
⎠
⎞⎜⎝
⎛−=
= 2
2,1;0;0 34333231 ====∴ uuuu
[ ] 33234213414343 / uululal −−=
( ) ( ) 1/3
532
1236 ⎟⎟⎠
⎞⎜⎜⎝
⎛⎟⎟⎠
⎞⎜⎜⎝
⎛−−⎟
⎠
⎞⎜⎝
⎛−−−=
= 2
therefore
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
====
2100
43
33
23
13
llll
Fourth Step:
344324421441444444 ulululaul −−−=
( ) ( ) ( )( )223
132
32312 −⎟⎟⎠
⎞⎜⎜⎝
⎛−−−⎟
⎠
⎞⎜⎝
⎛−=
= -2
Numerical Analysis/Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L6/V1/May 2004/14
2;2 4444 −==∴ ul
2;0;0;0 44434241 −====∴ uuuu
⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢
⎣
⎡
=
===
2
000
44
34
24
14
l
lll
Thus we get the LU decompositions,
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
−
−=
2232301342200320002
L ;
⎟⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜⎜
⎝
⎛
−
−
−
=
20002100
31
3530
23
21
212
U
in which iiii ul = , i.e. the corresponding diagonal entries of L and U have the same magnitude.
Note: Compare this with the L and U of page 32. What is the difference.
The U in page 36 can be obtained from the U of page 32 by
(1) replacing the ‘numbers’ in the diagonal of that U and keeping the same sign. Thus the first diagonal 2 is replaced by 2 ; 2nd diagonal 3 is replaced by 3 , third diagonal1 by 1 and 4th diagonal –2 by - 2 . These then give the diagonals of the U in page 36.
(2) Divide each entry to the right of a diagonal in the U of page 32 by these replaced diagonals.
Numerical Analysis/Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L6/V1/May 2004/15
Thus 1st row changes to 1st row of U in page 36
2nd row changes to 2nd row of U in page 36
3rd row changes to 3rd row of U in page 36
4th row changes to 4th row of U in page 36
This gives the U of page 36 from that of page 32.
The L in page 36 can be obtained from the L of page 32 as follows:
(1) Replace the diagonals in L by magnitude of the diagonals in U of page 36.
(2) Multiply each entry below the diagonal of L by this new diagonal entry.
We get the L of page 32 changing to the L of page 36.
Numerical Analysis/ Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L7/V1/May 2004/1
DOOLITTLE’S METHOD WITH ROW INTERCHANGES
We have seen that Doolittle factorization of a matrix A may fail the moment at stage i we encounter a uii which is zero. This occurrence corresponds to the occurrence of zero pivot at the ith stage of simple Gaussian elimination method. Just as we avoided this problem in the Gaussian elimination method by introducing partial pivoting we can adopt this procedure in the modified Doolittle’s procedure. The Doolittle’s method which is used to factorize A as LU is used from the point of view of reducing the system
Ax = y
To two triangular systems
Lz = y
Ux = z
as already mentioned in page 17.
Thus instead of actually looking for a factorization A = LU we shall be looking for a system,
A*x = y*
and for which A* has LU decomposition.
We illustrate this by the following example: The basic idea is at each stage calculate all the uii that one can get by the permutation of rows of the matrix and choose that matrix which gives the max. absolute value for uii.
As an example consider the system
Ax = y
where
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
−−−
−−
=
32131451
32221213
A y =
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
−
−
138
3
We want LU decomposition for some matrix that is obtained from A by row interchanges.
We keep lii = 1.
Stage 1:
1st diagonal of U. By Doolittle decomposition,
u11 = a11 = 3
Numerical Analysis/ Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L7/V1/May 2004/2
If we interchange 2nd or 3rd or 4th rows with 1st row and then find the u11 for the new matrix we get respectively u11 = 2 or 1 or 3. Thus interchange of rows does not give any advantage at this stage as we have already got 3 without row interchange for u11.
So we keep the matrix as it is and calculate 1st row of U, by Doolittle’s method.
.133;
31;3
2;111
4141
11
3131
11
212111 ========
ual
ua
luall
Thus
L is of the form
⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜
⎝
⎛
***1
01*31
00132
0001
; and
U is of the form ; A and Y remaining unchanged.
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛ −−
*000**00***01213
Stage 2
We now calculate the second diagonal of U: By Doolittle’s method we have
12212222 ulau −= ( )381
322 −=⎟
⎠⎞
⎜⎝⎛−−=
Suppose we interchange 2nd row with 3rd row of A and calculate u22 : our new a22 is 5.
But note that the L gets in the 1st column 2nd and 3rd row interchanged. Therefore new l21 is1/3.
Suppose instead of above we interchange 2nd row with 4th row of A:
New a22 = 1 and new l21 = 1 and therefore new u22 = 1 – (1) (1) = 0
Of these 14/3 has largest absolute value. So we prefer this. Therefore we interchange 2nd and 3rd row.
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
−−
=
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
−−−−−
=
18
33
;
3213322214511213
NewyNewA
Numerical Analysis/ Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L7/V1/May 2004/3
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛ −−
=
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
=
*000**00
**3
1401213
;
1**101*3
20013
10001
NewUNewL
Now we do the Doolittle calculation for this new matrix to get 2nd row of U and 2nd column of L.
13212323 ulau −= ( ) ( )3
102314 −=−⎟
⎠⎞
⎜⎝⎛−−=
14212424 ulau −= ( ) ( )321
311 −=−⎟
⎠⎞
⎜⎝⎛−−=
2nd column of L:
[ ] 2212313232 uulal ÷−= ( ) ( )3
141322 ÷⎥
⎦
⎤⎢⎣
⎡⎟⎠⎞
⎜⎝⎛−−=
74
−=
[ ] 1112414242 uulal ÷−= ( )( )[ ]3
14113 ÷−= = 0
Therefore new L has form
⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜
⎝
⎛
−
1*01
0174
32
00131
0001
New U has form
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛−−−−
*000**0032
310
3140
1213
This completes the 2nd stage of our computation.
Note: We had three choices of u22 to be calculated, namely –8/3, 14/3, 0 before we chose
14/3. It appears that we are doing more work than Doolittle. But this is not really so. For, observe, that the rejected u22 namely – 8/3 and 0 when divided by the chosen u22 namely 14/3 give the entries of L below the second diagonal.
Numerical Analysis/ Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L7/V1/May 2004/4
3rd Stage:
3rd diagonal of U:
233213313333 ululau −−= ( ) ⎟⎠⎞
⎜⎝⎛−⎟
⎠⎞
⎜⎝⎛−−−⎟
⎠⎞
⎜⎝⎛−=
310
742
322
710
=
Suppose we interchange 3rd row and 4th row of new A obtained in 2nd stage. We get new a33 = 2.
But in L also the second column gets 3rd and 4th row interchanges
Therefore new l31 = 1 and new l32 = 0
Therefore new u33 = a33 – l31 u13 – l32 u23 ( )( ) ( ) ⎟⎠⎞
⎜⎝⎛−+−−=
3100212 = 4.
Of these two choices of u33 we have 4 has the larges magnitude. So we interchange 3rd and 4th rows of the matrix of 2nd stage to get
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
−−
=
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
−
−−−−
=
81
33
3222321314511213
NewYNewA
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛−−
−−
=
⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜
⎝
⎛
−
=
*000*40032
310
3140
1213
;
1*74
32
0101
00131
0001
NewUNewL
Now for this set up we calculate the 3rd stage entries as in Doolittle’s method:
243214313434 ululau −−= ( )( ) ( ) 4320113 =⎟
⎠⎞
⎜⎝⎛ −−−−=
( ) 33234213414343 uululal ÷−−=
Numerical Analysis/ Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L7/V1/May 2004/5
( ) 43
10742
322 ÷⎥
⎦
⎤⎢⎣
⎡⎟⎠⎞
⎜⎝⎛−⎟
⎠⎞
⎜⎝⎛−−−⎟
⎠⎞
⎜⎝⎛−= = 5/14.
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛−−
−−
=
⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜
⎝
⎛
−
=∴
*000440032
310
3140
1213
;
1145
74
32
0101
00131
0001
NewUNewL
Note: The rejected u33 divided by chosen u33 gives l43.
4th Stage
[ ]3443244214414444 ulululau −−−=
( ) ( )4145
32
741
323 ⎟
⎠⎞
⎜⎝⎛−⎟
⎠⎞
⎜⎝⎛−⎟
⎠⎞
⎜⎝⎛−−−⎟
⎠⎞
⎜⎝⎛−= = 13/7.
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
−−
==
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
−
−−−−
==∴
81
33
3222321314511213
** YNewYANewA
New L = L* , New U = U*
,
713000440032
310
3140
1213
;
1145
74
32
0101
00131
0001
**
⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜
⎝
⎛−−
−−
=
⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜
⎝
⎛
−
= UL
and A* = L*U*
The given system Ax=y is equivalent to the system
A*x=y*
and hence can be split into the triangular systems,
L*z = y*
U*x = z
Now L*z = y* gives by forward substitution:
Z1 =3
Numerical Analysis/ Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L7/V1/May 2004/6
213331
221 =−=⇒=+ zzz
411 1331 −=−−=⇒−=+ zzzz
8145
74
32
4321 −=++− zzzz
( ) ( ) ( ) 841452
743
32
4 −=+−⎟⎠⎞
⎜⎝⎛+⎟
⎠⎞
⎜⎝⎛−⎟
⎠⎞
⎜⎝⎛ z
752
4 −=⇒ z
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
−
−=∴
7524
23
z
Therefore U*x = z gives by back-substitution;
752
713
4 −=x therefore x4 = -4.
311444 434343 =−−=⇒−=+⇒−=+ xxxxxx
therefore x3 = 3
232
310
314
432 =−− xxx
( ) ( ) 24323
310
314
2 =−⎟⎠⎞
⎜⎝⎛ −⎟
⎠⎞
⎜⎝⎛−x
22 =⇒ x
323 4321 =−−+ xxxx
134623 11 =⇒−+−+⇒ xx
Therefore the solution of the given system is
Numerical Analysis/ Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L7/V1/May 2004/7
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
−
=
4321
x
Some Remarks:
The factorization of a matrix A as the product of lower and upper triangular matrices is by no means unique. In fact, the diagonal elements of one or the other factor can be chosen arbitrarily; all the remaining elements of the upper and lower triangular matrices may then be uniquely determined as in Doolittle’s method; which is the case when we choose all the diagonal entries of L as 1. The name of Crout is often associated with triangular decomposition methods, and in crout’s method the diagonal elements of U are all chosen as unity. Apart from this, there is little distinction, as regards procedure or accuracy, between the two methods.
As already mentioned, Wilkinson’s suggestion is to get a LU decomposition in which niul iiii ≤≤= 1; .
We finally look at the cholesky decomposition for a symmetric matrix:
Let A be a symmetric matrix.
Let A = LU be a LU decomposition
Then A1 = U1 L1 U1 is also lower triangular
L1 is upper triangular
Therefore U1L1 is a decomposition of A1 as product of lower and upper triangular matrices. But A1 = A since A is symmetric.
Therefore LU = U1L1
We ask the question whether we can choose L as U1; so that
A = U1U (or same as LL1)
Now therefore determining U automatically gets L = U1
We now do the Doolittle method for this. Note that it is enough to determine the rows of U.
Stage 1: 1st row of U:
Numerical Analysis/ Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L7/V1/May 2004/8
∑∑==
==n
kkk
n
kk uula
11
21
1111 11 kk ul =Q 1UL =Q
for k>1 since U is upper triangular 112u= 01 =kuQ
1111 au =∴
We finally look at the cholesky decomposition for a symmetric matrix:
Let A be a symmetric matrix.
Let A = LU be a LU decomposition
Then A1 = U1 L1 U1 is also lower triangular
L1 is upper triangular
Therefore U1L1 is a decomposition of A1 as product of lower and upper triangular matrices. But A1 = A since A is symmetric.
Therefore LU = U1L1
We ask the question whether we can choose L as U1; so that
A = U1U (or same as LL1)
Now therefore determining U automatically gets L = U1
We now do the Doolittle method for this. Note that it is enough to determine the rows of U.
Stage 1: 1st row of U:
∑∑==
==n
kkk
n
kk uula
11
21
1111 11 kk ul =Q 1UL =Q
112u= for k>1 since U is upper triangular. 01 =kuQ
1111 au =∴
∑ ∑= =
==n
k
n
kkikkiki uuula
1 1111
iuu 111= 101 >= forkukQ
1111 au =
1111 / uau ii =∴ determines first row of U. and hence first column of L.
Numerical Analysis/ Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L7/V1/May 2004/9
Having determined the 1st i-1 rows of U; we determine the ith row of U as follows:
∑ ∑= =
===n
k
n
kkiikkikiikii uluula
1 1
2 Q
for k > i 01
2 == ∑=
ki
i
kki uu Q
ii
i
kki uu 2
1
1
2 += ∑−
=
∑−
=
−=∴1
1
22i
kkiiiii uau
∑−
=
−=∴1
1
2i
kkiiiii uau ; Note: uki are known for k ≤ i -1,
1st i-1 rows have already been obtained.
∑ ∑= =
==n
k
n
kkjkikjikij uuula
1 1 Now we need uij for j > i
∑=
=i
kkjkiuu
1 Because uki = 0 for k > i
∑−
=
+=1
1
i
kijiikjki uuuu
Therefore
ii
i
kkjkiijij uuuau ÷⎥
⎦
⎤⎢⎣
⎡−= ∑
−
=
1
1
Numerical Analysis/ Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L7/V1/May 2004/10
⎪⎪
⎩
⎪⎪
⎨
⎧
÷⎥⎦
⎤⎢⎣
⎡−=
−=∴
∑
∑−
=
−
=
ij
i
kkjkiijij
i
kkiiiii
uuuau
uau
1
1
1
1
2
determines the ith row of U in terms of the previous rows. Thus we get U and L is U1. This is called CHOLESKY decomposition.
Example:
Let
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
−−−
−
=
10131133133511111
A
This is a symmetric matrix. Let us find the Cholesky decomposition.
1st row of U
⎪⎪
⎩
⎪⎪
⎨
⎧
=÷==÷=
−=÷=
==
11
11
111414
111313
111212
1111
uauuauuau
au
2nd row of U
Numerical Analysis/ Direct methods for solving linear system of equation Lecture notes
VittalRao/IISc, Bangalore M1/L7/V1/May 2004/11
( ) ( )( )( )( ) ( )( )( )⎪
⎪⎩
⎪⎪⎨
⎧
=÷−−=÷−=−=÷−−−=÷−=
=−=−=
2211312113
215
2214122424
2213122323
122
2222
uuuauuuuau
uau
3rd row of U
( ) ( )( ) ( )( )( )⎪⎩
⎪⎨⎧
=÷−−−=÷−−=
=−−=−−=
21211111113
33242314133434
232
132
3333
uuuuuauuuau
4th row of U
144110342
242
142
4444 =−−−=−−−= uuuau
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛−
−
=∴
1000210021201111
U
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
−−
==∴
1221011100210001
1 LU and
A = LU
= LL1
= U1U
Numerical Analysis / Iterative methods for solving linear systems of equations Lecture notes
ITERATIVE METHODS FOR THE SOLUTION OF SYSTEMS EQUATION
In general an iterative scheme is as follows:
We have an nxn matrix M and we want to get the solution of the systems
x = Mx + y ……………………..(1)
We obtain the solution x as the limit of a sequence of vectors, { }kx which are obtained as follows:
We start with any initial vector x(0), and calculate x(k) from,
x(k) = Mx(k-1) + y ……………….(2)
for k = 1,2,3, ….. successively.
A necessary and sufficient condition for the sequence of vectors x(k) to converge to
solution x of (1) is that the spectral radius spM
of the iterating matrix M is less than 1 or
if 1<M for some matrix norm.
We shall now consider some iterative schemes for solving systems of linear equations,
Ax = y …………….(3)
We write this system in detail as
11212111 ..... yxaxaxa nn =+++
22222121 ..... yxaxaxa nn =+++ . . . . . . . .(4) ...... ...... ......
nnnnnn yxaxaxa =+++ .....2211
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
=
nnnn
n
n
aaa
aaaaaa
WehaveA
K
KKKK
K
K
21
22221
11211
. . . . . . . . . . . (5)
We denote by D, L, U the matrices
VittalRao/IISc, Bangalore M2/L1/V1/May 2004/1
Numerical Analysis / Iterative methods for solving linear systems of equations Lecture notes
)6......(....................
......00...............0...000......00......0
33
22
11
⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜
⎝
⎛
=
nna
aa
a
D
the diagonal part of A; and
)7....(..............................
0
0000000
121
3231
21
⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜
⎝
⎛
=
−nnn aaa
aaa
L
K
KKKKK
K
KK
KK
the lower triangular part of A; and
)8(........................................
0...000...............
...00
......0
223
112
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
= n
n
uuuu
U
the upper triangular part of A.
Note that,
A = D + L + U ……………………… (9).
We assume that ; i = 1, 2, ……, n …………(10) 0≠iia
So that D-1 exists.
We now describe two important iterative schemes, below, for solving the system (3).
VittalRao/IISc, Bangalore M2/L1/V1/May 2004/2
Numerical Analysis/Iterative methods for solving linear system of equation Lecture notes
JACOBI ITERATION
We write the system as in (4) as
11313212111 ..... yxaxaxaxa nn +−−−=
22323121222 ..... yxaxaxaxa nn +−−−= . . . . . . . .(11)
VittalRao/IISc, Bangalore M2/L2/V1/May 2004/1
...... ...... ......
nnnnnnnnn yxaxaxaxa +−−−= −− 112211 .....
We start with an initial vector,
( )
( )
( )
( )
)12........(....................
0
20
10
0
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
=
nx
xx
xM
and substitute this vector for x on the RHS of (11) and calculate x1,x2, ….., xn and this vector is called x(1). We now substitute this vector on the RHS of (11) to calculate again x1, x2, ….., xn and call this new vector as x(2) and continue this procedure to calculate the sequence x(k). We can describe this briefly as follows:
The equation (11) can be written as,
Dx = - (L + U) x + y …………………. (13)
which we can write as
x = -D-1 (L+U) x +D-1 y,
giving
yJxx ˆ+= ……………… (14)
where
J = -D-1 (L + U) …………….(15)
and, we get
x(0) starting vector
,.......2,1;ˆ)1()( =+= − kyJxx kk …………….(16)
as the iterating scheme. This is similar to (2) with the iterating matrix M as
J = -D-1 (L + U); J is called the Jacobi Iteration Matrix. The scheme will converge to the solution x of our system if 1<
spJ . We shall see an easier condition below:
Numerical Analysis/Iterative methods for solving linear system of equation Lecture notes
We have
1/a11
D-1 = 1/a22
1/ann
and therefore
( )
⎟⎟⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜⎜⎜
⎝
⎛
−−−
−−−
−−−
=+−=
−
−
0........................
....0
....0
121
22
2
22
23
22
21
11
1
11
13
11
12
1
nn
nn
nn
n
nn
n
n
n
aa
aa
aa
aa
aa
aa
aa
aa
aa
ULDJ
Now therefore the ith Absolute row sum for J is
( )∑≠
+− ++++++==ij
iiiniiiiiiii
iji aaaaaa
aa
R /........ 1121
∴ If Ri <1 for i =1,2,3,…..,n
then
{ } 1,.....,max 1 <=∞ nRRJ
and we have convergence.
Now Ri < 1 means
iiiniiiiii aaaaaa <++++++ +− .......... 1121
i.e. in each row of A the sum of the absolute x values of the nondiagonal entries is dominated by the absolute value of the diagonal entry. i.e. A is ‘strictly row diagonally dominant’. Thus the Jacobi iteration scheme for the system (3) converges if A is strictly row diagonally dominant (Of course this condition may not be satisfied) and still Jacobi iteration scheme may converge if .1<
spJ
VittalRao/IISc, Bangalore M2/L2/V1/May 2004/2
Numerical Analysis/Iterative methods for solving linear system of equation Lecture notes
Example:
Consider the system
x1 + 2x2 – 2x3 = 1
x1 + x2 + x3 = 0 ………….(I)
2x1 + 2x2 + x3 = 0
Let us apply the Jacobi iteration scheme with the initial vector as
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛==
000
)0( θx ………….(II)
We ; ⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛ −=
122111221
A⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛=
100010001
D
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛ −=+
022101220
UL ; ⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛=
001
y
( )⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
−−−−+−
=+−= −
022101220
1 ULDJ ; ⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛== −
001
ˆ 1 yDy
Thus the Jacobi scheme (16) becomes
( )
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛=
000
0x
( ) ( ) ,......2,1,ˆ1 =+= − kyJxx kk
( ) ( )
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛==+=+=∴
001
ˆˆˆ01 yyJyJxx θ
VittalRao/IISc, Bangalore M2/L2/V1/May 2004/3
Numerical Analysis/Iterative methods for solving linear system of equation Lecture notes
( ) ( )
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛+
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
−−−−+−
=+=001
001
022101220
ˆ12 yJxx
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
−−=
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛+
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
−−=
21
1
001
21
0
( ) ( )
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛+
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
−−
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
−−−−
−=+=
001
21
1
022101
220ˆ23 yJxx
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−=
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛+
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−=
011
001
012
( ) ( )
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛+
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
−−−−
−=+=
001
011
022101
220ˆ34 yJxx
( )3
011
001
012
x=⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛ −=
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛+
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛ −=
∴ x(4) = x(5) = x(6) = ………. = x(3)
∴ x(k) = x(3) and x(k) converges to x(3)
∴ The solution is =x∞→k
lim ( ) ( )
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−==
011
3xx k
Can easily check that this is the exact solution.
Here, there is no convergence problem at all.
VittalRao/IISc, Bangalore M2/L2/V1/May 2004/4
Numerical Analysis/Iterative methods for solving linear system of equation Lecture notes
Example 2:
8x1 + 2x2 – 2x3 = 8
x1 - 8x2 + 3x3 = 19
2x1 + x2 + 9x3 = 30
Let us apply Jacobi iteration scheme starting with ( )
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛=
000
0x
We have ⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−=
900080008
D
⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜
⎝
⎛
−=∴ −
9100
0810
0081
1D
( )⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
−−+
+−=+−= −
011111.022222.0375.00125.0
25.025.001 ULDJ
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−== −
33333.3375.2
1ˆ 1 yDy
Now the matrix is such that
4228 131211 =+=+= aaanda 131211 aaa +>∴
;4318 232122 =+=+= aaanda 232122 aaa +>∴
3129 323133 =+=+= aaanda 323133 aaa +>∴
Thus we have strict row diagonally dominant matrix A. Hence the Jacobi iteration scheme will converge. The scheme is,
VittalRao/IISc, Bangalore M2/L2/V1/May 2004/5
Numerical Analysis/Iterative methods for solving linear system of equation Lecture notes
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛=
000
0x
yJxx kk ˆ)1()( += − yx k ˆ011111.022222.0375.00125.025.025.00
)1( +⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
−−
−= −
( )
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−==
33333.3375.2
1ˆ1 yx
We continue the iteration until the components of x(k) and x(k+1) differ by at most, say; 3x10-5 = ( ) ( ) 51 103.. −
∞
+ ≤−∈ xxxei kk we get ( ) ( ) .33333.301 =−∞
xx So we
continue
( ) ( )
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−=+=
37500.300000.1
42708.2ˆ12 yJxx ( ) ( ) ≥∈=−
∞42708.112 xx
( ) ( ) ;90509.280599.0
09375.2ˆ23
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−=+= yJxx ( ) ( ) ≥∈=−
∞46991.023 xx
( ) ( ) ;95761.202387.1
92777.1ˆ34
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−=+= yJxx ( ) ( ) ≥∈=−
∞21788.034 xx
( ) ( ) ;01870.302492.1
99537.1ˆ45
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−=+= yJxx ( ) ( ) ≥∈=−
∞06760.045 xx
( ) ( ) ;00380.399356.0
01091.2ˆ56
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−=+= yJxx ( ) ( ) ≥∈=−
∞03136.056 xx
VittalRao/IISc, Bangalore M2/L2/V1/May 2004/6
Numerical Analysis/Iterative methods for solving linear system of equation Lecture notes
( ) ( ) ;99686.299721.0
99934.1ˆ67
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−=+= yJxx ( ) ( ) ≥∈=−
∞01157.067 xx
( ) ( ) ;99984.200126.1
99852.1ˆ78
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−=+= yJxx ( ) ( ) ≥∈=−
∞00405.078 xx
( ) ( ) ;00047.300025.1
00027.2ˆ89
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−=+= yJxx ( ) ( ) ≥∈=−
∞00176.089 xx
( ) ( ) ;99997.299979.0
00018.2ˆ910
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−=+= yJxx ( ) ( ) ≥∈=−
∞00050.0910 xx
( ) ( ) ;99994.299999.0
99994.1ˆ1011
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−=+= yJxx ( ) ( ) ≥∈=−
∞00024.01011 xx
( ) ( ) ;00001.300003.1
99998.1ˆ1112
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−=+= yJxx ( ) ( ) ≥∈=−
∞00008.01112 xx
( ) ( ) ;00001.300000.1
00001.2ˆ1213
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−=+= yJxx ( ) ( ) =∈=−
∞00003.01213 xx
∴ SOLUTION IS 2 = x1 ; -1 = x2, 3.00001 = x3 (Exact solution is x1 = 1, x2 = -2, x3 =3).
VittalRao/IISc, Bangalore M2/L2/V1/May 2004/7
Numerical Analysis/ Iterative methods for solving linear system of equation Lecture notes
Gauss – Seidel Method
Once again we consider the system
Ax = y …………….. (I)
In the Jacobi scheme we used the values of x(k)2, x(k)
3, ….., x(k)n obtained in the k the
iteration, in place of x2, x3, ….., xn in the first equation,
11212111 ..... yxaxaxa nn =+++
to calculate x(k+1)1 from
( ) ( ) ( ) ( )113132121
111 ..... yxaxaxaxa n
kn
kkk +−−−=+
Similarly, in the ith equation we used the values, x(k)1, x(k)
2, …., x(k)i-1, x(k)
i+1, …., x(k)n, in
place of x1, x2, ….., xi-1, xi+1, ….., xn to calculate x(k+1)i from
( ) ( ) ( ) ( )112211
1 ...... −−+ −−−−= i
kii
ki
kii
kii xaxaxaxa
( ) ( ) ......11 ink
inik
ii yxaxa +−−− ++
What Gauss – Seidel suggest is that having obtained x(k+1)1from the first equation use this
value for x1 in the second equation to calculate x(k+1)2 from
( ) ( ) ( ) ( )223231
1212
122 ...... yxaxaxaxa n
kn
kkk +−−−−= ++
and use these values of x(k+1)1, x(k+1)
2, in the 3rd equation to calculate x(k+1)3, and so on.
Thus in the equation use x(k+1)1, ….., x(k+1)
i-1 to calculate x(k+1)i from
( ) ( ) ( ) ( )1
112
121
11
1 ...... −+
−+++ −−−−= i
kii
ki
kii
kii xaxaxaxa
( ) ( ) ( )
ink
inik
iiik
ii yxaxaxa +−−− ++++ .....2211
In matrix notation we can write this as, ( ) ( ) ( ) yUxLxDx kkk +−−= ++ 11
which can be rewritten as,
( ) ( ) ( ) yUxxLD kk +−=+ +1 , and hence
Thus we get the Gauss – Seidel iteration scheme as,
( ) ( ) ( ) yLDUxLDx kk 111 −−+ +++−=
x(0) initial guess ( ) ( ) yGxx kk ˆ1 +=+ ……..(II)
VittalRao/IISc, Bangalore M2/L3/V1/May 2004/1
Numerical Analysis/ Iterative methods for solving linear system of equation Lecture notes
where,
G = -(D+L)-1U
is the Gauss – Seidel iteration matrix, and
( ) yLDy 1ˆ −+=
The scheme converges if and only if .1<sp
G Of course, the scheme will converge if
1<G in some matrix norm. But some matrix norm, 1≥G does not mean that the
scheme will diverge. The acid test for convergence is .1<sp
G
We shall now consider some examples.
Example 3:
Let us consider the system
x1 + 2x2 – 2x3 = 1
x1 + x2 + x3 = 0
2x1 + 2x2 + x3 = 0
considered on page 5; and for which the Jacobi scheme gave the exact solution in the 3rd iteration. (see page 6). We shall now try to apply the Gauss – Seidel scheme for this system. We have,
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛ −=
122111221
A ; ⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛=
001
y
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛=+
122011001
LD ; ⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−
−=−
000100
220u
( )⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
−−=+ −
120011001
1LD
Thus,
VittalRao/IISc, Bangalore M2/L3/V1/May 2004/2
Numerical Analysis/ Iterative methods for solving linear system of equation Lecture notes
( )⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−
−=
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−
−
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
−−=+−= −
200320
220
000100
220
120011001
1uLDG
Thus, Gauss – Seidel iteration matrix is,
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−
−=
200320
220G
Since G is triangular we get its eigenvalues immediately, as its diagonal entries. Thus λ1 = 0, λ2 = 2, λ3 = 2 are the three eigenvalues. Therefore,
12 >=sp
G
Hence the Gauss – Seidel scheme for this system will not converge. Thus for this system the Jacobi scheme converges so rapidly giving the exact solution in the third iteration itself whereas the Gauss – Seidel scheme does not converge.
Example 4:
Consider the system
021
21
0
121
21
321
321
321
=+−−
=++
=−−
xxx
xxx
xxx
Let us apply the Gauss – Seidel scheme to this system. We have,
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
−−
−−
=
121
21
11121
211
A ; ⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛=
001
y
⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜
⎝
⎛
−−
=+
121
21
011001
LD ; ( )⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜
⎝
⎛
−=+ −
1210
011001
1LD ,
VittalRao/IISc, Bangalore M2/L3/V1/May 2004/3
Numerical Analysis/ Iterative methods for solving linear system of equation Lecture notes
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
−=−000100
21
210
u .
Thus,
( )⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
−⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜
⎝
⎛
−=+−= −
000100
21
210
1210
011001
1 uLDG
..(*)..........
210023
210
21
210
⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜
⎝
⎛
−
−−=∴G
is the Gauss – Seidel matrix for this sytem.
The Gauss – Seidel scheme is ( ) ( )
( )
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛=
+=+
000
ˆ
0
1
x
yGxx kk
where
( ) andyLDy ;01
1
001
1210
011001
ˆ 1
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−=
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜
⎝
⎛
−=+= −
where G is given (*).
VittalRao/IISc, Bangalore M2/L3/V1/May 2004/4
Numerical Analysis/ Iterative methods for solving linear system of equation Lecture notes
Notice that G is upper triangular and hence we readily get the eigenvalues of G as its diagonal entries. Thus the eigenvalues of G are, λ1 = 0, λ2 = -1/2, λ3 = -1/2. Hence
121<=
spG . Hence the Gauss – Seidel scheme will converge.
Let us now carry out a few steps of the Gauss – Seidel iteration, since we have now been assured of convergence. (We shall first do some exact calculations).
( ) ( )
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−=
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−+
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛=+=
01
1
01
1
000
ˆ01 GyGxx
( ) ( )
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−+
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−=+=
01
1
01
1ˆ12 GyGxx
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−+
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−
⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜
⎝
⎛
−
−−=01
1
01
1
210023
210
21
210
⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜
⎝
⎛
⎟⎠⎞
⎜⎝⎛ −−
−
=
0211
211
( ) ( ) ( )⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜
⎝
⎛
+−−
+−
=+=
02
12
112
12
11
ˆ 2
2
23 yGxx
If we continue this process we get
VittalRao/IISc, Bangalore M2/L3/V1/May 2004/5
Numerical Analysis/ Iterative methods for solving linear system of equation Lecture notes
( )
( )
( )
⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜
⎝
⎛
⎟⎠⎞
⎜⎝⎛ −+−+−−
−+−+−
= −
−
−
−
02
1.....21
211
21.....2
12
11
1
1
2
1
1
2
k
k
k
k
kx
Clearly,
( ) ( )⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜
⎝
⎛
−+−−
+−+−
→
0.....2
12
12
11
.....21
21
21
211
32
432
kx
and by summing up the geometric series we get,
( )
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛→
03232
kx
which is the exact solution.
Of course, here ‘we’ knew ‘a priori’ that the sequence is going to sum up neatly for each component and so we did exact calculation. If we had not noticed this we still would have carried out the computations as follows:
( ) ( )
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−=+=01
1ˆ01 yGxx as before
( ) ( )
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−=+=
05.0
5.0ˆ12 yGxx
( ) ( )
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−=+=
0625.0
625.0ˆ23 yGxx
( ) ( )
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−=+=
06875.0
6875.0ˆ34 yGxx
VittalRao/IISc, Bangalore M2/L3/V1/May 2004/6
Numerical Analysis/ Iterative methods for solving linear system of equation Lecture notes
( ) ( )
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−=+=
065625.0
65625.0ˆ45 yGxx ; ( ) ( ) 03125.045 =−
∞xx
( ) ( )
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−=+=
0671875.0
671875.0ˆ56 yGxx ; ( ) ( ) 025625.056 =−
∞xx
( )
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−=
0664062.0
664062.07x ; ( ) ( ) 007813.067 =−
∞xx
( )
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−=
0667969.0
667969.08x ; ( ) ( ) 003907.078 =−
∞xx
( )
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−=
0666016.0
666016.09x ; ( ) ( ) 001953.089 =−
∞xx
( )
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−=
0666504.0
666504.010x ; ( ) ( ) 000488.0910 =−
∞xx
(Since now error is < 10-3 we may stop here and take x(10) as our solution for the system. Or we may improve our accuracy by doing more iterations, to get,
( )
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−=
0666748.0
666748.011x ; ; ( )
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−=
0666626.0
666626.012x ( )
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−=
0666687.0
666687.013x
( )
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−=
0666656.0
666656.014x ( ) ( ) 41314 10000031.0 −
∞<=− xx
and hence we can take x(14) as our solution within error 10-4.
Let us now try to apply the Jacobi scheme for this system. We have
VittalRao/IISc, Bangalore M2/L3/V1/May 2004/7
Numerical Analysis/ Iterative methods for solving linear system of equation Lecture notes
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
−−
−−
=
121
21
11121
211
A ; and therefore,
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
−−=
021
21
10121
210
J
We have the characteristic polynomial of J as
⎟⎠⎞
⎜⎝⎛ +−⎟⎠⎞
⎜⎝⎛ +=
−−++−−
=− 122
1
21
21
112
12
12 λλλ
λλ
λλ JI
Thus the eigenvalues of J are
415
21;4
1521;
21
321 ii −=+=−= λλλ
2416
415
41;
21
321 ==+===∴ λλλ
2=∴sp
J which is >1. Thus the Jacobi scheme for this system will not converge.
Thus, in example 3 we had a system for which the Jacobi scheme converged but Gauss – Seidel scheme did not converge; where in example 4 above we have a system for which the Jacobi scheme does not converge, but the Gauss – Seidel scheme converges. Thus, these two examples demonstrate that, in general, it is not ‘correct’ to say that one scheme is better than the other.
Let us now consider another example.
Example 5:
2x1 – x2 =y1
-x1 + 2x2 – x3 = y2
-x2 + 2x3 –x4 =y3
-x3 + 2x4 = y4
VittalRao/IISc, Bangalore M2/L3/V1/May 2004/8
Numerical Analysis/ Iterative methods for solving linear system of equation Lecture notes
Here
,
21001210
01210012
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
−−−
−−−
=A is a symmetric tridiagonal matrix.
The Jacobi matrix for this scheme is
⎟⎟⎟⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜⎜⎜⎜
⎝
⎛
=
02100
210
210
0210
21
00210
J
The characteristic equation is,
16λ 4 - 12λ 2 + 1 = 0 ………………(CJ)
Set λ 2 = α
Therefore
16α2 - 12α + 1 = 0 ………………(CJ1)
∴ λ is the square root of the roots of (CJ1).
Thus the eigenvalues of J are ± 0.3090; ± 0.8090. Hence
8090.0=sp
J ; and the Jacobi scheme will converge.
The Gauss – Seidel matrix for the system is found as follows:
VittalRao/IISc, Bangalore M2/L3/V1/May 2004/9
Numerical Analysis/ Iterative methods for solving linear system of equation Lecture notes
( )⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
−−
−=+
2100021000210002
LD
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
=−
0000100001000010
U
( )
⎟⎟⎟⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜⎜⎜⎜
⎝
⎛
=+ −
21
41
81
161
021
41
81
0021
41
00021
1LD
( )⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
⎟⎟⎟⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜⎜⎜⎜
⎝
⎛
=+−= −
0000100001000010
21
41
81
161
021
41
81
0021
41
00021
1ULDG
⎟⎟⎟⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜⎜⎜⎜
⎝
⎛
=
41
81
1610
21
41
810
021
410
00210
The characteristic equation of G is
0=−GIλ , which becomes in this case
)(....................01216 234GC=+− λλλ
VittalRao/IISc, Bangalore M2/L3/V1/May 2004/10
Numerical Analysis/ Iterative methods for solving linear system of equation Lecture notes
This can be factored as
( ) 011216 22 =+− λλλ
Thus the eigenvalues of G are roots of
λ2 = 0 ; and
16λ2 - 12λ + 1 = 0 ………….(CG1) Thus one of the eigenvalues of G is 0 (repeated twice), and two eigenvalues of G are roots of (CG1). Notice that roots of (CG1) are same as those of (CJ1). Thus nonzero eigenvalues of G are squares of eigenvalues of J. ∴ the nonzero eigenvalues of G are,
0.0955, 0.6545.
Thus,
16545.0 <=sp
G
Thus the Gauss – Seidel scheme also converges. Observe that
spspJG 2= ; spsp
JG <
Thus the Gauss – Seidel scheme converges faster than the Jacobi scheme.
In many class of problems where both schemes converge it is the Gauss – Seidel scheme that converges faster. We shall not go into any further details of this aspect.
VittalRao/IISc, Bangalore M2/L3/V1/May 2004/11
Numerical Analysis ( Iterative methods for solving linear systems of equations) Lecture notes
VittalRao/IISc, Bangalore M2/L4/V1/May 2004/1
SUCCESSIVE OVERRELAXATION METHOD (SOR METHOD)
We shall now consider SOR method for the system
Ax = y ………..(I)
We take a parameter ω ≠ 0 and multiply both sides of (I) to get an equivalent system,
ωAx = ωy ………………(II)
Now
( )ULDA ++=
We write (II) as
(ωD + ωL + ωU)x = ωy,
i.e.
(ωD + ωL) = - ωUx + ωy
i.e.
(D + ωL)x + (ω-1) Dx = - ωUx +ωy
i.e.
(D + ωL)x = - [(ω – 1)D + ωU]x + ωy
i.e.
x = - (D + ωL)-1 [(ω-1)D + ωU]x + ω [D + ωL]-1y.
We thus get the SOR scheme as
( ) ( )
( ) ;
ˆ0
1
θω
=
+=+
x
yxMx kk
initial guess ……………(III)
where,
( ) ( )[ ]uDLDM ωωωω +−+−= − 11
and
( ) yLDy ωω 1ˆ −+=
Mω is the SOR matrix for the system.
Numerical Analysis ( Iterative methods for solving linear systems of equations) Lecture notes
VittalRao/IISc, Bangalore M2/L4/V1/May 2004/2
Notice that if ω = 1 we get the Gauss – Seidel scheme. The strategy is to choose ω such that ,1<isM
spω and is al small as possible so that the scheme converges as rapidly as
possible. This is easier said than achieved. How does one choose ω? It can be shown that convergence cannot be achieved if ω ≥ 2. (We assume ω > 0). ‘Usually’ ω is chosen
between 1 and 2. Of course, one must analyse sp
Mω as a function of ω and find that
value ω0 of ω for which this is minimum and work with this value of ω0.
Let us consider an example of this aspect.
Example 6:
Consider the system given in example 5.
For that system,
Mω = - (D +ω L)-1 [(ω-1) D +ωU]
⎟⎟⎟⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜⎜⎜⎜
⎝
⎛
+−+−+−−
+−+−−
+−−
−
=
23243243
23232
22
411
81
21
21
161
41
41
81
81
21
411
81
21
21
41
41
021
411
21
21
00211
ωωωωωωωωωω
ωωωωωωωω
ωωωωω
ωω
and the characteristic equation is
( ) ( ) ( )ωλωλωλωλω MC.................0112116 24224 =++−−+−
Thus the eigenvalues of Mω are roots of the above equation. Now when is λ = 0 a root? If λ = 0 we get from (CMω),
16(ω-1)4 = 0 ⇒ ω = 1,
i.e. in the Gauss – Seidel case. So let us take ω ≠ 1; so λ = 0 is not a root. So we can divide the above equation (CMω) by ω4λ2 to get
( ) ( ) 01112116 2
22
2
2
=++−
−⎥⎦
⎤⎢⎣
⎡ +−λωλω
λωλω
Setting
Numerical Analysis ( Iterative methods for solving linear systems of equations) Lecture notes
VittalRao/IISc, Bangalore M2/L4/V1/May 2004/3
( )λωλωµ 2
22 1+−= we get
011216 24 =+− µµ
which is the same as (CJ). Thus
.8090.0;3090.0 ±±=µ
Now
( ) 22
21 µλωλω
=+−
= 0.0955 or 0.6545 ……….(*)
Thus, this can be simplified as
( ) ( ) 21
2222 1411
21
⎭⎬⎫
⎩⎨⎧ −−±−−= ωωµµωωωµλ
as the eigenvalues of Mω.
With ω = 1.2 and using the two values of µ2 in (*) we get,
λ = 0.4545, 0.0880, -0.1312 ± i (0.1509).
as the eigenvalues. The modulus of the complex roots is 0.2
Thus
sp
M ω when ω = 1.2 is 0.4545
which is less that sp
J = 0.8090 and sp
G = 0.6545 computed in Examples. Thus for
this system, SOR with ω = 1.2 is faster than Jacobi and Gauss – Seidel scheme.
We can show that in this example when ω = ω0 = 1.2596, the spectral radius 0ωM is
smaller than ωM for any other ω. We have
2596.1M = 0.2596
Thus the SOR scheme with ω = 1.2596 will be the method which converges fastest.
Note:
We had sp
M 2.1 = 0.4545
Numerical Analysis ( Iterative methods for solving linear systems of equations) Lecture notes
VittalRao/IISc, Bangalore M2/L4/V1/May 2004/4
And
spM 2596.1 = 0.2596
Thus a small change in the value of ω brings about a significant change in the spectral
radius spM ω .
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L2/V1/May2004/1
EIGENVALUES AND EIGENVECTORS
Let A be an nxn matrix. A scalar α is called an eigenvalue of A if there exists a nonzero nx1 vector x such that
Ax = αx
Example:
Let
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
−−−
=7816438449
A
1−=αLet
Consider
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛=
021
x . We have
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−=
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−−
=⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
−−−
=021
1021
021
7816438449
Ax
( ) xx α=−= 1
1−=∴α is such that there exists a nonzero vector x such that Ax = αx. Thus α is an eigenvalue of A.
Similarly, if we take α = 3, we find that ⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛=
021
x
Ax = αx. Thus, α = 3 is also an eigenvalue of A.
Let α be an eigenvalue of A. Then any nonzero x such that Ax = αx is called an eigenvector of A.
Let α be an eigenvalue of A. Let,
{ }.: xAxCx n αω α =∈=
Then : (i) ωα is nonempty. αωθ ∈= nxQ
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L2/V1/May2004/2
α
α
ωαααω
∈+⇒+=+⇒==⇒∈
yxyxyxAyAyxAxyxii )()(,,)(
(iii)
For any constant )(; xxAx κακακκ == )()( xxA κακ =⇒
αωκ ∈⇒ x
Thus ωα is a subspace of Cn. This is called the characteristic subspace or the eigensubspace corresponding to the eigenvalue α.
Example:
Consider the A in the example on page 1. We have sum α = -1 is an eigenvalue. What is ω-1, the eigensubspace corresponding to –1?
We want to find all x such that
Ax = -x
i.e., (A+I)x = θ.
i.e., we want to find all solutions of the homogeneous system Mx = θ ; where
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
−−−
=+=7816448448
IAM
We now can use our row reduction to find the general solution of the system.
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛ −−
⎯⎯ →⎯⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛ −−−
−→
000000
21
211
000000448
112
13
81
2
RRR
RRM
Thus, 321 21
21 xxx +=
Thus the general solution of (A+I) x = θ is
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L2/V1/May2004/3
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛+
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛=
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛ +
201
21
021
212
121
32
3
2
32
xxxx
xx
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛+
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛=
201
021
21 AA
where A1 and A2 are arbitrary constants.
Thus ω-1 consists of all vectors of the form
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛+
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
201
021
21 AA .
Note: The vectors form a basis for ω⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
201
,021
-1 and therefore
dim ω-1 = 2.
What is ω3 the eigensubspace corresponding to the eigenvalue 3 for the above matrix We need to find all solutions of Ax = 3x,
i.e., Ax – 3x = θ
i.e., Nx = θ Where
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
−−−
=−=48164084412
3 IAN
Again we use row reduction
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L2/V1/May2004/4
⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜
⎝
⎛
−
−
⎯⎯ →⎯
⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜
⎝
⎛
−
−
−+
−
−
→00034
380
4412
34
380
34
380
441243
12
13
32
34
RRRR
RR
N
321 4412 xxx +=∴
32 34
38 xx = 23 2 xx =∴
2221 128412 xxxx =+=∴
12 xx =∴
12312 22; xxxxx ===∴
∴ The general solution is
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛=
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
211
21
1
1
1
xx
xx
Thus ω3 consists of all vectors of the form
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
211
κ
Where κ is an arbitrary constant.
Note: The vector forms a basis for ω⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
211
3 and hence
dim. ω3 = 1.
Now When can a scalar α be an eigenvalue of a matrix A? We shall now investigate this question. Suppose α is an eigenvalue of A.
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L2/V1/May2004/5
This There is a nonzero vector x such that Ax = αx.
.;)( θθα ≠=−⇒ andxxIA
The system θα =− xIA )( has at least one nonzero
solution.
nullity (A - αI) ≥ 1
rank (A - αI) < n
(A - αI) is singular
det. (A - αI) = 0
Thus, α is an eigenvalue of A det. (A - αI) = 0.
Conversely, α is a scalar such that det. (A - αI) = 0.
This (A - αI) is singular
rank (A - αI) < n
nullity (A - αI) ≥ 1
The system θα =− xIA )( has nonzero solution.
α is an eigenvalue of A.
Thus, α is a scalar such that det. (A - αI) = 0 α is an eigenvalue.
Combining the two we get,
α is an eigenvalue of A
det. (A - αI) = 0
det. (αI - A) = 0
Now let C(λ) = det. (λI - A) Thus we see that,
“The eigenvalues of a matrix A are precisely the roots of C(λ) = det. (λI - A)”.
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L2/V1/May2004/6
( )
nnnn
n
n
aaa
aaaaaa
C
−−−
−−−−−−
=
λ
λλ
λ
K
KKKK
KKKK
K
K
21
22221
11211
( ) ( ) Aaa nnnn
n .det1111 −++++−= − KK λλ
Thus ; C(λ) is a polynomial of degree n. Note the ‘leading’ coefficient of C(λ) is 1. We say C(λ) is a ‘monic’ polynomial of degree n. This is called CHARACTERISTIC POLYNOMIAL of A. The roots of the characteristic polynomial are the eigenvalues of A. The equation C(λ) = 0 is called the characteristic equation.
Sum of the roots of C(λ) = Sum of the eigenvalues of A = a11 + . . . . . . + ann , and this is called the TRACE of A.
Product of the roots of C(λ) = Product of the eigenvalues of A = det. A. In our example in page 1 we have
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
−−−
=7816438449
A
( )7816
438449
).(det−−−−−−+
=−=∴λ
λλ
λλ AIC
781431441
321
−−+−−+−−+
⎯⎯⎯ →⎯ ++
λλλλ
λCCC
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L2/V1/May2004/7
( )781
431441
1−−
−−−−
+=λ
λλ
340
010441
)1(12
13 −−+
−−+=→
−
− λλλ
RR
RR
( )( )( )311 −++= λλλ
( ) ( )31 2 −+= λλ
Thus the characteristic polynomial is
( ) ( )31)( 2 −+= λλλC
The eigenvalues are –1 (repeated twice) and 3.
Sum of eigenvalues = (-1) + (-1) + 3 = 1
Trace A = Sum of diagonal entries.
Product of eigenvalues = (-1) (-1) (3) = 3 = det. A.
Thus, if A is an nxn matrix, we define the CHARACTERISTIC POLYNOMIAL as,
AIC −= λλ)( . . . . . . . . . . . . .(1)
and observe that this is a monic polynomial of degree n. When we factorize this as,
( ) ( ) ( ) kak
aaC λλλλλλλ −−−= KK2121)( . . . . . . . .(2)
Where λ1, λ2, . . . . . ., λk are the distinct roots; these distinct roots are the distinct eigenvalues of A and the multiplicities of these roots are called the algebraic multiplicities of these eigenvalues of A. Thus when C(λ) is as in (2), the distinct eigenvalues are λ1, λ2, . . . . . ., λk and the algebraic multiplicities of these eigenvalues are respectively, a1, a2, . . . . . , ak.
For the matrix in Example in page 1 we have found the characteristic polynomial on page 6 as
( ) ( 31)( 2 −+= λλλC )
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L2/V1/May2004/8
Thus the distinct eigenvalues of this matrix are λ1 = -1 ; and λ2 = 3 and their algebraic multiplicities are respectively a1 = 2 ; a2 = 1.
If λi is an eigenvalues of A the characteristic subspace corresponding to λi is iλω and is
defined as
{ }xAxx iiλωλ == :
The dimension of iλω is called the GEOMETRIC MULTIPLICITY of the eigenvalue λi
and is denoted by gi.
Again for the matrix on page 1, we have found on pages 3 and 4 respectively that, dim ω-
1 = 2 ; and dim. ω3 = 1. Thus the geometric multiplicities of the eigenvalues λ1 = -1 and λ2 = 3 are respectively g1 = 2 ; g2 = 1. Notice that in this example a1 = g1 = 2 ; and a2 = g2 = 1. In general this may not be so. It can be shown that for any matrix A having C(λ) as in (2),
1 ≤ gi ≤ ai ; 1 ≤ i ≤ k . . . . . . . . . . . .(3)
i.e., for any eigenvalue of A,
1 ≤ geometric multiplicity ≤ algebraic multiplicity. We shall study the properties of the eigenvalues and eigenvectors of a matrix. We shall start with a preliminary remark on Lagrange Interpolation polynomials :
Let α1, α2, . . . . . . . ., αs be a distinct scalars, (i.e., αi ≠ αj if i ≠ j ). Consider,
)())(())(()())(())((
)(1121
1121
siiiiiii
siiip
αααααααααααλαλαλαλαλ
λ−−−−−
−−−−−=
+−
+−
KK
KK
)()(
1 ji
j
sjij
αααλ
−
−∏=≠≤≤ for i = 1,2, . . . . . . ., s . . . . . . .. (4)
Then pi(λ) are all polynomials of degree s-1.
Further notice that ( ) ( ) ( ) 0)(111 ====== +− siiiiii pppp αααα KK
( ) 1=iip α
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L2/V1/May2004/9
Thus pi(λ)are all polynomials of degree s-1 such that,
( ) ijjip δα = if j ≠ i . . . . . . . . . . (5)
We call these the Lagrange Interpolation polynomials. If p(λ) is any polynomial of degree ≤ s-1 then it can be written as a linear combination of p1(λ),p2(λ), . . ., ps(λ) as follows:
( ) ( ) ( ) ( ) ( ) ( ) ( )λαλαλαλ ss ppppppp +++= L2211 . . . . (6)
( ) (∑=
=s
iii pp
1λα )
With this preliminary, we now proceed to study the properties of the eigenvalues and eigenvectors of an nxn matrix A.
Let λ1, . . . . , λk be the distinct eigenvalues of A. Let φ1, φ2, . . . , φk be eigenvectors corresponding to these eigenvalues respectively ; i.e., φi are nonzero vectors such that
Aφi = λiφi . . . . . . . . . . . .(6)
From (6) it follows that
iiiiiiii AAAAA φλφλφλφφ 22 )()( ====
iiiiiiii AAAAA φλφλφλφφ 32223 )()( ====
and by induction we get
iim
imA φλφ = for any integer m ≥ 0 . . . . . . . . . . .(7)
(We interpret A0 as I).
Now let,
s
saaap λλλ +++= KK10)(be any polynomial. We define p(A) as the matrix,
s
s AaAaIaAp +++= KK10)(
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Now
is
si AaAaIaAp φφ )()( 10 +++= KK
is
sii AaAaa φφφ +++= KK10
by (6) iis
siii aaa φλφλφ +++= KK10
iis
si aaa φλλ )( 10 +++= KK
.)( iip φλ=
Thus,
envalues λ
Np
a
N
Now are the eigenvectors, φ1, φ2, . . . . , φk corresponding to the distinct eig
If λi is any eigenvalue of A and φi is an eigenvector corresponding to λi then
for any polynomial p(λ) we have .)()( iii pAp φλφ =
Vittal rao/IISc.Bangalore M3/L2/V1/May2004/10
1, λ2, . . . . , λk of A, linearly independent ? In order to establish this linear independence, we must show that
0212211 ====⇒=+++ KnKK CCCCCC KK θφφφ . . . (8)
ow if in (4) & (5) we take s = k ; αi = λi then we get the Lagrange Interpolation olynomials as
( ))()(
1 ji
j
kji
ij
pλλλλ
λ−
−∏=≠≤≤ ; i = 1,2,….., k …………(9)
nd
( ) ijjip δλ = if j ≠ i …………(10)
ow,
nkkCCC θφφφ =+++ ....2211
For 1≤ i ≤ k,
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L2/V1/May2004/11
( )[ ] ( ) nnikki ApCCCAp θθφφφ ==+++ ....2211
( ) ( ) nkikii ApCApCApC θφφφ =+++⇒ ....)( 2211
( ) ( ) ,....)( 222111 nkkikii pCpCpC θφλφλφλ =+++⇒ (by property I on page 10)
;1; kiC ii ≤≤=⇒ θφ by (10)
kiCi ≤≤=⇒ 1;0 since φi are nonzero vectors
Thus
0........ 212211 ====⇒=+++ nnkk CCCCCC θφφφ proving (8). Thus we have
Eigen vectors corresponding to distinct eigenvalues of A are linearly independent.
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L3/V1/May2004/1
SIMILAR MATRICES
We shall now introduce the idea of similar matrices and study the properties of similar matrices. DEFINITION
An nxn matrix A is said to be similar to a nxn matrix B if there exists a nonsingular nxn matrix P such that,
P-1 A P = B
We then write,
A ∼ B Properties of Similar Matrices
(1) Since I-1 A I = A it follows that A ∼ A
(2) A ∼ B ∃ P, nonsingular show that., P-1 A P = B
A = P B P-1
A = Q-1 B P, where Q = P-1 is nonsingular
∃ nonsingular Q show that Q-1 B Q = A
B ∼ A
Thus
A ∼ B B ∼ A
(3) Similarly, we can show that
A ∼ B, B ∼ C A ∼ C.
(4) Properties (1), (2) and (3) above show that similarity is an equivalence relation on the set of all nxn matrices.
(5) Let A and B be similar matrices. Then there exists a nonsingular matrix P such that
A = P-1 B P
Now, let CA(λ) and CB (λ) be the characteristic polynomials of A and B respectively. We have,
( ) BPPIAIC A1−−=−= λλλ
BPPPP 11 −− −= λ
( )PBIP −= − λ1
PBIP −= − λ1
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L3/V1/May2004/2
1sin 1 =−= − PPceBIλ
= CB (λ )
Thus “ SIMILAR MATRICES HAVE THE SAME CHARACTERISTIC POLYNOMIALS ”.
(6) Let A and B be similar matrices. Then there exists a nonsingular matrix P such that
A = P-1 B P
Now for any positive integer k, we have
( )( ) ( )4444 34444 21
ktimes
k BPPBPPBPPA 111 ..... −−−=
= P-1 Bk P
Therefore, Ak = On P-1 Bk P = On
Bk = On
“ Thus if A and B are similar matrices then Ak = On Bk = On ”.
Now let p(λ) = a0 + a1λ + ….. + akλ be any polynomial.
Then
( ) kk AaAaIaAp +++= .....10
PBPaPBPaBPPaIa kk
1212
110 ..... −−− ++++=
[ ]PBaBaBaIaP kk++++= − .....2
2101
( )PBpP 1−=
Thus
( ) ( ) nn OPBpPOAp =⇔= −1
( ) nOBp =⇔
Thus “ IF A and B ARE SIMILAR MATRICES THEN FOR ANY POLYNOMIAL p (λ); p (A) = On p (B) = On ”.
(7) Let A be any matrix. By A(A) we denote the set of all polynomials p(λ) such that
p(A) = On, i.e.
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L3/V1/May2004/3
A (A) = {p(λ) : p(A) = On}
Now from (6) it follows that,
“ IF A AND B ARE SIMILAR MATRICES THEN A(A) = A (B) ”.
Then set A (A) is called the set “ ANNIHILATING POLYNOMIALS OF A ”. Thus similar matrices have the same set of annihilating polynomials.
We shall discuss more about annihilating polynomials later.
We now investigate the following question? Given an nxn matrix A when is it similar to a “simple matrix”? What are simple matrices? The simplest matrix we know is the zero matrix On. Now A ∼ On . There is a nonsingular matrix P such that A = P-1 On P = On.
∴ “ THE ONLY MATRIX SIMILAR TO On IS ITSELF ”.
The next simple matrix we know is the identity matrix In. Now A ∼ In there is a nonsingular P such that A = P-1 In P A = In.
Thus “THE ONLY MATRIX SIMILAR TO In IS ITSELF ”.
The next class of simple matrices are the DIAGONAL MATRICES. So we now ask the question “ Which type of nxn matrices are similar to diagonal matrices”?
Suppose now A is an nxn matrix; and A is similar to a diagonal matrix,
λ1
D = λ2
λn
(λI not necessarily distinct).
Then there exists a nonsingular matrix P such that
P-1 A P = D
AP = PD ………..(1)
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
=
nnninn
ni
ni
PPPP
PPPPPPPP
LetP
..........
..........
..........
21
222221
111211
MMMMMM
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L3/V1/May2004/4
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
=
nnnn
n
n
aaa
aaaaaa
A
.........................
.....
.....
21
22221
11211
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
=
ni
i
i
i
P
PP
LetPM2
1
denote the ith column of P.
Now the ith column of AP is
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
+++
++++++
ninninin
ninii
ninii
PaPaPa
PaPaPaPaPaPa
.................................................
.....
.....
2211
2222121
1212111
which is equal to APi.
Thus the ith column of A P, the L.H.S. of (1), is A Pi.
Now the ith column of P D is
ii
ni
i
i
i
ini
ii
ii
P
P
PP
P
PP
λλ
λ
λλ
=
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
=
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
MM2
1
2
1
Thus the ith column of P D, the R.H.S. of (1), is λI Pi. Since L.H.S. = R.H.S. by (1) we have
APi = λi Pi ; i = 1, 2, …., n ……………..(2)
Note that since P is nonsingular no column of P can be zero vector. Thus none of the column vectors Pi are zero. Thus we conclude that,
“IF A IS SIMILAR TO A DIAGONAL MATRIX D THEN THE DIAGONAL ENTRIES OF D MUST BE THE EIGENVALUES OF A AND IF P-1AP = D THEN THE ith COLUMN VECTOR MUST BE AN EIGENVECTOR CORRESPON DING TO THE EIGENVALUE WHICH IS THE ith DIAGONAL ENTRY OF D”.
Note:
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L3/V1/May2004/5
The n columns of P must be linearly independent since P is nonsingular and thus these n columns give us n linearly independent eigenvectors of A
Thus the above result can be restated as follows: A is similar to a diagonal matrix D and P-1 A P = D A has n linearly independent eigenvectors; taking these as the columns of P we get P-1 A P we get D where the ith diagonal entry of D is the eigenvalue corresponding to the ith eigenvector.
Conversely, it is now obvious that if A has n linearly independent eigenvectors then A is similar to a diagonal matrix D and if P is the matrix whose ith column is the eigenvector, then D is P-1 A P and ith diagonal of D is the eigenvalue corresponding to the ith eigenvector.
When does then a matrix have n linearly independent eigenvectors’. It can be shown that a matrix A has n linearly independent eigenvectors the algebraic multiplicity of each eigenvalue of A is equal to its geometric multiplicity. Thus
A IS SIMILAR TO A DIAGONAL MATRIX
FOR EVERY EIGENVALUE OF A, ALGEBRAIC MULTIPLICITY IS EQUAL TO ITS GEOMETRIC MULTPLICITY”.
RECALL; if ( ) ( ) ( ) ( ) kak
aaC λλλλλλλ −−−= .....2121 where λ 1, λ 2, ….., λ k are
the distinct eigenvalues of A, then ai is called the algebraic multiplicity of the eigenvalue λ i. Further, let
{ }xAxx ii λω == :
be the eigensubspace corresponding to λ i. Then gi = dim ωi is called the geometric multiplicity of λ i.
Therefore, we have,
“ If A is an nxn matrix with C(λ ) = (λ -λ 1)a1…… (λ -λ k) ak where λ 1, ….., λ k are the
district eigenvalues of A, then A is similar to a diagonal matrix ai = gi (=dimωi) ; 1≤ i ≤ k”.
Example:
Let us now consider
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
−−−
=7816438449
A
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L3/V1/May2004/6
On page 6, we found the characteristic polynomial of A as
C(λ ) = (λ +1)2 (λ - 3)
Thus λ 1 = -1 ; a1 = 2
λ 2 = 3 ; a2 = 1
On pages 3 and 4 we found,
ω1 = eigensubspace corresponding to λ = -1
⎪⎭
⎪⎬
⎫
⎪⎩
⎪⎨
⎧
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛+
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛==
201
021
: 21 AAxx
ω2 = eigensubspace corresponding to λ = 3
⎪⎭
⎪⎬
⎫
⎪⎩
⎪⎨
⎧
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛==
211
: kxx
Thus dim ω1 = 2 ∴ g1 = 2
dim ω2 = 2 ∴ g2 = 1
Thus a1 = 2 = g1 and ence A must be similar.
a2 = 1 = g2
to a diagonal matrix. How do we get P such that P-1AP is a diagonal matrix? Recall the columns of P must be linearly independent eigenvectors. From ω1 we get two linearly
eigenvectors, namely, and ; and from ω⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
021
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
201
2 we get third as . ⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
211
Thus if we take these as columns and write,
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛=
220102111
P
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L3/V1/May2004/7
Then ⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜
⎝
⎛
−−−
−
=−
1122
1122
1011P ; and it can be verified that
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
−−−
⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜
⎝
⎛
−−−
−
=−
220102111
7816438449
1122
1122
1011 APP
a diagonal matrix. ⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−
−=
300010001
Thus we can conclude that A is similar to a diagonal matrix, i.e., P-1 AP = D A has n linearly independent eigenvectors namely the n columns of AP.
Conversely, A has n linearly independent eigenvectors P-1 AP is a diagonal matrix where the columns of P are taken to be the n linearly eigenvectors.
We shall now see a class of matrices for which it is easy to decide whether they are similar to a diagonal matrix; and in which case the P-1 is easy to compute. But we shall first introduce some preliminaries.
If ; are any two vectors in C
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
=
nx
xx
xM2
1
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
=
ny
yy
yM2
1
n, we define the INNER PRODUCT OF
x with y (which is denoted by (x,y)) as,
( ) nn yxyxyxyx +++= K2211, ∑=
=n
iii yx
1
Example 1:
If ; ; then, ⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
−+=1
2 ii
x⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−=i
iy 11
( )( ) ( )( )iiiiyx 1121.),( −+−++=
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L3/V1/May2004/8
( )( ) ( )( ) iiiii 51112 +=−−++++=
Whereas ( ) ( )( ) ( )( ) iiiiixy 511211),( −=−++−+=
We now observe some of the properties of the inner product, below:
(1) For any vector x in Cn, we have
( ) ,,1
2
1∑∑==
==n
ii
n
iii xxxxx
Which is real ≥ 0. Further,
( ) ∑=
=⇔=n
iixxx
1
2 00,
nix i ≤≤=⇔ 1;0
nx θ=⇔
Thus,
(x,x) is real and ≥ 0 and = 0 x = θn
(2) ( ) ⎟⎟⎠
⎞⎜⎜⎝
⎛== ∑∑
==
n
iii
n
iii xyyxyx
11,
( )xy ,=
Thus,
( ) ( )xyyx ,, =
(3) For any complex number α, we have,
( ) ( ) ∑∑==
==n
iii
n
iii yxyxyx
11
, ααα
( )yx ,α=
Thus
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L3/V1/May2004/9
)
(αx,y) = α (x,y) for any complex number α.
We note,
( ) ( xyyx ,, αα = by (2)
( ) ( ) ( )yxxyxy ,,, ααα ===
(4) ( ) ( )∑=
+=+n
iiii zyxzyx
1, ∑ ∑
= =
+=n
i
n
iiiii zyzx
1 1
( ) ( zxyx ,, += )
Thus
(x + y, z) = (x,z) + (y,z) and similarly
(x, y + z) = (x, y) + (x, z)
We say that two vectors x and y are ORTHOGONAL if (x, y) = 0.
Example :
(1) If x = ,0
1;
1
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−=
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
−iy
ii
then,
( ) ( ) ( ) ( )( )011, iiiyx −++−=
= -1 + 1 = 0
Thus x and y are orthogonal.
(2) If ⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−=
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
−=
1
1,
1ay
iix
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L3/V1/May2004/10
then
( ) iiayx −+−= 1,
∴ x, y orthogonal
( )
( )
ia
iiii
ia
iai
+=⇔
−=+−=⎟⎠⎞
⎜⎝⎛ +
=⇔
=++−⇔
1
11101
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L4/V1/May 2004/1
HERMITIAN MATRICES
Let A = (aij); be an nxn matrix. We define the Hermitian conjugate of A, denoted by A* as ; A* = (a*ij) where a*
ij = aji.
A* is the conjugate of the transpose of A.
Example 1:
⎟⎟⎠
⎞⎜⎜⎝
⎛−
=iii
A1
Transpose of ⎟⎟⎠
⎞⎜⎜⎝
⎛ −=
iii
A1
⎟⎟⎠
⎞⎜⎜⎝
⎛−−
=∴ii
iA
1*
Example 2:
⎟⎟⎠
⎞⎜⎜⎝
⎛−
=2
1i
iA
Transpose of ⎟⎟⎠
⎞⎜⎜⎝
⎛ −=
21i
iA
⎟⎟⎠
⎞⎜⎜⎝
⎛−
=∴2
1*
ii
A
Observe that in Example 1. A* ≠ A. Whereas in Example 2, A* = A.
DEFINITION: An nxn matrix A is said to be HERMITIAN if
A* = A.
We now state some properties of Hermitian matrices.
(1) If A = (aij) ; A* = (a*ij) and A = A* then aii = a*
ii = aii
Thus the DIAGONAL ENTRIES OF A HERMITIAN MATRIX ARE REAL.
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L4/V1/May 2004/2
(2) Let be any two vectors in C
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
=
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
=
nn y
yy
y
x
xx
xMM
2
1
2
1
; n.
Let
( )( )
( )
( )( )
( ) ⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
=
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
=
nn Ay
AyAy
Ay
Ax
AxAx
AxMM
2
1
2
1
;
We have
( ) ( ) .;11
∑∑==
==n
iijij
n
jjiji yaAyxaAx
Now
( ) ( )∑=
=n
iii yAxyAx
1,
∑ ∑= =
⎟⎟⎠
⎞⎜⎜⎝
⎛=
n
ii
n
jjij yxa
1 1
∑ ∑= =
⎟⎠
⎞⎜⎝
⎛=
n
j
n
iiijj yax
1 1
∑ ∑= =
⎟⎠
⎞⎜⎝
⎛=
n
j
n
iiijj yax
1 1
∑ ∑= =
⎟⎠
⎞⎜⎝
⎛=
n
j
n
iijij yax
1 1)sin( *AceAaa jiij ==Q
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L4/V1/May 2004/3
( )∑=
=n
jjj Ayx
1
= (x, Ay)
Thus IF A IS HERMITIAN THEN
(Ax, y) = (x, Ay)
FOR ANY TWO VECTORS x, y.
(3) Let λ be any eigenvalue of A. Then there is an x ∈ Cn, x ≠ θn such that
Ax = λx. Now,
( ) ( ) (
( )
)
( )xx
xxAxx
xAxxxxx
,
,),(
,,,
λ
λ
λλ
=
==
==
A is Hermitian.
( )( ) ( ) nxxxButxx θλλ ≠≠=−∴ Q0,.0,
λλλλ =∴=−∴ 0 ∴ λ is real.
THUS THE EIGENVALUES OF A HERMITIAN MATRIX ARE ALL REAL.
(4) Let λ, µ be two different eigenvalues of A and x, y corresponding eigenvectors. We have,
Ax = λx and Ay = µy and λ, µ are real by (3). Now,
( ) ( yxyx ,, )λλ =
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L4/V1/May 2004/4
( )( )( )
( )( ) .,
,
,)2(,
,
isrealyxyx
yxbyAyx
yAx
µµµ
µ
Q==
===
( )( ) µλµλ ≠=−∴ Butyx .0,
∴ (x,y) = 0 x and y are orthogonal.
THUS IF A IS A HERMITIAL MATRIX THEN THE EIGENVECTORS CORRESPONDING TO DISTINCT EIGENVALUES ARE ORTHOGONAL.
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L5/V1/May 2004/1
Gramm – Schmidt OrthonormalizationWe shall now discuss the Gramm – Schmidt Orthonormalization process:
Let U1, U2, …., Uk be k linearly independent vectors in Cn. The Gramm – Schmidt process is the method to get an orthonormal set kφφφ ,.....,, 21 show that the subspace ω
spanned by U1, ….., Uk is the same as the subspace spanned by kφφ ,.....,1 thus providing an orthonormal basis for ω.
The process goes as follows:
Let ;11 U=ψ
( )1
, 111
1
1
11 === φ
ψψψ
ψψ
φ Note
Next, let,
( ) 11222 φφψ UU −=
Note that
( )12φψ
( ) ( )( )112212 ,, φφφφ UU −=
( ) ( )( 112212 , )φφφφ UU −=
( ) ( ) ( ) 1, 111212 =−= φφφφ QUU
.12 φψ ⊥∴
Let
;2
22 ψ
ψφ = clearly ( ) 0,,1,1 2112 === φφφφ
Also
x = α1 U1 + α2 U2 then
( )( )1122211 , φφψαψα Ux ++=⇔
( )[ ]112222111 , φφφψαφψα Ux ++=⇔
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L5/V1/May 2004/2
2211 φβφβ +=⇔ x where
( )122111 ,φαψαβ U+=
222 ψαβ =
Thus xε subspace spanned by U1, U2
xε subspace spanned by φ1, φ2.
Thus φ1, φ2 is an orthonormal basis for the subspace [U1,U2].
Having defined φ1, φ2,….., φi-1 we define φi as follows:
( )∑−
=
−=1
1
,i
piiiii UU φφψ Clearly ( ) 0, =pi φψ 1≤ p ≤ i-1
and i
ii ψ
ψφ =
Obviously ( ) 110,1 −≤≤== ijforand jii φφφ
and xε [U1, U2, ….., Ui] xε [φ1, ….., φi]
and thus φ1, φ2, ….., φi is an orthonormal basis for [U1, ….., Uk]. Thus at the kth stage we get an orthonormal basis φ1, …., φk for [U1, ….., Uk].
Example:
Let
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
=
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
−=
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
=
0132
;
01
11
;
0111
321 UUU
be l.i. Vectors in R4. Let us find an orthonormal basis for the subspace ω spanned by U1, U2, U3 using the Gramm – Schmidt process.
;
0111
11
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
== Uψ ( )⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
==
0111
31
, 11
11 ψψ
ψφ
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L5/V1/May 2004/3
⎟⎟⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜⎜⎜
⎝
⎛
=
∴
03
13
13
1
1φ
( ) 11222 , φφψ UU −=
⎟⎟⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜⎜⎜
⎝
⎛
⎟⎟⎠
⎞⎜⎜⎝
⎛−+−
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
−=
03
13
13
1
31
31
31
01
11
⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜
⎝
⎛
−
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
−=
0313
13
1
01
11
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
−=
03
43
23
2
and 3
629
1694
94
2 =++=ψ
⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜
⎝
⎛
−=
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
−==∴
06
26
16
1
03
43
23
2
623
2
22 ψ
ψφ
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L5/V1/May 2004/4
Thus
⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜
⎝
⎛
−=
06
26
16
1
2φ
Finally,
( ) ( ) 22311333 ,, φφφφψ UUU −−=
⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜
⎝
⎛
−⎟⎟⎠
⎞⎜⎜⎝
⎛−
⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜
⎝
⎛
⎟⎟⎠
⎞⎜⎜⎝
⎛−
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
=
06
26
16
1
63
03
13
13
1
36
0132
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
−−
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
−
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
=
012
12
1
0222
0132
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛ −
=
002
12
1
21
21
41
41
3 ==+=ψ
⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜
⎝
⎛ −
=
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛ −
==∴
00
21
21
002
12
1
23
33 ψ
ψφ
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L5/V1/May 2004/5
Thus the required orthonormal basis for ω, the subspace spanned by U1,U2, U3 is φ1, φ2, φ3, where
⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜
⎝
⎛−
=
⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜
⎝
⎛
−=
⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜
⎝
⎛
=
00
21
21
;
06
26
16
1
;
03
13
13
1
321 φφφ
Note that these φi are mutually orthogonal and have, each, ‘length’ one.
We now get back to Hermitian matrices. We had seen that the eigenvalues of a Hermitian matrix are all real; and that the eigenvectors corresponding to district eigenvalues are mutually orthogonal. We can further show the following: (We shall not give a proof here, but illustrate with an example).
Let A be any nxn Hermitian matrix. Let
( ) ( ) ( ) ( ) kak
aaC λλλλλλλ −−−= .....2121 be its characteristic polynomial,
where λ1, λ2, ….., λk are its distinct eigenvalues and a1, ….., ak are their algebraic multiplicities. If ωi is the characteristic subspace corresponding to the eigen value λi ; that is,
{ }xAxx ii λω == :
then it can be shown that dim is ωi = ai.
We then choose any basis for ωi and orthonormalize it by G-S process and get an orthonormal basis for ωi. If we now take all these orthonormal basis vectors for ω1, . . ., ωk and write them as the columns of a matrix P then
P*AP
Will be a diagonal matrix.
Example :
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
−−−
−=
312132
226A
Notice
A* = A1 = A1 = A.
Thus the matrix A is Hermitian.
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L5/V1/May 2004/6
Characteristic Polynomial of A:
312132226
−−−
−−=−
λλ
λλ AI
( )
3121320222
21 2
−−−−−
⎯⎯⎯ →⎯ +
λλλλ
RR
( )312
132021
2−−
−−=λ
λλ
( )350
170021
212
13
2
2 −−−=→
−
+ λλλ
RR
RR
( ) ( )( )[ ]5372 −−−−= λλλ
( )[ ]16102 2 +−−= λλλ
( )( )( 822 −−−= )λλλ
( ) ( 82 2 −−= λλ ) Thus
( ) ( ) ( )82 2 −−= λλλC
∴ λ1 = 2 a1 = 2
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L5/V1/May 2004/7
λ2 = 8 a2 = 1 The characteristic subspaces:
{ }xAxx 2:1 ==ω
( ){ }θ=−= xIAx 2:
i.e. We have to solve
(A – 2I) x = θ
i.e. ⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛=
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
−−−
−
000
112112
224
3
2
1
xxx
2x1 – x2 + x3 = 0
x3 = - 2x1 + x2
arbitraryxxxx
xx
x 21
21
2
1
,;2 ⎟
⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
+−=∴
⎪⎭
⎪⎬
⎫
⎪⎩
⎪⎨
⎧
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
+−==∴ scalarsxx βα
βαβα
ω ,;2
:1
∴ A basis for ωi is
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛=
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
−=
110
;2
01
21 UU
We now orthonormalize this:
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
−==
201
11 Uψ 51 =ψ 1
11 ψ
ψφ =
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L5/V1/May 2004/8
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
−
=∴
52
05
1
1φ
( ) 11222 , φφψ UU −=
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
−⎟⎟⎠
⎞⎜⎜⎝
⎛−−
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛=
52
05
1
52
110
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
−
+⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛=
54
052
110
⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜
⎝
⎛
=
5115
2
530
2530
2511
254
2 ==++=ψ
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
=⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜
⎝
⎛
==∴
301
305
302
5115
2
305
2
22 ψ
ψφ
∴ φ1, φ2 is an orthonormal basis for ω1.
{ }xAxx 8:2 ==ω
( ){ }θ=−= xIAx 8:
So we have to solve
(A-8I) x = θ i.e.
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L5/V1/May 2004/9
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛=
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
−−−−−
−−
000
512152
222
3
2
1
xxx
This yields x1 = -2x2 = 2x3 and therefore the general solution is
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−=
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
−11
2
2
21γ
γ
γγ
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−=∴11
2: 3UBasis
∴ Orthonormalize: only one step:
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−==11
2
33 Uψ
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
−=⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−==
61
61
62
11
2
61
3
33 ψ
ψφ
∴ If
⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜
⎝
⎛
−
−=
61
301
52
61
3050
62
302
51
P
Then
P* = P1 and
;800020002
1*
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛== APPAPP adiagonal matrix.
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L5/V1/May 2004/10
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L6/V1/May 2004/1
VECTOR AND MATRIX NORMS
Consider the space,
,,; 212
12
⎭⎬⎫
⎩⎨⎧
∈⎟⎟⎠
⎞⎜⎜⎝
⎛== Rxx
xx
xR
our ‘usual’ two-dimensional plane. If x = is any vector in this space we
defineits‘usual’ ‘length’ or ‘norm’ as
⎟⎟⎠
⎞⎜⎜⎝
⎛
2
1
xx
22
12 xxx +=
We observe that
(i) 0≥x for every vector x in R2
0≥x if and only if x is θ;
(ii) xx αα = for any scalar α; for any vector x.
(iii) yxyx +≤+ for any two vectors x and y. (The inequality (iii) is usually referred to as the triangle inequality).
We now generalize this idea to define the concept of a norm on Cn or Rn.
The norm on a vector space V is a rule which associates with each vector x in V, a real number x satisfying,
(i) 0≥x for every x ε V and
0≥x if and only if x = θ;
(ii) xx αα = for every scalar α and every vector x in V,
(iii) yxyx +≤+ for every x, y in V.
Examples of Vector Norms on Cn and Rn
Let be any vector x in C
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
=
nx
xx
xM2
1
n (or Rn)
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L6/V1/May 2004/2
We can define various norms as follows:(1)
[ ] 21
1
221
222
212
..... ⎥⎦
⎤⎢⎣
⎡=+++= ∑
=
n
iin xxxxx
(2) ∑=
=+++=n
iin xxxxx
1211
....
In general for 1 ≤ p < ∞ we can define,
(3) pn
i
pip
xx
1
1 ⎭⎬⎫
⎩⎨⎧
= ∑=
If we set p = 2 in (3) we get 2
x as in (1) and if we set p = 1 in (3) we get 1
x as in (2).
(4) { }nxxxx ,.....,,.max 21=∞
All these can be verified to satisfy the conditions (i), (ii) and (iii) required of a norm. Thus these give several types of norms on Cn and Rn.
Example:
(1) Let in R⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
−−=
12
1x 3
Then
( ) 621141
4121
2
1
=++=
=++=
x
x
{ }
( ) 41
41
4444
18121
21,2,1.max
=++=
==∞
x
x
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L6/V1/May 2004/3
(2) Let in C⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
−=
iix2
13
Then
( ) 6141
4121
21
2
1
=++=
=++=
x
x
{ }
( ) 31
31
10121
21,2,1.max
3333
=++=
==∞
x
x
Consider a sequence of vectors in C{ } 1)(
=∞
kkx n (or Rn)
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
=
nk
k
k
k
x
xx
x
)(
2)(
1)(
)(
M
Suppose )(2
1
nn
n
orRC
x
xx
x ∈
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
=M
DEFINITION:
We say that the sequence { })( kx of vectors CONVERGES to the vector x if the
sequence of numbers, { }1)( kx converges to the number x1; { }2
)( kx converges to x2, ….
and { }nkx )( converges to xn i.e.
As k → ∞; x(k)i → xi for every i=1, 2, …., n.
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L6/V1/May 2004/4
Example:
Let
⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜
⎝
⎛
+
−=
11
21
2
)(
k
k
ki
x k be a sequence of vectors in R3.
Let . Here ⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛=
010
x 11)( 01 x
kx k =→=
22)( 121 x
kx k =→−=
323)( 0
11 x
kx k =→
+=
forxx iik →∴ )( I=1,2,3.
xx k →∴ )(
If { })(kx is a sequence of vectors such that in some norm, the sequence of real numbers, xx k −)( converges to the real number 0 then we say that the sequence of vectors
converges to x with respect to this norm. We then write,
xx k ⎯→⎯)(
For example consider the sequence,
⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜
⎝
⎛
+
−=
11
21
1
2
)(
k
k
kx k in R3 as before and,
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛=
010
x
We have
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L6/V1/May 2004/5
( )
⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜
⎝
⎛
+
−=−
11
2
1
2k
k
kxx k
Now
01
12121
)( →+
++=−kkk
xx k
xx k ⎯→⎯∴ 1)(
Similarly
021
1,2,1.max 2)( →=
⎭⎬⎫
⎩⎨⎧
+=−
∞ kkkkxx k
xx k ⎯⎯ →⎯∴ ∞)(
( ) 01
121 21
22222
)( →⎪⎭
⎪⎬⎫
⎪⎩
⎪⎨⎧
+++=−
kkkxx k
xx k ⎯→⎯∴ 2)(
Also,
( ) 01
1211
2
)( →⎪⎭
⎪⎬⎫
⎪⎩
⎪⎨⎧
++⎟
⎠⎞
⎜⎝⎛+=−
p
p
p
pp
k
kkkxx
xx pk ⎯⎯→⎯∴ )( ∞≤≤∀ pp 1;
It can be shown that
“ IF A SEQUENCE { })(kx OF VECTORS IN Cn (or Rn) CONVERGES TO A VECTOR x IN Cn (or Rn) WITH RESPECT TO ONE VECTOR NORM THEN THE SEQUENCE CONVERGES TO x WITH RESPECT TO ALL VECTOR NORMS AND ALSO THE
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L6/V1/May 2004/6
SEQUENCE CONVERGES TO x ACCORDING TO DEFINITION ON PAGE 40. CONVERSELY IF A SEQUENCE CONVERGES TO x AS PER DEFINITION ON PAGE 40 THEN IT CONVERGES WITH RESPECT TO ALL VECTOR NORMS”.
Thus when we want to check the convergence of a sequence of vectors we can choose that norm which is convenient to that sequence.
MATRIX NORMS
Let M be the set of all nxn matrices (real or complex). A matrix norm is a rule, which associates a real number A with each matrix A and satisfying,
(i) 0≥A for all matrices A
0=A if and only if A = On,
(ii) AA αα = for every scalar α and every matrix A,
(iii) BABA +≤+ for all matrices A and B,
(iv) BAAB ≤ for all matrices A and B.
Before we give examples of matrix norms we shall see a method of getting a matrix norm starting with a vector norm.
Suppose . is a vector norm. Then, consider x
Ax (where A is an nxn matrix); for x ≠
θn. This given us an idea to by what proportion the matrix A has distorted the length of x. Suppose we take the maximum distortion as we vary x over all vectors. We get
nx θ≠
max
xAx
a real number. We define
=Anx θ≠
max
xAx
We can show this is a matrix norm and this matrix norm is called the matrix norm subordinate to the vector norm . We can also show that
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L6/V1/May 2004/7
=Anx θ≠
max
xAx
= 1max
=x Ax
For example,
=1
A 1max
1=x 1
Ax
=A 1max
2=x 2
Ax
=A 1max
=∞
x ∞Ax
=A 1max
=p
x pAx
How hard on easy is it to compute these matrix norms? We shall give some idea of computing
1A ,
∞A and
2A for a matrix A.
Let
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
=
nnnn
n
n
aaa
aaaaaa
A
.........................
.....
.....
21
22221
11211
The sum of the absolute values of the entries in the ith column is called the absolute column sum and is denoted by Ci. We have
∑=
=++++=n
iin aaaaaC
1113121111 .....
∑=
=++++=n
iin aaaaaC
1223222122 .....
….. ….. ….. ….. ….. ….. …..
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L6/V1/May 2004/8
∑=
=n
iijj aC
1 ; 1 ≤ j ≤ n
Thus we have n absolute column sums, C2 , C2, ….., Cn.
Let
{ }nCCCC ,.....,,.max 21=
This is called the maximum absolute column sum. We can show that,
{ }nCCCA ,.....,.max 11==
nj ≤≤=1
max⎥⎦
⎤⎢⎣
⎡ ∑=
n
iija
1
For example, if
,423
101321
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
−−−
−=A
then
and C = max. {5, 4, 8} = 8 8413
;4202;5311
3
2
1
=++==++=
=++=
CCC
81=∴ A
Similarly we denote by Ri the sum of the absolute values of the entries in the ith
row ∑=
=+++=n
jjn aaaaR
11112111 .....
∑=
=+++=n
jjn aaaaR
12222212 .....
….. ….. ….. ….. ….. ….
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L6/V1/May 2004/9
∑=
=+++=n
jijiniii aaaaR
121 ..... and define R, the maximum absolute row
sum as, R = max {R1, ….., Rn}
It can be show that,
{ }nRRRA ,.....,max 1==∞
ni ≤≤=1
max
⎭⎬⎫
⎩⎨⎧∑=
n
jija
1
For example, for the matrix
,423
101321
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
−−−
−=A we have
R1 = 1 + 2 + 3 = 6;
R2 = 1 + 0 + 1 =2; and R = max {6, 2, 9}= 9
R3 = 3 + 2 + 4 = 9
9=∴∞
A
The computation of 1A and ∞
A for a matrix are thus fairly easy. However, the
computation of 2
A is not very easy; but somewhat easier in the case of the Hermitian matrix.
Let A be any nxn matrix; and
( ) ( ) ( ) kak
aC λλλλλ −−= L11 , be its characteristic polynomial, where
λ1, λ2, ….., λk are the district characteristic values of A.
Let
{ }kP λλλ ,,,.max 21 K=
This is called the spectral radius of A and is also denoted by spA
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L6/V1/May 2004/10
It can be show that for a Hermitian matrix A,
2A = P = sp
A
For example, for the matrix,
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
−−−
−=
312132
226A
which is Hermitian we found on page 33, the district eigenvalues as λ1 = 2; λ2 = 8
{ 8,2.max== }∴ PAxp = 8
82
==∴sp
AA
If A is any general nxn matrix (not Hermitian) then let B = A* A. Then B* = A* A = B, and hence B is Hermitian and its eigenvalues are real and in fact its eigenvalues are nonnegative. Let the eigenvalues (district) of B be µ1, µ2, ….., µr. Then let
µ = max {µ1, µ2, ….., µr}
We can show that
{ }nA µµµ ,.....,max 12==
If follows from the matrix norm definition subordinate to a vector norm, that
=Anx θ≠
max
xAx
∴ For any x in Cn or Rn , we have, if x ≠ θn
≤x
Axnx θ≠
maxA
xAx
=
and therefore
xAAx ≤ for all x ≠ θn
But this is obvious for x = θn
Thus if A is a matrix norm subordinate to the vector norm x then
Numerical analysis /Eigenvalues and Eigenvectors Lecture notes
Vittal rao/IISc.Bangalore M3/L6/V1/May 2004/11
xAAx ≤
for every vector x in Cn (or Rn).
Numerical analysis/ Computations of eigenvalues Lecture notes
Vittal rao/IISc.Bangalore M4/L1/V1/May2004/1
COMPUTATION OF EIGEN VALUES
In this section we shall discuss some standard methods for computing the eigenvalues of an nxn matrix. We shall also briefly discuss some methods for computing the eigenvectors corresponding to the eigenvalues.
We shall first discuss some results regarding the general location of the eigenvalues.
Let A = (aij) be an nxn matrix; and let λ1, λ2, ….., λn be its eigenvalues (including multiplicities). We defined
{ }nxpAP λλλ ,.....,,max 21==
Thus if we draw a circle of radius P about the origin in the complex plane, then all the eigenvalues of A will lie on or inside this closed disc. Thus we have
(A) If A is an nxn matrix then all the eigenvalues of A lie in the closed disc { }P≤λλ : in the complex plane.
This result give us a disc inside which all the eigenvalues of A are located. However, to locate this circle we need P and to find P we need the eigenvalues. Thus this result is not practically useful. However, from a theoretical point of view, this suggests the possibility of locating all the eigenvalues in some disc. We shall now look for other discs which can be easily located and inside which the eigenvalues can all be trapped.
Let A be any matrix norm. Then it can be shown that AP ≤ . Thus if we draw a disc
of radius A and origin as center then this disc will be at least as big as the disc given in (A) above and hence will trap all the eigenvalues. Thus, the idea is to use a matrix norm, which is easy to compute. For example we can use
∞A or
1A which are easily
computed as MARS or MACS respectively. Thus we have,(B) If A is an nxn matrix then all its eigenvalues are trapped in the closed disc { }
∞≤ Aλλ : or the disc
{ }1
: A≤λλ . (The idea is to use ∞
A if it is smaller than 1
A if it is smaller than
∞A ).
COROLLORY
(C) If A is Hermitian, all its eigenvalues are real and hence all the eigenvalues lie in the intervals,
{ }pp ≤≤− λλ : by (A)
Numerical analysis/ Computations of eigenvalues Lecture notes
Vittal rao/IISc.Bangalore M4/L1/V1/May2004/2
{ }{ }
11:
:
AA
AA
≤≤−
≤≤−∞∞
λλ
λλ by (B).
Example 1:
Let ⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−
−=
021321211
A
Here ‘Row sums’ are P1 = 4 P2 = 6 P3 = 3
6==∴∞
MARSA
Thus the eigenvalues are all in the disc ; { }6: ≤λλ
The ‘Column sums’ are C1 = 3, C2 = 5, C3 = 5.
51
==∴ MACSA
∴ The eigenvalues are all in the disc, { },5: ≤λλ
In this example ;651
=<=∞
AA and hence we use 1
A and get the smaller disc
{ ,5: ≤λλ } inside which all eigenvalues are located.
The above results locate all the eigenvalues in one disc. The next set of results try to isolate these eigenvalues to some extent in smaller discs. These results are due to GERSCHGORIN.
Let A = (aij) be an nxn matrix.
The diagonal entries are
;111 a=ξ ;222 a=ξ ….., ;nnn a=ξ
Now let Pi denote the sum of the absolute values of the off-diagonal entries of A in the ith row.
iniiiiiii aaaaaP ++++++= +− .......... 1121
Numerical analysis/ Computations of eigenvalues Lecture notes
Vittal rao/IISc.Bangalore M4/L1/V1/May2004/3
Now consider the discs:
{ }11111 ::;: PradiusPCentreG ≤−ξλλξ
{ }222212 ::;: PradiusPCentreG ≤−ξλλξ
….. ….. ….. ….. ….. ….. ….
and in general
{ }iiiii PradiusPCentreG ≤−ξλλξ ::;:
Thus we get n discs G1, G2, ….., Gn. These are called the GERSCHGORIN DISCS of the matrix A.
The first result of Gerschgorin is the following:
(D) Every eigenvalue of A lies in one of the Gerschgorin discs.
Example 2:
Let ⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
−
−=
513140011
A
The Gerschgorin discs are found as follows:
ξ1 = (1,0) ; ξ2 = (4,0) ; ξ3 = (-5,0)
P1 = 1 ; P2 = 1 ; P3 = 4
G1 : Centre (1,0) radius 1
G2 : Centre (4,0) radius 1
G3 : Centre (-5,0) radius 4.
Numerical analysis/ Computations of eigenvalues Lecture notes
Vittal rao/IISc.Bangalore M4/L1/V1/May2004/4
G1 (1,0) G2 (4,0)
G3 (-5,0)
Thus every eigenvalue of A must lie in one of these three discs.
Example 3:
Let ⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
−=
2035.15.0101
1410A
(It can be shown that the eigenvalues are exactly λ1 = 8, λ2 = 12, λ3 = 20).
Now for this matrix we have,
ξ1 = (10,0) ξ2 = (10,0) ξ3 = 20
P1 = 5 P2 = 1.5 P3 = 4.5
Thus we have the three Gerschgorin discs
{ }{ }{ }5.420:
5.110:
510:
3
2
1
≤−=
≤−=
≤−=
λλ
λλ
λλ
G
G
G
Numerical analysis/ Computations of eigenvalues Lecture notes
Vittal rao/IISc.Bangalore M4/L1/V1/May2004/5
G1 G3
Thus all the eigenvalues of A are in these discs. But notice that our exact eigenvalues are 8,12 and 20. Thus no eigenvalue lies in G2; and one eigenvalue lie in G3 (namely 20) and two lie in G1 (namely 8 and 12).
Example 4:
Let ⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛=
501021101
A
Now,
ξ1 = (1,0) ξ2 = (2,0) ξ3 = (5,0) P1 = 1 P2 = 1 P3 = 1
The Gerschgorin discs are
{ }{ }{ }15:
12:
11:
3
2
1
≤−=
≤−=
≤−=
λλ
λλ
λλ
G
G
G
Numerical analysis/ Computations of eigenvalues Lecture notes
Vittal rao/IISc.Bangalore M4/L1/V1/May2004/6
G1 (1,0) G2 (2,0) G3 (5,0)
Thus every eigenvalue of A must lie in one of these three discs.
In example 2, all the Gerschgorin discs were isolated; and in examples 3 and 4 some discs intersected and others were isolated. The next Gerschgoin result is to identify the location of the eigenvalues in such cases.
(E) If m of the Gerschgorin discs intersect to form a common connected region and the remaining discs are isolated from this region then exactly m eigenvalues lie in this common – region. In particular if Gerschgorin disc is isolated from all the rest then exactly one eigenvalue lies in this disc.
Thus in example 2 we have all three isolated discs and thus each disc will trap exactly one eigenvalue.
In example 3; G1and G2 intersected to form the connected (shaded) region and this is isolated from G3. Thus the shaded region has two eigenvalues and G3 has one eigenvalue.
In example 4, G1and G2 intersected to form a connected region (shaded portion) and this is isolated from G3. Thus the shaded portion has two eigenvalues and G3 has one eigenvalue.
REMARK:
In the case of Hermitian matrices, since all the eigenvalues are real, the Gerschgorin discs, { } { iiiiii PPaG ≤−=≤−= ξλλλλ :: } can be replaced by the Gerschgorin intervals,
{ } { }iiiiiii PPPG +≤≤−=≤−= ξλξλξλλ ::
Example 5:
Numerical analysis/ Computations of eigenvalues Lecture notes
Vittal rao/IISc.Bangalore M4/L1/V1/May2004/7
Let ⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜
⎝
⎛
−−
−=
2101051111
A
Note A is Hermitian. (In fact A is real symmetric)
Here; ξ1 = (1,0) P1 = 2
ξ2 = (5,0) P2 = 1
ξ3 = (-1/2,0) P3 = 1 Thus the Gerschgorin intervals are
G1 : -1≤ λ ≤ 3
G2 : 4 ≤ λ ≤ 6
G3 : -3/2 ≤ λ ≤ ½
-2 -1 0 1 2 3 4 5 6
G1 G2
G3
Note that G1 and G3 intersect and give a connected region, -3/2 ≤ λ ≤ 3; and this is isolated from G2 : 4 ≤ λ ≤ 6. Thus there will be two eigenvalues in –3/2 ≤ λ ≤ 3 and one eigenvalue in 4 ≤ λ ≤ 6. All the above results (A), (B), (C), (D), and (E) give us a location of the eigenvalues inside some discs and if the radii of these discs are small then the centers of these circles give us a good approximations of the eigenvalues. However if these discs are of large radius then we have to improve these approximations substantially. We shall now discuss this aspect of computing the eigenvalues more accurately. We shall first discuss the problem of computing the eigenvalues of a real symmetric matrix.
Numerical analysis/ Computations of eigenvalues Lecture notes
Vittal rao/IISc.Bangalore M4/L2/V1/May2004/1
COMPUTATION OF THE EIGENVALUES OF A REAL SYMMETRIC MATRIX
We shall first discuss the method of reducing the given matrix to a similar tridiagonal matrix and then computing the eigenvalues of a real symmetric tridiagonal matrix. Thus the process of determining the eigenvalues of A = (aij), a real symmetric method involves two steps:
STEP 1:
Find a real symmetric tridiagonal matrix T which is similar to A.
STEP 2:
Find the eigenvalues of T. (The eigenvalues of A will be same as those of T since A and T are similar).
We shall first discuss step 2.
Numerical analysis/ Computations of eigenvaues Lecture notes
VittalRao/IISc Bangaolre M4/L3/V1/May 2004/1
DETERMINATION OF THE EIGENVALUES OF A REAL SYMMETRIC TRIDIAGONAL MATRIX
Let
⎟⎟⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜⎜⎜
⎝
⎛
=
−
−−−
nn
nnn
abbab
babbab
ba
T
1
112
332
221
11
0..........000..........0
...................................0.....000.....000....000
be a real symmetric tridiagonal matrix.
Let us find Pn (λ) = det [T - λI]
λ
λ
λλ
−−
−−−
−
−−
=
nn
nnn
abbab
babba
1
112
221
11
0..........00.....0
..............................0.....00..........0
The eigenvalues of T are precisely the roots of Pn (λ) = 0
(Without loss of generality we assume bi ≠ 0 for all i. For if bi = 0 for some i then the above determinant reduces to two diagonal blocks of the same type and thus the problem reduces to that of the same type involving smaller sized matrices).
We define Pi (λ) to be the ith principal minor of the above determinant. We have
( )( )( ) ( ) ( ) ( )λλλλ
λλλ
212
1
11
0 1
−−− −−=
−==
iiiii PbPaP
aPP
…….. (I)
What we are interested in finding the zeros of ( )λnP . To do this we analyse the polynomials ( ) ( ) ( )λλλ nPPP ,.....,, 10 .
Let C be any real number. Compute ( ) ( ) ( )CPCPCP n,.....,, 10 (which can be calculated recursively by (I)). Let N (C) denote the agreements in sign between two consecutive in
Numerical analysis/ Computations of eigenvaues Lecture notes
VittalRao/IISc Bangaolre
the above sequence of values, ( ) ( ) ( )CPCPCP n,.....,, 10 . [If for some i, , we take its sign to be the same as that of
( ) 0=CPi
( )CPi 1− ]. Then we have
(F) There are exactly N (C) eigenvalues of T that are ≥ C.
Example:
If for an example we have an 8 x 8 matrix T (real symmetric tridiagonal)
giving use to,
( )( )( )( )( )( )( )( )( ) 21
4101
1161
2131
2111
8
7
6
5
4
3
2
1
0
−===−=
=−=−=
==
PPPPPPPPP
Here
( ) ( )( ) ( )( ) ( )1,1
1,11,1
65
32
10
PPPPPP
(Because since P6 (1) = 0 we h
Thus three pairs of sign agreeeigenvalues of T greater than o
It is this idea of result (F) that repeated applications of (F) thmeans of an example.
agree in sign
agree in sign
M4/L3/V1/May 2004/2
ave to take its sign as the same as that of P5 (1).
ments are achieved. Thus N (C) = 3; and there will be 3 r equal to 1; and the remaining 5 eigen values are < 1.
will be combined with (A), (B), (C), (D) and (E) and clever at locate the eigenvalues of T. We now explain this by
Numerical analysis/ Computations of eigenvaues Lecture notes
VittalRao/IISc Bangaolre M4/L3/V1/May 2004/3
Example 7:
Let
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
−−
−=
31001240
04120021
T
Here we have
Absolute Row sum 1 = 3
Absolute Row sum 2 = 7
Absolute Row sum 3 = 7
Absolute Row sum 4 = 4
and therefore,
7==∞
MARST
(Note since T is symmetric we have MARS = MACS and therefore TTT ==∞1
). Thus by our result (C) we have that the eigenvalues are all in the interval –7 ≤ λ ≤ 7
[ ]
-7 -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7
Now the Gerschgorin (discs) intervals are as follows:
G1 : Centre 1 radius : 2 ∴ G1 : [-1, 3]
G2 : Centre -1 radius : 6 ∴ G2 : [-7, 5]
G3 : Centre 2 radius : 5 ∴ G3 : [-3, 7]
G4 : Centre 3 radius : 1 ∴ G4 : [-2, 4]
Numerical analysis/ Computations of eigenvaues Lecture notes
VittalRao/IISc Bangaolre M4/L3/V1/May 2004/4
G3
G1
[ ]
-7 -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7
G4 G2
We see that G1, G2, G3 and G4 all intersect to form one single connected region [-7, 7]. Thus by (E) there will be 4 eigenvalues in [-7, 7]. This gives therefore the same information as we obtained above using (C). Thus so far we know all eigenvalues are in [-7, 7]. Now we shall see how we use (F) to locate the eigenvalues.
First of al let us see how many eigenvalues will be ≥ 0. Let C = 0. Find N (0) and we will get the number of eigenvalues ≥ 0 to be N (0).
Now
λλ
λλ
λ
−−−−
−−−
=−
31001240
04120021
IT
( ) 10 =λP ( ) λλ −= 11P
( ) ( ) ( ) ( )( ) ( ) ( ) ( )( ) ( ) ( ) ( )λλλλ
λλλλλλλλ
234
123
012
316241
PPPPPPPPP
−−=−−=−+−=
Now, we have,
( )( )( )( )( ) 730
26050
1010
4
3
2
1
0
−=−=−=
==
PPPPP
Numerical analysis/ Computations of eigenvaues Lecture notes
VittalRao/IISc Bangaolre M4/L3/V1/May 2004/5
We have
as three consecutive pairs having sign agreements.
( ) ( )( ) ( )( ) ( )0,0
0,00,0
43
32
10
PPPPPP
( ) 30 =∴ N
∴ Three are 3 eigenvalues ≥ 0 and ∴ one eigenvalue < 0.
i.e. there are eigenvalues in [0, 7]and there is 1 eigenvalue in [-7, 0]
One eigenvalue 3 eigenvalues
-7 -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7
Fig.1
Let us take C = -1 and calculate N (C). We have
( )( )( )( )( ) 1881
48141
2111
4
3
2
1
0
−=−−=−−=−
=−=−
PPPPP
Again we have N (-1) = 3. ∴ There are 3 eigenvalues ≥ -1 compare this with figure1. We get
One eigenvalue 3 eigenvalues
Numerical analysis/ Computations of eigenvaues Lecture notes
VittalRao/IISc Bangaolre M4/L3/V1/May 2004/6
-7 -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7
(Fig.2)
Let us take the mid point of [-7, -1] in, which the negative eigenvalue lies.
So let C = -4.
P0 (-4) = 1 Again there are three pairs of sign agreements.
P1 (-4) = 5 ∴ N (-4) = 3. ∴ There are 3 eigenvalues ≥ -4.
P2 (-4) = 11 Comparing with fig. 2 we get
P3 (-4) = -14
P4 (-4) = -109
that the negative eigenvalue is in [-7, -4] ………..(*)
Let us try mid pt. C = -5.5
We have
P0 (-5.5) = 1
P1 (-5.5) = + 6.5 ∴ N (-5.5) = 4. ∴ 4 eigenvalues ≥ -5.5.
P2 (-5.5) = 25.25 Combining this with (*) and fig. 2 we get
P3 (-5.5) = 85.375 that negative eigenvalue is in [-5.5 – 4].
P4 (-5.5) = 683.4375
We again take the mid pt. C and calculate N (C) and locate in which half of this interval does this negative eigenvalue lie and continue this bisection process until we trap this negative eigenvalue in as small an interval as necessary.
Now let us look at the eigenvalues ≥ 0. We have from fig. 2 three eigenvalues in [0, 7]. Now let us take C = 1
P0 (1) = 1
P1 (1) = 0 ∴ N (1) = 3
Numerical analysis/ Computations of eigenvaues Lecture notes
VittalRao/IISc Bangaolre M4/L3/V1/May 2004/7
P2 (1) = - 4 ∴ all the eigenvalues are ≥ 1 ………….. (**)
P3 (1) = - 4
P4 (1) = - 4
C = 2
P0 (2) = 1
P1 (2) = -1 ∴ N (2) = 2∴ There are two eigenvalues
P2 (2) = - 1 ≥ 2. Combining this with (**) we get one
P3 (2) = 16 eigenvalue in [1, 2) and two in [2, 7].
P4 (2) = 17
C = 3
P0 (3) = 1 ∴ N (3) = 1 ∴ one eigenvalue ≥ 3
P1 (3) = -2 Combining with above observation we get
P2 (3) = 4 one eigenvalue in [1, 2)
P3 (3) = 28 one eigenvalue in [2, 3)
P4 (3) = - 4 one eigenvalue in [3, 7)
Let us locate the eigenvalue in [3, 7] a little better. Take C = mid point = 5
P0 (5) = 1
P1 (5) = - 4 ∴ N (5) = 1
P2 (5) = 20 ∴ this eigenvalue is ≥ 5
P3 (5) = 4
P4 (5) = -28
∴ This eigenvalue is in [5, 7]
Let us take mid point C = 6
P0 (6) = 1
P1 (6) = - 5 ∴ N (6) = 0
Numerical analysis/ Computations of eigenvaues Lecture notes
VittalRao/IISc Bangaolre M4/L3/V1/May 2004/8
P2 (6) = 31 ∴ No eigenvalue ≥ 6
P3 (6) = - 44 ∴ the eigenvalue is in [5, 6)
P4 (6) = 101
Thus combining all, we have,
one eigenvalue in [-5.5, -4)
one eigenvalue in [1, 2)
one eigenvalue in [2, 3)
one eigenvalue in [5, 6)
Each one of these locations can be further narrowed down by the bisection applied to each of these intervals.
We shall now discuss the method of obtaining a real symmetric tridiagonal T similar to a given real symmetric matrix A.
Numerical analysis/Computations of eigenvaues Lecture notes
Vittal rao/IISc.Bangalore M4/L4/V1/May 2004/1
TRIDIAGONALIZATION OF A REAL SYMMETRIC MATRIX
Let A = (aij) be a real symmetric nxn matrix. Our aim is to get a real symmetric tridiagonal matrix T such that T is similar to A. The process of obtaining this T is called the Givens – Householder scheme. The idea is to first find a reduction process which annihilates the off – tridiagonal matrices in the first row and first column of A and repeatedly use this idea. We shall first see some preliminaries.
Let be a real nx1 vector. Then (U ≠ θ
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
=
nU
UU
UM
2
1
n)
H = UUt is an nxn real symmetric matrix. Let α be a real number (which we shall suitably choose) and consider
( )IUUIHIP t ...........αα −=−=
We shall choose α such that P is its own inverse. (Note that Pt = P). So we need
P2 = I
i.e.
(I - α H) (I - α H) = I
i.e.
(I - α UUt) (I - α UUt) = I
I – 2 α UUt + α2 UUt UUt = I
So we choose α such that
α2 UUt UUt = 2 α UUt
Obviously, we choose α ≠ 0. Because otherwise we get P = I; and we don’t get any new transformation.
∴ We need
Numerical analysis/Computations of eigenvaues Lecture notes
Vittal rao/IISc.Bangalore M4/L4/V1/May 2004/2
α UUt UUt = 2. UUt
But UtU = U21 + U2
2 + ….. + U2n =
2U is a real number ≠ 0 and thus we have
α (Ut U). UUt = 2 UUt
and hence
)..(..........2 IIUU t=α
Thus if we U is an nx1 vector and different from θn and α is as in (II) then P defined as
)....(.......... IIIUUIP tα−=
is such that
)....(..........1 IVPPP t −==
Now we go back to our problem of tridiagonalization of A. Our first aim is to find a P of the form (IV) such that PAPAPPt = has off tridiagonal entries in 1st row and 1st column as zero. We can choose the P as follows:
Let ( )Vaaas n ..................... 12
312
2122 +++=
(the sum of the squares of the entries below the 1st diagonal entry in A)
Let s = nonnegative square root of s2.
Let
⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜
⎝
⎛+
=
1
31
2121 .sgn0
na
aasa
UM
………. (VI)
Thus U is the same as the 1st column of A except that the 1st component is taken as 0 and second component is a variation of the second component in the 1st column of A. All others are same as 1st column of A.
Then
Numerical analysis/Computations of eigenvaues Lecture notes
Vittal rao/IISc.Bangalore M4/L4/V1/May 2004/3
1
2
−
⎥⎦
⎤⎢⎣
⎡=
UU t
α
( )[ ] 11
241
231
222121 2/......sgn
−+++++= naaaasa
[ ] 11
231
221
221
2 2/.....2 −+++++= naasasa
( )[ ]{ } 121
21
231
221
2 2/2..... −+++++= assaaa n
21
2
1ass +
=
212
1ass +
=∴α (VII)
Thus if α is as in (VII) and U is as in (VI) where s is as in (V) then
P = I - α UUt
is s.t. P = Pt = P-1, and it can be shown that
A2 = PA1P = PAP (i.e let A1 = A)
is similar to A and has off tridiagonal entries in 1st row and 1st column as 0.
Now we apply this procedure to the matrix obtained by ignoring 1st column and 1st row of A2.
Thus we now choose
22
422
3222 ..... naaas +++=
(where now aij denote entries of A2)
(i.e. s2 is sum of squares of the entries below second diagonal entry of A2)
s = Positive square root of s2
Numerical analysis/Computations of eigenvaues Lecture notes
Vittal rao/IISc.Bangalore M4/L4/V1/May 2004/4
⎟⎟⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜⎜⎜
⎝
⎛
+=
2
42
3232 ).(00
na
asasigna
U
M
322
1ass +
=α
P = I - α UUt
Then
A3 = PA2P
has off tridiagonal entries 1n 1st, 2nd rows and columns as zero. We proceed similarly and annihilate all off tridiagonal entries and get T, real symmetric tridiagonal and similar to A.
Note: For an nxn matrix we get tridiagonalization in n – 2 steps.
Example:
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
=
4211241111541145
A
A is a real symmetric matrix and is 4 x 4. Thus we get tridiagonalization after (4 – 2) i.e. 2 steps.
Step 1:
24264.418
18114 2222
==
=++=
s
s
( )( ) 97056.341
424264.41811
212 =
+=
+=
assα = 0.02860
Numerical analysis/Computations of eigenvaues Lecture notes
Vittal rao/IISc.Bangalore M4/L4/V1/May 2004/5
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛+
=
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛+
=
1124264.440
.sgn0
41
31
2121
aa
asaU
With this α, U, we get
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
−−−−−−−
=−=
97140.002860.023570.0002860.097140.023570.0023570.023570.094281.000001
tUUIP α
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
−−
−−−−
==
5.35.1105.15.31011624264.4
0024264.45
2 PAPA
Step 2
( ) ( )41421.12
211 222
==
=+−=
s
s
( )( ) 29289.041421.3
1141421.12
11
322 ==
+=
+=
assα
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
−−
=
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
−−−
=
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
+=
141421.200
141421.11
00
.sgn00
42
3232
aasa
U
Numerical analysis/Computations of eigenvaues Lecture notes
Vittal rao/IISc.Bangalore M4/L4/V1/May 2004/6
P = I - α UUt
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
−−−
=
70711.070711.00070711.070711.00000100001
A3 = PA2P
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛−
−
=
20000541421.10041421.1624264.40024264.45
which is tridiagonal.
Thus the Givens – Householder scheme for finding the eigenvalues involves two steps, namely,
STEP 1: Find a tridiagonal T (real symmetric) similar to T (by the method
described above)
STEP 2: Find the eigenvalues of T (by the method of sturm sequences and bisection described earlier)
However, it must be mentioned that this method is used mostly to calculate the eigenvalue of the largest modulus or to sharpen the calculations done by some other method. If one wants to calculate all the eigenvalues at the same time then one uses the Jacobi iteration which we now describe.
Numerical analysis/ Computations of eigenvaues Lecture notes
Vittal rao/IISc.Bangalore M4/L5/V1/May 2004/1
JACOBI ITERATION FOR FINDING EIGENVALUES OF A REAL SYMMETRIC MATRIXSome Preliminaries:
Let be a real symmetric matrix. ⎟⎟⎠
⎞⎜⎜⎝
⎛=
2212
1211
aaaa
A
Let ; (where we choose ⎟⎟⎠
⎞⎜⎜⎝
⎛ −=
θθθθ
CosCos
Psin
sin4
πθ ≤ for
purposes of convergence of the scheme) Note
and ⎟⎟⎠
⎞⎜⎜⎝
⎛−
=θθθθ
CosCos
P t
sinsin
IPPPP tt ==
Thus P is an orthogonal matrix. Now
⎟⎟⎠
⎞⎜⎜⎝
⎛ −⎟⎟⎠
⎞⎜⎜⎝
⎛⎟⎟⎠
⎞⎜⎜⎝
⎛−
==θθθθ
θθθθ
cossinsincos
cossinsincos
2212
12111
aaaa
APPA t
⎟⎟⎠
⎞⎜⎜⎝
⎛+−++−+
⎟⎟⎠
⎞⎜⎜⎝
⎛−
=θθθθθθθθ
θθθθ
cossinsincoscossinsincos
cossinsincos
22122212
12111211
aaaaaaaa
( ) ( )( ) ( ) ⎟⎟
⎠
⎞⎜⎜⎝
⎛
+−−++−−++−++
=θθθθθθθθθθθθθθθθ
22212
211
22122211
22122211
22212
211
coscossin2sinsincoscossinsincoscossinsincossin2cos
aaaaaaaaaaaa
Thus if we choose θ such that,
( ) ( ) 0sincoscossin 22122211 =−++− θθθθ aaa . . . (I)
We get the entries in (1,2) position and (2,1) position of A1 as zero.
(I) gives
( ) 02cos2sin2 12
2211 =+⎟⎠⎞
⎜⎝⎛ +− θθ aaa
θθ 2sin2
2cos 221112
aaa −=⇒
Numerical analysis/ Computations of eigenvaues Lecture notes
Vittal rao/IISc.Bangalore M4/L5/V1/May 2004/2
( )( )
2211
221112
2211
12 sgn222tanaa
aaaaa
a−
−=
−=⇒ θ
βα
= . . . . . (II)
Where )sgn(2 221112 aaa −=α . . . . . (III)
2211 aa −=β . . . . . (IV)
θθ 2tan12sec 22 +=∴
2
2
1βα
+= from (II)
2
22
ββα +
=
22
22 2cos
βαβθ+
=∴
22
2
221cos22cos
βα
βθβα
βθ+
=−⇒+
=∴
⎥⎥⎦
⎤
⎢⎢⎣
⎡
++=⇒
221
21cos
βα
βθ . . . . . . . (V)
and
22
22 12cos12sincossin2
βαβθθθθ+
−=−==
2222
2
βα
αβα
α
+=
+=
22cos2
sinβαθ
αθ+
=∴ . . . . . .(VI)
Numerical analysis/ Computations of eigenvaues Lecture notes
Vittal rao/IISc.Bangalore M4/L5/V1/May 2004/3
(V) and (VI) give sinθ, cosθ and if we choose
⎟⎟⎠
⎞⎜⎜⎝
⎛ −=
θθθθ
cossinsincos
P with these values of cosθ, sinθ, then
PtAP = A1 has (2,1) and (1,2) entries as zero.
We now generalize this idea.
Let A = (aij) be an nxn real symmetric matrix.
Let 1 ≤ g < p < n. (Instead of (1,2) position above choose (q, p) position)
Consider,
)sgn(2 ppqqqp aaa −=α . . . . . . . (A)
ppqq aa −=β . . . . . . . (B)
⎥⎥⎦
⎤
⎢⎢⎣
⎡
++=
221
21cos
βα
βθ . . . . . . . (C)
22cos21sin
βα
αθ
θ+
= . . . . . . . (D)
q p
Numerical analysis/ Computations of eigenvaues Lecture notes
Vittal rao/IISc.Bangalore
⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜
⎝
⎛
−=
1
cossin
1sincos
11
O
O
O
θθ
θθP
then A1 = Pt AP has the entries in (q, p) position and (p, q) pos
In fact A1 differs from A only in qth row, pth row and qth columbe shown that these new entries are
θθ
θθ
cossin
sincos1
1
piqipi
piqiqi
aaa
aaa
+−=
+= i ≠ q, p (qth row
θθ
θθ
cossin
sincos1
1
ipiqip
ipiqiq
aaa
aaa
+−=
+= i ≠ q, p (qth column p
.0
ccossin2sin
scossin2cos
11
21
21
==
+−=
++=
pqqp
ppqpqqpp
ppqpqqqq
aa
aaaa
aaaa
θθθ
θθθ
Now the Jacobi iteration is as follows.
Let A = (aij) be nxn real symmetric.
q
p
M4/L5/V1/May 2004/4
ition as zero.
n and pth column and it can
pth row) . .(E)
th column) . .(F)
os
in2
2
θ
θ
. . . (G)
Numerical analysis/ Computations of eigenvaues Lecture notes
Vittal rao/IISc.Bangalore M4/L5/V1/May 2004/5
Find 1 ≤ g < p ≤ n such that qpa is largest among the absolute values of all the off diagonal entries in A.
For this q, p find P as above. Let A1 = Pt AP. A1 can be obtained as follows:
All rows of A1 are same as A except qth row,
Column
pth row, qth column, pth column which are obtained from (E), (F), (G).
Now A1 has 0 in (q, p), (p, q) position.
Replace A by A1 and repeat the process. The process converges to a diagonal matrix the diagonal entries of which give the eigenvalues of A.
Example:
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
−−−
=
3241242242931237
A
Entry with largest modulus is at (2, 4) position.
∴ q = 2, p = 4.
( ) ( ) 244422 .sgn2.sgn2 aaaaaa qpppqq −=−=α
( )( )( ) .8412 ==
639 =−=−= ppqq aaβ
10022 =+∴ βα ; 1022 =+ βα
⎥⎥⎦
⎤
⎢⎢⎣
⎡
⎟⎟
⎠
⎞
⎜⎜
⎝
⎛
++=∴
221
21cos
βα
βθ
Numerical analysis/ Computations of eigenvaues Lecture notes
Vittal rao/IISc.Bangalore M4/L5/V1/May 2004/6
89442.08.054
1061
21
===⎟⎠⎞
⎜⎝⎛ +=
108
)89442.0(21
cos21sin
22=
+=
βαα
θθ = 0.44721
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛−
=∴
89442.0044721.000100
44721.0089442.000001
P
A1 = PtAP will have a124 = a1
42 = 0.
Other entries that are different from that of A are a121, a1
22, a123 ; a1
41, a142, a1
43, a144 ; (of
course by symmetric corresponding reflected entries also change).
We have,
1305.3sincos 4121211 =+= θθ aaa
44721.0cossin 4121411 −=+−= θθ aaa
89443.0sincos 4323231 −=+= θθ aaa
68328.2cossin 4323431 =+−= θθ aaa
11sincossin2cos 24424
22222
1 =++= θθθθ aaaa
1coscossin2sin 24424
22244
1 =+−= θθθθ aaaa
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
−−−
−−
=∴
00000.168328.2044721.068328.2489443.020000.089443.0111305.344721.021305.37
1A
Now we repeat the process with this matrix.
The largest absolute value is at (1, 2) position.
∴ q = 1, p = 2.
Numerical analysis/ Computations of eigenvaues Lecture notes
Vittal rao/IISc.Bangalore M4/L5/V1/May 2004/7
441172211 =−=−=−=−= aaaa ppqqβ
( ) ( )( )11305.32sgn2 −=−= ppqqgp aaaα = - 6.2610.
42968.7
200121.5522
22
=+
=+
βα
βα
87704.0121cos
22=
⎥⎥⎦
⎤
⎢⎢⎣
⎡
++=
βα
βθ ;
48043.0cos21sin
22−=
+=
βα
αθ
θ
∴ The entries that change are
71484.12coscossin2sin
28516.5sincossin2cos
21485.0cossin
39222.0sincos
17641.0cossin
18378.2sincos
0
22212
21122
1
22212
21111
1
2414241
2414141
2313231
2313131
211
121
=+−=
=++=
−=+−=
−=+=
=+−=
=+=
==
θθθθ
θθθθ
θθ
θθ
θθ
θθ
aaaa
aaaa
aaa
aaa
aaa
aaa
aa
and the new matrix is
Numerical analysis/ Computations of eigenvaues Lecture notes
Vittal rao/IISc.Bangalore M4/L5/V1/May 2004/8
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
−−−
−−
168328.221485.039222.068328.2417641.018378.221485.017641.071484.12039222.018378.2028516.5
Now we repeat with q = 3, p = 4 and so on.
And at the 12th step we get the diagonal matrix
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
−09733.2000
060024.5000071986.12000078305.5
giving eigenvalues of A as 5.78305, 12.71986, -5.60024, 2.09733.
Note: At each stage when we choose (q, p) position and apply the above transformation to get new matrix A1 then sum of squares of off diagonal entries of A1 will be less than that of A by 2a2
qp.
Numerical analysis/Computations of eigenvaues Lecture notes
The Q R decomposition: Let A be an nxn real nonsingular matrix. Then we can find an orthogonal matrix Q and an upper triangular matrix R (with rii >0) such that A=QR called the QR decomposition of A. The Q and R are found as follows: Let a(1) ; a(2) ; ……. , a(n) be the columns of q(1) ; q(2) ; ……… , q(n) be the columns of Q r(1) , r(2) , ………., r(n) be the columns of R. Note: Since Q is Hermitian we have ( ) ( ) ( ) ( )Aqqq n ...................1
22
2
2
1 ====
( ) ( )( ) 0, =ji qq if i ≠ j …………….. (B). and since R is upper triangular we have
( ) ( )Cr
rr
r ii
i
i
i ......................
0
0
2
1
⎟⎟⎟⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜⎜⎜⎜
⎝
⎛
=
M
M
Also the ith column of QR is; Qr(i) and ∴ ith column of
( ) ( ) ( ) ( )DqrqrqrQR iiiii .........................2
21
1 +++= We want A = QR. Comparing 1st column on both sides we get a(1) = QR ’s first column = Qr(1)
= r11q(1) by (D)
( ) ( ) ( )2
1112
1112
1 qrqra ==∴
= r11 ∵ r11 > 0 and ( )Abyq 1
2
1 =
Vittal rao/IISc.Bangalore M4/L6/L1/May2004/1
Numerical analysis/Computations of eigenvaues Lecture notes
( )
2
111 ar =∴ and ( ) ( ) ( )Ea
rq .................1 1
11
1 =
giving 1st columns of R and Q. Next comparing second columns on both sides we get a(2) = Qr(2) = r12 q(1) +r22 q(2) ……….. (*) Therefore from (*) we get
( ) ( )( ) ( ) ( )( ) ( ) ( )( )1222
1112
12 ,,, qqrqqrqa += ( ) ( )( ) ( ) ( )Abyqqqr 1, 2
211112 === Q
( ) ( )( ) ( )Bbyqqand 0, 12 =
( ) ( )( ) ( )Fqar ............, 1212 =∴
∴ (*) gives
( ) ( ) ( )
( ) ( ) ( )
( ) ( ) ( )
( ) ( ) ( )[ ] ( )Hqrar
q
and
Gqrar
qraqrand
qraqr
......................1
....................
112
2
22
2
2
112
222
2
112
2
2
222
112
2222
−=
−=∴
−=∴
−=
(F), (G), (H) give 2nd columns of Q and R. We can proceed having got the first i - 1 columns of Q and R we get ith columns of Q and R as follows:
( ) ( )( ) ( ) ( )( ) ( ) ( )( )11
212
11 ,...,,.........,;, −
− === iiiii
ii qarqarqar
( ) ( ) ( ) ( )
2
11
22
11 ....... −
−−−−= iiiii
iii qrarqrar
( ) ( ) ( ) ( ) ( )[ ]1
12
21
1 .......... −−−−−= i
iiii
ii
i qrqrqrariq
Vittal rao/IISc.Bangalore M4/L6/L1/May2004/2
Numerical analysis/Computations of eigenvaues Lecture notes
Example:
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛=
110101121
A
1st column of Q and R( ) 211 221
11 =+== ar
( )
( )
⎟⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜⎜
⎝
⎛
==
02
12
1
1 1
11
1 ar
q
2nd column of Q and R:
( ) ( )( ) 22
2
02
12
1
,102
, 1212 ==
⎟⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜⎜
⎝
⎛
⎟⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜⎜
⎝
⎛
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛== qar
( ) ( ) 311
1
011
102
22
2
112
222 =
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−=
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛=−= qrar
( ) ( ) ( )[ ]
⎟⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜⎜
⎝
⎛
−=−=
31
313
1
31 1
1222 qraq
Vittal rao/IISc.Bangalore M4/L6/L1/May2004/3
Numerical analysis/Computations of eigenvaues Lecture notes
3rd column of Q and R:
( ) ( )( ) 22
2
02
12
1
,111
, 1313 ==
⎟⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜⎜
⎝
⎛
⎟⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜⎜
⎝
⎛
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛== qar
( ) ( )( )
31, 23
23 == qar
( ) ( ) ( )2
231
133
33 qrqrar −−=
231
313
1
31
011
111
⎟⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜⎜
⎝
⎛
−−⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛−
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛=
32
32
94
91
91
323
13
1
2
==++=
⎟⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜⎜
⎝
⎛−
=
and
( ) ( ) ( ) ( )[ ]223
113
3
33
3 1 qrqrar
q −−=
Vittal rao/IISc.Bangalore M4/L6/L1/May2004/4
Numerical analysis/Computations of eigenvaues Lecture notes
⎟⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜⎜
⎝
⎛−
=
⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜
⎝
⎛ −
=
326
16
1
323131
23
⎟⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜⎜
⎝
⎛
−
−
=∴
32
310
61
31
21
61
31
21
Q ;
⎟⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜⎜
⎝
⎛
=
3200
3130222
R
and
AQR =⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛=
110101121
giving us QR decomposition of A.
Vittal rao/IISc.Bangalore M4/L6/L1/May2004/5
Numerical analysis/Computations of eigenvaules Lecture notes
Vittal rao/IISc.Bangalore M4/L7/V1/May 2004/1
QR algorithmLet A be any nonsingular nxn matrix. Let A = A1 = Q1 R1 be its QR decomposition. Let A2 = R1 Q1. Then find the QR decomposition of A2 as A2 = Q2 R2 Define A3 = R2 Q2 ; find QR decomposition of A3 as A3 = Q3 R3. Keep repeating the process. Thus A1 = Q1 R1 A2 = R1 Q1and the ith step is Ai = Ri-1 Qi-1Ai = Qi Ri Then Ai ‘ converges’ to an upper triangular matrix exhibiting the eigenvalues of A along the diagonal.