Upload
others
View
6
Download
0
Embed Size (px)
Citation preview
Copyright ©2008 by K. Pattipati 1
Lecture 10: Unsymmetric Eigen Value
Problem
Prof. Krishna R. Pattipati
Dept. of Electrical and Computer Engineering
University of Connecticut Contact: [email protected] (860) 486-2890
Fall 2008
October 29, 2008
ECE 6435Adv Numerical Methods in Sci Comp
Copyright ©2008 by K. Pattipati 2
Outline of Lecture 10
What is the Eigen value ?
Major Applications
Properties of Eigen values and Eigen vectors
Eigen value conditioning problem
Power method for finding the maximum Eigen value and its Eigen
vector
Inverse power method
• Smallest Eigen value and its Eigen vector
• Iterative improvement
QR method
• Hessenberg decomposition
• Shifted QR
• Implicit Q-theorem
• Francis double QR-step
Copyright ©2008 by K. Pattipati 3
What is the Eigen value problem?
• Computing the Eigen values and Eigen vectors of a matrix A
• Eigen values are also called modes, spectrum, characteristic
values or latent roots
Major applications
• Converting a system into a canonical form (e.g., SCF,
SOF, Diagonal form, Jordan form)
• Solution of Riccati equation by Potter’s method
Properties of Eigen values and Eigen vectors:
• Consider:
λ: Eigen value of A; x: Eigen vector of A
•
Preliminaries
x Ax bu
Ax x
0 ( ) 0 or ( )Ax I x A I x x N A I
1
2 1| | 0 ch. equation of ... 0n n
nA I A a a a
Copyright ©2008 by K. Pattipati 4
•
•
•
•
•
•
Properties of Eigen Values & Eigen Vectors
1
1 1
( ) ; | |nn
i n i
i i
tr A a A a
;
is called the right Eigen vector of
is called the left Eigen vector of = Eigen vector of
T T
i ii ii i
i
T
i
Ax x y A y
x A
y A A
1
1 1
1
0 why?
... ; ( ... )
0;
T
i jT
n
T
n
T T T T
i i i ii i ij j i i
x y i j
y
T AT I T T x x
y
y Ax y x y Ax y x
min max
For symmetric matrices, we have
/ ( )
/ ( ) is called the Rayleigh quotient
T T
T T
x Ax x x
x Ax x x
minFor PD and symmetric 0 min
iia
| | || || a pessimistic boundi
A
Copyright ©2008 by K. Pattipati 5
Gershgorin Circle Theorem
Gershgorin Circle Theorem
• Valid for arbitrary n x n matrices
• Eigen values lie in the union of circles, i.e.,
where
Proof:
Let λ be an Eigen value with Eigen vector,
But don’t know k a priori. So, take all possible circles
( ) , 1,...,i i
e
A D i n 1
{ : | | | | }n
i ii ij i
jj i
D C a a r
1 2( ... )T
nx x x x
1
1
1 1
( ) 1,...,
pick | | || || , i.e., | | max | |
| |Then, | | | | | |
| |
n
ii i ij j
jj i
k k jj n
n n
j
kk kj kj k
j jk
j k j k
Ax x a x a x i n
i k x x x x
xa a a r
x
Copyright ©2008 by K. Pattipati 6
Example:
1 2 3
10 2 3
1 0 2 ; (A)=(10.226, 0.3870+2.2216i, 0.3870 - 2.2216i)
1 -2 1
{ : | -10| 5}; { : | | 3}; { : | -1| 3};
A
D D D
Gershgorin Circle Theorem
x
3
0 1
x
x
3
D1
D2
D3
5
10
Copyright ©2008 by K. Pattipati 7
Eigen Value Conditioning Problem - 1
Eigen value conditioning problem
•
• Suppose A is perturbed A+dA=B
•
Example:
10
10
10 10 1
0 1 . . . 00 1 . . . 0
0 0 1. . 00 0 1. . 0,
0 . . . 0 10 . . . 0 1
0 . . . 0 0 10 . . . 0
10 | | 0
SCF for | | 10 | | 10
A B
n I A
I B
Q: how close ( ) and ( ) are?i i
A B
1 2Suppose A has Eigen values ... .
n
Copyright ©2008 by K. Pattipati 8
Analysis
• Consider where x is the left Eigen vector
and y is the right Eigen vector
•
• So, sensitivity ≈ inverse of inner product of left and right Eigen
vectors corresponding to the Eigen value
• Now, to formalize this notion
–
–
– Noting that and neglecting second order
terms, we obtain:
– So, smaller the inner product of left and right Eigen vectors,
larger is the sensitivity of Eigen values
, T T
Ax x y A y
2 2Assume || || || || 1 (normalized)x y
/ ( ) sensitivity 1/ | |T
d d S y x
if inner product is small trouble
Consider ( )( ) ( )A A x x d x xd d d Ax A x A x A x x x x xd d d d d d d d
T T
y A x y xd d
/ ( ) | |||| |||| || / | | || || / | |T T T T
y Ax y x A x y y x A y xd d d d
Eigen Value Conditioning Problem - 2
Copyright ©2008 by K. Pattipati 9
– So, condition number of the Eigen value problem is related to the
inverse of the inner product of left and right Eigen vectors
How are Eigen vectors affected• To first order, we have for Eigen value and Eigen vector
• Sensitivity of xk
depends on Si(λ) and EV separation
• Eigen vectors associated with repeated Eigen values are wobbly!
nearly to problems
1Sensitivity, ( )
| |
r
T
x y
Sy x
1
1
.
( )
(since 0 )( )
( )
k k k kk k
n
k i i k ki i i k k
i i k
T
Tki
ki iT
ik i i
Tk
ki
k iT
i ik i i
A x A x x x
x x x Ax x
y Axi k y x i k
y x
y Axx x
y x
d d d d
d d d
d
dd
k
kx
Eigen Value Conditioning Problem - 3
Copyright ©2008 by K. Pattipati 10
•
• Take any arbitrary u0. Then, since {x
i} span R
n,
• Let us consider the following iteration
•
•
• So, a good idea is to normalize uk
after each iteration
maxPower Method for finding ( )A
1 2
1 2
Suppose { } are simple, real and | | | | ... | |
Eigen vectors ... are independent
i n
nx x x
0
1
n
ii
i
u c x
1
1
1 11 1
21 1
[ ]
k k
kn
k i i
k i
i
u Au
cu c x x
c
1
1 1 11 1As , proportional to k
kk u c x x
1
1
Note that if | | 1 || || 0 as not good
if | | 1 || || as
ki
ki
u k
u k
Power Method for max min( ) ( )A or A
Copyright ©2008 by K. Pattipati 11
Normalized scheme
• But, a better way is to treat it as an overdetermined set of equations
1 2
1 21 2
1 1
1
1 1
1
1 1
1 1
1
1
...|| ||
and continue|| ||
as ,
2 successive iterates are linearly dependent
( ) can get
( )
k kk
n
k nn
k
k
k k k
k
k k
k
k
u c x c x c xu
uu Au u
u
k u u
u
u
1 1 11
can get ; 1T T
k k k k k ku u u u recall u u
Normalized Power Method
Copyright ©2008 by K. Pattipati 12
Power Method & Rayleigh Quotient
What happens in the symmetric case ?
Example
2 1
2
1 1
1
1
Since 0 , we have
1 O
Reyleigh quotient for symmetric matrices
T
i j
k
T
k k
T T
k k k k
x x j i
u u
u u u Au
11
22
0 1 2 4
2 -1 0.7071 has 3 with
-1 2 0.7071
0.7071and 1 with
0.7071
The power method gives a sequence:
1 0.8994 0.7809 , , ,
0 0.4472 0.6247
A x
x
u u u u
10
0.7328 0.7071, . . .,
0.6805 0.7071u u
Copyright ©2008 by K. Pattipati 13
Power algorithm for Real Eigen values:
Power Algorithm for Real Eigen values - 1
max
2
Given an x matrix and a tolerance parameter , the following algorithm
computes ( ).
Initialize arbitrary
0
0
|| ||
For 0, 1, 2, . . . DO
|| ||
/
old
n n A
A
u
k
s
c A
k
u
w u
u
1max
If | | stop: ,
else
1
end DO
T
new
new old new
old new
Aw
s u w
s s c s x w
k k
s s
Copyright ©2008 by K. Pattipati 14
Don’t expect convergence for at least iterations
Note: can be thought of as QR decomposition of uk
Can I get p biggest λi? Yes !!
What if λiis complex:
• Fact: complex Eigen values always occur in pairs
• So, start with
•
n
ˆ || ||k k k
u u u
0
2
1
Start with an x orthogonal matrix
0 and || || 1
For 1, 2, ... DO
SIMULTANEOUS ITERATION
(QR factorization)
end DO
T
i j
i
k k
k k k
n p Q q q
i j q
k
Z A Q
Q R Z
*
0 1 21 1...u c x c x
2
1 2
1 2 1 2
Assume and are rooots of 0
and
p q
p q
Power Algorithm for Real Eigen values - 2
Copyright ©2008 by K. Pattipati 15
• Then*
1 21 1 2 1
1 1 *
1 1 21 1 2 1
1 2 * 2
2 1 22 1 2 1
2 11 2 1 2
...
...
...
as , it is easy to prove that
( + ) 0 . . . (
k k
k
k k
k k
k k
k k
k k k
u c x c x
u c x c x A u
u c x c x A u
k
u u u
2
1
1 1
1 2
2 1 1 2 22 2
1)
3 successive iterates are linearly dependent
solve for , as follows:
unit norm vector || ||
; =|| ||
|| || || ||
k
k
k
k
k k k
k
k k k k k
p q
uu
u
uu A u u
u
u A u u u u
Power Method for Complex Eigen values - 1
Copyright ©2008 by K. Pattipati 16
Least-squares normal equations
1 2 2 1 12 2 2
1 2
1 1 2 22 2
then, from (1),
|| || || || || || 0
|| |||| || || ||
k k k k k k
k
k k k k k
u u u p u u qu
p uu u u u u
q
11 1 22
1 22 2
1 2
|| ||1 || || || ||
1
T T
kk k k k
k kT T
k k k k
p uu u u uu u
qu u u u
Power Method for Complex Eigen values - 2
Copyright ©2008 by K. Pattipati 17
1 2 1 2
2 2 2
1
2 2 1
2 22
1
2 1 1 2
1 22 2 2
1
2
1 22 2
|| ||
1
|| ||
1
|| || || ||
1
|| || || ||
T T T
k k k k k k
kT
k k
TT
k k k k k
k T
k k
T T T
k k k k k k
k kT
k k
T
k k k
k k
u u u u u u
p u
u u
u u u u u
u
u u
u u u u u u
q u u
u u
u u u
u u
1 2 1
2
11
T
k k
T
k k
u u
u u
Power Method for Complex Eigen values - 3
Copyright ©2008 by K. Pattipati 18
Power Algorithm for Complex Eigen Values - 1
Algorithm for finding complex Eigen values
• Given an n x n matrix A and tolerance parameter ε, the following
algorithm finds the complex Eigen values
0
1 0
11 2
2 1
22
1 2
0 1
0 2
1 2
0
any unit norm
|| ||
|| ||
0
0
For 0,1, 2, ... DO
|| ||
old
old
T
T
T
k
u
u Au
r u
c A
p
q
k
u Au
r u
r r r
a u u
b u u
c u u
Copyright ©2008 by K. Pattipati 19
2
2 0 12
2
0 2 1
1
2
1
( ) / (1 )
( ) / (1 )
If | | and | | :
Re( ) / 2
Im( ) 4 / 2
else
T
new
T
new
new old new old
new
new new
ol
p r u bu u a
q ru u cu a
p p c q q c
p
p q
p
0 1
1 2
1 2
1
end if
end DO
d new
old new
p
q q
u u
u u
r r
k k
Power Algorithm for Complex Eigen values - 2
Copyright ©2008 by K. Pattipati 20
2 2
11 2
1 2
1 2
1 2
1 1
- if 1; 1
- docomponent wise:
( ) (( ) ( ) )( ) ( ) ( )
( ) ( ) 2( ) ( )
where , , weregenerated from theusualalgorithm.
( )( ) ( )
2
k
k
k i k i k ik i k i k i
k i k i k i k i
k k k
k kk
k i i k k
u x a k
u u uu u u
u u u u
u u u
a au x a
a a
2
2 2( 1) 2
2 2
2
(1 )tofirst order
(1 )
- so,can doAitken -iteration onceevery threeiterations
k
kk
i ik
a
ax a x
a
d
• To improve convergence, use Aitken - iteration 2d
Aitken’s Acceleration
Copyright ©2008 by K. Pattipati 21
Inverse Power iteration
• Useful for finding the smallest Eigen value as well as for iterative
improvement
do LU on A
store L and U factors
• can be used for iterative improvement as follows:
1
1 1k k k kv A v Av v
1 1
2
1 2
1
1 1
1
Suppose and areapproximations to and .
isnormalized,so that || || 1
| | | | ... | |
- then ( )hasEigen value 0with Eigen vector .
- ( ) hasEigen value( ) big with Eigen vector .
- usepower method
n
i
i
x x
x x
A I x
A I x
1on ( )
Do LUon ( )
B A I
A I LU
Copyright ©2008 by K. Pattipati 22
Inverse Power Iteration Algorithm - 1 • Example again:
- the inverse power method gives the sequence
Inverse Power Iteration Algorithm:
1
1k kv A v
0 1 2 3 9
0.6 0.6727 0.6958 0.7034 0.7071, , , ,...,
0.8 0.7399 0.7182 0.7108 0.7071v v v v v v
1 1
1 1
2
Given andapproximate and for and respectively,
thealgorithmcomputesbetter approximation to and
For =0,1,2,...DO
|| ||
solve
solve
old
A x x
x
u x
s
k
uw
u
Lv w
Uu y
Copyright ©2008 by K. Pattipati 23
1 1
1
1 1
end DO
if :Stop / ,
else
1
end if
end DO
- when haveconverged / ( )
-method worksextremely well:
since( )
,
-shifting of spectrumis the
T
new
T T
new old
old new
s T T
k k k k k
ni
k iki i
k
s u w
s s u Au u u x u
s s
k k
u u Au u u
cu x
if then u x
keyidea of QR algorithmas well
-shifting of spectrumimprovesconvergence rate
Inverse Power Iteration Algorithm - 2
Copyright ©2008 by K. Pattipati 24
QR Method
A form of simultaneous iteration and much more
Key ideas of QR
• Reduce A to upper Hessenberg
- this can be done via Householder
- Perform QR decomposition
For k=1,2,…
- Note:
1 0 0 0
0 1
a similarity transformation
x x x x x x x x
x x x x x x x x
x x x x 0 x x x
x x x x 0 0 x x
TA H Q AQ
A H A
0 0
TQ AQ
1
via Givens transformation
~ upper Hessenberg if is!!
k k k
T
k k k k k k k
A Q R
A R Q Q A Q A
0 1( ) ( )and ( ) ( )i i i k i kA H A A
Copyright ©2008 by K. Pattipati 25
Example
• Illustration of
• So, the off-diagonal elements go from s to s3
• Subsequent iterations go to s9, s27, s81 …. because of a
shifting operation we will employ in a practical
scheme
almost diagonal and we can obtain the
Eigen values very fast!!
• Theorem: If is cyclic (i.e., n independent
Eigen vectors), then the sequence
1
T
k k k k k k k k kA Q R and A R Q Q A Q
2
2 3
3 2
1
0 0
(1 )
c s c s csA
s s c s
c s sRQ
s cs
Q R
1kA
1kA A
1
1 0 0... ....
T
k k k k k k k k k
T T T
k k k
A Q R A R Q Q A Q
Q Q Q AQ Q
Converges to upper Schur form
or upper quasi-triangular matrix
Copyright ©2008 by K. Pattipati 26
• Eigen values from Rii subblocks
• If Rii is 1x1, we get a simple Eigen value
• If Rii is 2x2, we get a complex Eigen value
Why do we transform to upper Hessenberg matrix first?
• QR decomposition on a full n x n matrix requires O(2n3/3)
operations
• If A is upper Hessenberg, QR decomposition takes only
O(4n2) operations
• If Q is accumulated, a further 4n2 operations
• More importantly, QR iteration maintains upper
Hessenberg structure
11 12 1
22 2
..
0 ..
..
n
n
nn
R R R
R R
R
where each Rii is a 1x1 or a 2x2 matrix
Upper Schur Form
Copyright ©2008 by K. Pattipati 27
Hessenberg Decomposition - 1
• Want an upper Hessenberg matrix H0 from A such that
• Note that if first r components of u are zero, then
• Suppose r = 3, n=5 and
0 0 0 0 0
0
and
~ product of Householder matrices [ 2 /( )]
x x x x x y y y x y y y
x x x x y y y y y y * *
x x x x 0 y y y 0 * * *
x x x x 0 y y y 0 0 * *
T T
T T
H Q AQ Q Q I
Q I uu u u
W1 W2
Done! n-2 steps
in general.
11 12
21 22
11
0where
0
and
where x block of
r TI
W
A AWAW
A A
A r r A
x x x | x x
x x x | x x
- - - + -
0 0 x | x x
0 0 x | x x
A
Copyright ©2008 by K. Pattipati 28
• 3x3 subblock does not change with Housholder
• In general, at step k, pivot on element ak+1,k
• pivot column
4
4 5 6 53 63
2 1/2
3 4 43 3
4
pick 0 0 0 ..
pick , , to zeroout , , .
5 ( )
T
n
n
i i i
i
u u u
u u u a a etc
u a i u a sign a
2 1/2
1, 1,
1
2,
,
0
.
( )( )
.
n
k k k k ik
i k
k k
n k
a sign a au
a
a
Hessenberg Decomposition - 2
Copyright ©2008 by K. Pattipati 29
• Given a matrix A, the following algorithm computes
an upper Hessenberg decomposition.
- For k=1, 2,…,n-2 Do
Determine Householder
2 1/2
1,
11,
,
1, 1,
2,
( )( )
0.
.
0
[ ]
( )
where.
n k
n
k k ik
i kk k
n k
n k
T
n k n kn k n k
k k k k
k k
n k
nk
sign a aa
a
u uI
a sign a s
au
a
Hessenberg Decomposition - 3
Copyright ©2008 by K. Pattipati 30
• Finding QR decomposition of H:
- need H=QR
- but, actually need Hnew = RQ in the QR-algorithm
- use Givens since H is almost upper .
For k = 1, 2,…, n-1 DO
2 1/ 2
1
1
11 12
21 22
( ) | |
( , )
end DO
n
ik k
i k
k k n k
n k
k k
n k n k n k
s a and u s
W Diag I
A AA W AW
A A
x x x x x
x x x x x
x x x
x x x
QR Decomposition of H - 1
Copyright ©2008 by K. Pattipati 31
1, 2 2
1,
1,
1,
, 1
Determine cos and sin
; , , sgn( )0
sgn( ) affectsonly two rows; 0
For 1,...,
k k k k
kkk k k kkkkk kk k k
k kk k
kk kk k k
kj kjk k
k j k k
c s
hc s x hhc s r h h k
hs c r r
h h r h
j k n DO
h hc s
h s c
1,
end DO
end DO
k jh
, 1 , 1
For =1, 2,..., -1
For =1, 2,..., +1
end DO
k k
ik i k ik i k
k k
k n
i k
c sh h h h
s c
QR Decomposition of H - 2
Copyright ©2008 by K. Pattipati 32
• This algorithm requires O(4n2) operations
• Example:
form of upper Hessenberg is maintained throughout
subsequent iterations
1 2
2 1 1 2
3 1 2 .6 .8 0 1 0 0
4 2 3 ; .8 .6 0 ; 0 .9996 .0249
0 .01 1 0 0 1 0 .0249 .9996
4.76 2.5442 5.4653
.32 .1856 2.1786
0 .0263 1.054
T T
H J J
J J HJ J
QR Decomposition of H - Example
Copyright ©2008 by K. Pattipati 33
• Obtain via Householder Transformation
- For k=1,2,…
Obtain via Givens Transformation
If at some point Ak+1 is a subdiagonal element ap+1,p is
“small”, i.e., |ap+1,p| ≤ [|app|+|ap+1,p+1|] then
set |ap+1,p| ~0 and break Ak+1 into subblocks as
A11 ~ upper Hessenberg
A22 ~ upper Hessenberg
- If p=n-1, then
- If p=n-2, solve 2x2 subblock A22 for
- For other p, work on A22 and then on A11
• Convergence rate:
where
• Typically the above algorithm converges very slowly
Crude Version of QR
1 0 0A H Q AQ
k k kA Q R
1
T
k k k k k kA R Q Q A Q
11 12
1
220k
A AA
A
nn na
1n nand
1| / |kn n
1 2| | | | ..... | |n
Copyright ©2008 by K. Pattipati 34
• To speed up, use a shifted version of QR
• Consider a shifted version of Ak: Ak-skI
• recall that
Choice of sk
• Compute eigen values of lower 2x2 subblock
Acceleration Methods
1
1, 1
, 1 1
1
( ) ( )
~ similar to
convergenceof 0 with rate| | / | |
so,if then 0 with rate | | / | |
if , Nogood!should beexpected from the
i k k i k
k k k k
T
k k k k k k k k
k k
p p p k p k
k k
n k n n n k n k
n n
A s I A s
A s I Q R
A R Q s I Q A Q A
a s s
s a s s
conditioning
argumentsearlier
1, 1 1, *
, 1 ,
* *
1, 1 1, 1 , 1 1,
, (complex)
;
n n n n
k k
n n n n
k k n n nn k k n n nn n n n n
a as s
a a
s s a a s s a a a a
Copyright ©2008 by K. Pattipati 35
• Now, consider the following double shift QR
• What is happening here?
• What is
Double Shifts for Complex Eigen Values - 1
1
*
1 1 1 1 1
2 1 1 1
;
k k k k
k k k k
k k k k k k
k k k k
A s I Q R
A R Q s I
A s I Q R s s
A R Q s I
2 1 1( ) ( ) unitary
(real)
H H
k k k k k k k
T
k
A Q Q A Q Q Q
Q A Q
1 1 ?k k k kM Q Q R R QR
1 1
2
1 1 1
1
1
( )
( ) ( ) ( )
( ) ( )
( ) ( )
k k k k
k k k k k k k k k k k k k k
k k k k k k k k k k
k k k k k k
Q A s I R
Q A R s A s I Q R s Q R s A s I
A s I Q R s Q R s A s I
A A s I s A s I
Copyright ©2008 by K. Pattipati 36
M is a real matrix
Since M is real, is real
so, have :
Ak+2 can be obtained as a QR decomposition of real
matrix:
• But, computation of M requires O(n3) operations
• Q: can we obtain Q directly? Yes
• A: via FRANCIS QR-DOUBLE STEP
1
2 *
1
( )( )
( )
k k k k
k k k k k k
A s I A s I
A s s A s s I
1k kQ Q Q
2 *
1
2
( )k k k k k k
T
k k
A s s A s s I M
and A Q A Q
2 *
1
2
-form ( )
-form
-form
k k k k k k
T
k k
M A s s A s s I
M QR
A Q A Q
Double Shifts for Complex Eigen Values - 2
Copyright ©2008 by K. Pattipati 37
• Let us get greedy and ask: Q: can we go from Ak →Ak+2 in O(n2)
operations?
• A: Yes, via Francis QR-double step
Ideas of Francis double step
- Compute the first column of M=Me1
- Find a Householder matrix W0 W0Me1=e1
- Apply Householder matrices W1W2….Wn-2 to W0 AkW0 to reduce
it to an upper Hessenberg form.
• Essentially, we have used Q1TAkQ1, where
• A theorem called implicit Q-theorem (see below) tells us that Q=Q1
• Fact: Me1 has only 3 non-zero components
Me1 = [x y z 0….0]T
where
Francis Double Step
1 2 3 2 1 0....n nQ W W W WW
2 *
11 12 21 1 11
*
1
21 11 22 1
21 32
( ) ;
( : )
( ( ))
; where areelementsof
k k k k
k k
k k
ij k
x a a a s s a s s
note s s
y a a a s s
z a a a A
Copyright ©2008 by K. Pattipati 38
• When n=6, W0AkW0 will change rows and columns 1, 2 &3 only
• Implicit Q-Theorem:
- suppose Q = (q1 q2 … qn) and V=(v1 v2 … vn) are orthogonal
matrices QTAQ = H and VTAV = G are upper Hessenberg (e.g., H
and G are obtained via Householder or Givens transformation)
- if v1=q1, then vi= ±qi and |hi,i-1|=|gi,i-1| for i=1,2,…,n
- formally, AQ=QH, AV=VG and v1=q1 then
Q=VD, V=D-1GD=DGD; |di|=1 i
If AQ=QH and the first column of Q is given, then Q and H are
essentially determined.
0 0
x x x x x x
x x x x x x
x x x x x x
x x x x x x
0 0 0 0 0 0
0 0 0 0 0 0
kW A W
Implict Q- Theorem - 1
Copyright ©2008 by K. Pattipati 39
• Proof: Consider
- If we insist on positive hk+1,k, then qi are uniquely determined for i 2 .
- Both Givens and Householder provide the same Q, since q1=e1 for both.
11 12 1
21 22 2
32 31 2 1 2
, 1
1 11 1 21 2 11 1 1 21 2 1 11 1
21 1 11 1 2 21
2 12 1
...
...
0 ...[ ... ] [ ... ]
. . . .
0 ..
;
| | || || so that is determined to within +or -sign
n
n
nn n
n n nn
T
h h h
h h h
h hAq Aq Aq q q q
h h
Aq h q h q h q Aq h q Aq h q
h Aq h q h
Aq h q
22 2 32 3
12 1 2 22 2 2
32 2 12 1 22 2 2
; ;
| | || ||
T T
h q h q
h q Aq h q Aq
h Aq h q h q
Implict Q- Theorem - 2
Copyright ©2008 by K. Pattipati 40
• Coming back to QR Algorithm
- the matrix we have is
- Francis QR step avoids forming explicitly
- Direct formation of M requires O(n3) operations and not practical
- We lose advantage of the upper Hessenberg structure
• We use implicit Q-theorem as follows:
- Step 1: where Ak is an upper Hessenberg
matrix,
2 * *( )k k k k k kM A s s A s s I
2 * *( )k k k k k kM A s s A s s I
*
1 1( )( )k k k kMe A s I A s I e
11 12 1
21 12 2
32 3
, 1
* *
11 11 11 12 21
*
21 21 11 22 21
1 32 21
...
...
0 ...
0
( )( )
( ) ( )
00
. ..
0 00
k n
k n
n
n n nn
k k k
k k
a s a a
a a s a
a a
a a
xa s a s a s a a
ya a a s a s a
zMe a a
Use of Implicit Q-Theorem - 1
Copyright ©2008 by K. Pattipati 41
- Step 2: Find
• Key: Similarity transformation with W0 affects rows 1, 2 and 3 and
columns 1, 2 and 3 only.
2 2 2 1/ 2
0 0 0
* sgn( )( )
0
0, ,
0 . 0
. . .
0 0 0
T
x x x x y z
y y
z zuuW W W I u
0 0
x x x xx x x ... x
x x x x x x x x x x x x xx x x ... x
x x x 0 x x x 0 x x x x x x x 0=x x ... x
x x x x x x 00 x x x x x. . . . .
0 I 0 I .....0 . 0 I0 .. 0 x x
000 x
kW A W
Use of Implicit Q-Theorem - 2
Copyright ©2008 by K. Pattipati 42
x x x x x x
x x x x x x
x x x x x x
= x x x x x x
0 0 0 x x x
. . . .
0 0 0 0 x x
two more nonzero elements than needed
(3,1), (4,1), (4,2)
- Step 3: Compute Householder matrices W1, W2,…, Wn-1 to convert
W0AkW0 such that Wn-2Wn-3…W1W0W1..Wn-2 is upper Hessenberg
adds two more elements in each column
x x x x x x x x x xx x x x x x x x x x x x
x x x x x x x x x xx x x x x x x
0 x x x x 0 x x x xx x x x x x
. x x x x . 0 x ..x x x x x x
. x x x x . 0 x ..0 0 x x x x
0 0 .. . x ... . . . .
. . . . . .0 0 0 0 x x
0 0 0 x x 0 0 0 x x
x x x x x x
x x x x x x x x x x x
0 x x x x x 0 x x x x x
etc... .. x x x . 0 x x x x
. .. x x x . . . x x x
. . . . . . . 0 x x
0 0 0 0 0 x 0 0 0 0 0 x
Use of Implicit Q-Theorem - 3
Copyright ©2008 by K. Pattipati 43
• The mechanization of Francis QR-double step is as follows:
Francis QR-double step:
For k=1, 2, …, n-2
3
2 2 2
k k k n k
n n n
W Diag I W I
W Diag I W
2
100
01
x x x
x x x
x x x
I
W
Note: Wke1=e1W0 and Q have the same first
column Q1e1=Qe1 where Q1=W0W1…Wn-2
Q1=Q by the implicit Q-theorem
2
11 12 21 11
21 11 22
21 32
1;
( )
( )
mm nn
mm nn mn nm
m n s a a
t a a a a
x a a a a s t
y a a a s
z a a
Formalizing Francis QR-double Step - 1
Copyright ©2008 by K. Pattipati 4444
3
2 2
2 2 2
*
if - 2 then Determine 0
0
; [ ]
* else Determine
0
; [
k k
k k k k k n k
n n
n n n
x
k n W W y
z
A W AW W Diag I W I
xW W
y
A W AW W Diag
2 2
2, 1 3, 1
4, 1
]
end if
;
if -3
end if
end DO
n n
k k k k
k k
I W
x a y a
k n
z a
Formalizing Francis QR-double Step - 2
Copyright ©2008 by K. Pattipati 4545
The overall QR-algorithm employing Francis double-step is as follows:
Overall QR-algorithm
Form A0 = Q0TAQ0 upper Hessenberg via Householder transformation
Form the orthogonal matrix Q0 ← Wn-2 … W1
Repeat
set to zero all subdiagonal elements satisfying
Find the largest nonnegative q and smallest p
where A33 isupper quasi-triangular and A22 is unreduced upper
Hessenberg
If q = n, then
upper triangularize all 2 by 2 diagonal blocks in A that have real eigen
values. Accumulate orthogonal transformations if necessary and quit .
1, 1, 1| | [| | ||]i i ii i ia a a
11 12 13
22 23
33
- -
0 note: can have 0
0 0
p n q p q
A A A p
A A A n p q q p
A q
Overall QR- Algorithm - 1
Copyright ©2008 by K. Pattipati 4646
Apply a Francis QR double – step to A22:
else
A22 ← Q1TA22Q1
if Q and upper schur form of A are desired then
Q ← Diag(Ip Q1 Iq)
A12 ← A12Q1
A23 ← Q1TA23
go to Repeat
• Total Computational effort O(15n3) operations.
• If only Eigen values are desired, the computational load is O(8n3)
• Balance A if A has widely differing Eigen values
Overall QR- Algorithm - 2
Copyright ©2008 by K. Pattipati 47
Summary
Properties of Eigen values and Eigen vectors
Eigen value conditioning problem
Power method for finding the maximum Eigen value and
its Eigen vector
Inverse power method
• Smallest Eigen value and its Eigen vector
• Iterative improvement
QR method
• Hessenberg decomposition
• Shifted QR
• Implicit Q-theorem
• Francis double QR-step