Time-Discrete Signals and SystemsAdaptation Course for Master Studies
Dr.-Ing. Volker KühnInstitute for Telecommunications and High-Frequency Techniques
Department of Communications EngineeringRoom: N2300, Phone: 0421/218-2407
Lecture
Thursday, 08:30 – 10:00 in N1170
Dates for exercises will be announced during lectures.
Tutors
Ronald BöhnkeRoom: N2380
Phone [email protected]
www.ant.uni-bremen.de/teaching
2UniversitätBremen
Volker Kühn Time-Discrete Signals and Systems
OutlineSummary of Laplace and Fourier TransformationZ-Transformation
Time-discrete SignalsProperties of Z-Transformation and Convergence System description by Z-transformation
Stochastic ProcessesCharacterization of stochastic processes and random variablesProbabilities, densities, distributions, moments, stationary and ergodic processesCentral limit theoremCorrelation and spectral power density (real and complex signals)System analysis for stochastic input signals (Theorem of Wiener-Khintchine)
Multiple-Input Multiple-Output SystemsSystem DescriptionLinear Algebra (eigenvalues and eigenvectors, pseudo inverse) -- briefDecompositions (QR, unitary matrices, singular value, Cholesky )Statistical representation (multivariate distributions)
Outline
3UniversitätBremen
Volker Kühn Time-Discrete Signals and Systems
Linear Algebra
Notations and definitionsVectors and matrices
Elementary operations, Matrix multiplicationDeterminants
Special MatricesSymmetric, orthogonal, complex, circulant, Toeplitz...
Linear equation systemsGaussian elimination, Cramer’s rule, iterative methods
Matrix factorizationsLU, Cholesky, QR (Householder, Givens, Gram-Schmidt)
Eigenvalues and eigenvectors, SVD (condition, pseudo-inverse)
Least squares, matrix inversion lemma
Multiple-Input Multiple Output Systems
4UniversitätBremen
Volker Kühn Time-Discrete Signals and Systems
Notations and Definitions (1)
VectorsColumn vectors (preferred): boldface lower case
Row vectors: underlined boldface lower case
MatricesBoldface capital letters
Column vectors are just matrices
Row vectors are just matrices
Multiple-Input Multiple Output Systems
1
2
n
x
x
x
=
x
[ ]1,1 1,2 1, 1
2,1 2,2 2, 21 2
,1 ,2 ,
n
nn
m m m n m
a a a
a a a
a a a
= = =
a
aA a a a
a
[ ]1 2 nx x x=x
1m×1 n×
5UniversitätBremen
Volker Kühn Time-Discrete Signals and Systems
Notations and Definitions (2)
Some special matricesIdentity matrix and zero matrix
Diagonal, lower and upper triangular matrices
Multiple-Input Multiple Output Systems
1 0 0
0 1 0
0 0 1
=
I
1
2
0 0
0 0
0 0 n
d
d
d
=
D
1,1
2,1 2,2
,1 ,2 ,
0 0
0
n n n n
l
l l
l l l
=
L
1,1 1,2 1,
2,2 2,
,
0
0 0
n
n
n n
u u u
u u
u
=
U
0 0 0
0 0 0
0 0 0
=
0
6UniversitätBremen
Volker Kühn Time-Discrete Signals and Systems
Let A, B, C be matrices and , be scalarsAddition and scalar multiplication are defined element-wise
PropertiesAddition commutativeAddition associative
Neutral element of addition
Inverse element of additionScalar multiplication associative
Neutral element of scalar multiplication
Scalar multiplication distributiveScalar multiplication distributive
Basic Operations and Properties
Multiple-Input Multiple Output Systems
+ = +A B B A
m n×
( ) ( )+ + = + +A B C A B C
α β
1,1 1,1 1, 1,
,1 ,1 , ,
n n
m m m n m n
a b a b
a b a b
+ + + = + +
A B1,1 1,
,1 ,
n
m m n
a a
a a
α α α = α α
A
( )α + = α + αA B A B
( ) ( )αβ = α βA A
( )α + β = α + βA A A
+ =A 0 A( )+ − =A A 0
1 =A A
7UniversitätBremen
Volker Kühn Time-Discrete Signals and Systems
Let A be a matrix and B be a matrix
The product C=AB is a matrix with elements“row times column”
Note: number of columns of A has to equal number of rows of BEquivalent formulations of the matrix multiplication:
Matrix Multiplication (1)
Multiple-Input Multiple Output Systems
m n× n p×
m p× , , ,1
n
i j i k k jk
c a b=
= ⋅∑
[ ]1,1 1, 1
1
,1 ,
n
n
m m n m
a a
a a
= = =
a
A a a
a
1,1 1, 1
1
,1 ,
p
p
n n p n
b b
b b
= = =
b
B b b
b
1, ,1 1, ,1 1 1 1 1 1
11
1, ,1 , ,
1 1
n n
k k k k pk k p n
p k kkn n
m m p mm k k m k k p
k k
a b a b
a b a b
= =
=
= =
⋅ ⋅ = = = = = ⋅ ⋅
∑ ∑∑
∑ ∑
a b a b a B
C Ab Ab a b
a b a b a B
8UniversitätBremen
Volker Kühn Time-Discrete Signals and Systems
Special casesm=1, n>1, p=1 (row vector times column vector)
m=1, n>1, p>1 (row vector times matrix)
m>1, n>1, p=1 (matrix times column vector)
m>1, n=1, p>1 (column vector times row vector)
Matrix Multiplication (2)
Multiple-Input Multiple Output Systems
1
n
k kk
c a b=
= =∑ab
1
n
k kk
a=
= =∑c aB b
1
n
k kk
b=
= =∑c Ab a
1,1 1,1 1,1 1,
,1 1,1 ,1 1,
p
m m p
a b a b
a b a b
= = ⋅
C ab
scalar
row vector
column vector
matrix
Inner or scalar product
Outer or dyadic product
Matrix-vector products
9UniversitätBremen
Volker Kühn Time-Discrete Signals and Systems
PropertiesMatrix multiplication distributive
Matrix multiplication distributive Mixed scalar / matrix multiplication associative
Matrix multiplication associative
Note: matrix multiplication is not commutative in generalExample
Matrix Multiplication (3)
Multiple-Input Multiple Output Systems
( ) ( )=AB C A BC
( )+ = +A B C AC BC
( )+ = +A B C AB AC( ) ( ) ( )α = α = αAB A B A B
2 6
1 7
=
A3 1
2 1
− − =
B15 6
1 20
=
C
6 4
11 6
=
AB7 25
5 19
− − =
BA
36 132
22 146
=
AC36 132
22 146
=
CA
≠AB BA
=AC CA
⇒
⇒
10UniversitätBremen
Volker Kühn Time-Discrete Signals and Systems
Transpose of a matrix
Row vectors become column vectors and vice versa
Hermitian transpose of a complex matrix
Transpose of the complex conjugate matrix
Properties
and
Transpose and Hermitian Transpose
Multiple-Input Multiple Output Systems
( )T T T+ = +A B A B( )TT =A A
( )T T T=AB B A
[ ]1,1 1, 1
1
,1 ,
n
n
m m n m
a a
a a
= = =
a
A a a
a
( )* * * *1,1 1, 1,1 ,1 1
*1
* * * *,1 , 1, ,
T Hn m
TH H Hm
Hm m n n m n n
a a a a
a a a a
= = = = =
a
A A a a
a
( )H H H+ = +A B A B( )HH =A A
( )H H H=AB B A
1,1 ,1 1
1
1, ,
Tm
T T Tm
Tn m n n
a a
a a
= = =
a
A a a
a
⇒
11UniversitätBremen
Volker Kühn Time-Discrete Signals and Systems
Determinant of a matrix
Determinant of a matrix (Sarrus’ rule)
Determinant of a matrixLet the matrix Ai,j equal A without the i-th row and j-th column
Recursive definition of determinant by cofactor expansion
1,1 1,21,1 2,2 2,1 1,2
2,1 2,2
deta a
a a a aa a
= = −A
Determinants (1)
Multiple-Input Multiple Output Systems
2 2×
3 3×
1,1 1,2 1,3
2,1 2,2 2,3 1,1 2,2 3,3 1,2 2,3 3,1 1,3 2,1 3,2
3,1 3,2 3,3 3,1 2,2 1,3 3,2 2,3 1,1 3,3 2,
1,1 1,2
2,1 2,2
3 1 1,1 ,, 23 2
det
a a a
a a a a a a a a a a a a
a a a a a a a a a a a
a a
a
a a
a
a
= = + +− − −
A
n n×( ) ( )1 1n n− × −
, ,1
det ( 1) detn
i ji j i j
i
a+
=
= −∑A A , ,1
det ( 1) detn
i ji j i j
j
a+
=
= −∑A A
column expansion row expansion
12UniversitätBremen
Volker Kühn Time-Discrete Signals and Systems
Fundamental propertiesLinearity in columns (rows)
Exchanging two columns (rows)Determinant of identity matrix
Some additional propertiesSymmetry in columns and rows
Zero column (row)
Two equal columns (rows)Multiple of one column (row)
Scalar multiplication
Adding two columns (rows)Determinant of matrix product
All properties valid for arbitrary matrices
Determinants (2)
Multiple-Input Multiple Output Systems
det detT =A A
( )det det det= ⋅AB A B
( )det detnα = αA A
1 1 2 1 2 1 2′ ′ ′ ′α + α = α ⋅ + α ⋅a a a a a a a
2 1 1 2= −a a a adet 1=I
2 0=0 a
1 1 0=a a
1 2 1 2α = α ⋅a a a a
1 2 2 det+ α =a a a A
n n×
13UniversitätBremen
Volker Kühn Time-Discrete Signals and Systems
Full expansion of the determinant by recursive application of the definition
Sum over all n! vectors p containing permutations of 1,2,...,n
per{p} is the number of pairwise exchange operations needed for the permutation
Determinant of diagonal or triangular matrixAt least one factor is zero for all
Efficient calculation of determinantDeterminant unaffected by adding multiples of rows (columns) to rows (columns)
Transform A into triangular matrix by elementary row (column) operations
Determinants (3)
Multiple-Input Multiple Output Systems
{ } { }! !
1 1
per per, ,
1 1
det ( 1) ( 1)n n
i i
n n
i p p ii i
a a= == =
= − = −∑ ∑∏ ∏p p
p p
p p p p
A
,1
detn
i ii
l=
= ∏L ,1
detn
i ii
u=
= ∏U,
1
detn
i ii
d=
= ∏D
[ ]1 2T
n≠p
14UniversitätBremen
Volker Kühn Time-Discrete Signals and Systems
System of m linear equations in n unknowns
Matrix-vector notation
Geometric interpretationsx is the intersection of m hyperplanesb is a linear combination of the column vectors
Linear Equation Systems (1)
Multiple-Input Multiple Output Systems
1,1 1 1,2 2 1, 1
2,1 1 2,2 2 2, 2
,1 1 ,2 2 ,
n n
n n
m m m n n m
a x a x a x b
a x a x a x b
a x a x a x b
+ + + =+ + + =
+ + + =
1,1 1,2 1, 1 1
2,1 2,2 2, 2 2
,1 ,2 ,
n
n
m m m n n m
a a a x b
a a a x b
a a a x b
⋅ =
1,1 1,2 1, 1
2,1 2,2 2, 2
,1 ,2 ,
n
n
m m m n m
a a a b
a a a b
a a a b
=Ax b
i ib=a x
1
n
i ii
x=
=∑ a b
⇔
Extended coefficient matrix
15UniversitätBremen
Volker Kühn Time-Discrete Signals and Systems
Illustration for system (hyperplanes straight lines)
Linear Equation Systems (2)
Multiple-Input Multiple Output Systems
1a
2a
1 1x a
2 2x a
b
1a2a
b
1a 2ab
1 1b=a x
2 2b=a x1 1b=a x
2 2b=a x1 1b=a x
2 2b=a x
1x
2x
1x
2x
1x
2x
2 2×
intersecting straight lines parallel straight lines identical straight lines
a1, a2 linearly independent a1, a2 parallel a1, a2, b parallel
unique solution no solution infinite number of solutions
16UniversitätBremen
Volker Kühn Time-Discrete Signals and Systems
Square linear equation system Ax=b with n equations in n unkownsCramer’s rule
Let Aj equal A with the j-th column replaced by b
Then the j-th element of x is
Proof: substitute into Aj and use linearity in columns
Three possibilitiesunique solutionno solution
infinite number of solutions
Linear Equation Systems (3)
Multiple-Input Multiple Output Systems
1 1 1j j j n− + = A a a b a a
1
n
i ii
x=
=∑b a
det
detj
jx =A
A
det 0≠A
det 0 det 0 and for all j j= =A A
det 0 det 0 and for some j j= ≠A A
17UniversitätBremen
Volker Kühn Time-Discrete Signals and Systems
Example: system
Gaussian Elimination (1)
Multiple-Input Multiple Output Systems
1,2 1,3 1
2,2 2,3 2
3,2 3,3 3
1,1 1,2 1,3 1(1) (1)2,3 2(1) (1)3,3 3
1,
1,1
(1)2,2
(2)3,
1 1,2 1,3 1(1) (1) (1)
2,1
3,1
(1)3,
2,2 2,3 2(233
)
2
0
0
0
0 0
a a b
a a b
a a b
a a a b
a b
a b
a
a
a a b
a a b
ba
a
a
a
a−
−
−
( )( )
(2) (2)3 3 3,3
(1) (1) (1)2 2 2,3 3 2,2
1 1 1,2 2 1,3 3 1,1
/
/
/
x b a
x b a x a
x b a x a x a
=
= −
= − −
(1) EliminationSubtracting multiples of rows to create zerosTransform system into upper triangular form
(2) Back-substitutionSolve for unknownsComputation in reverse order
3 3×
Pivot elements
Extension to (1): If If for some k > j
exchange rows
If for all k > jmove to next column
( 1), 0j
j ja − =( 1), 0j
k ja − ≠
( 1), 0j
k ja − =Reduced systems
2,1,1 1,12 /l aa⋅ =
3,1,1 1,13 /l aa⋅ =
(1)3,23
(1,2)
, 2 2/l aa⋅ =
18UniversitätBremen
Volker Kühn Time-Discrete Signals and Systems
Special casesAll diagonal elements nonzero
Zero row in coefficient matrix, corresponding right hand side nonzero
Zero rows in coefficient matrix, corresponding right hand sides zero
Gaussian Elimination (2)
Multiple-Input Multiple Output Systems
0
0 0
• ∗ ∗ ∗• ∗ ∗
• ∗
0
0 0 0
∗ ∗ ∗ ∗∗ ∗ ∗
•
0
0 0 0 0
• ∗ ∗ ∗• ∗ ∗ 0 0
0 0 0 0
• ∗ ∗ ∗• ∗ 0 0 0 0
0 0 0 0
• ∗ ∗ ∗
unique solution
no solution
0 0 0 0
0 0 0 0
0 0 0 0
infinite number of solutions
3x 2x 2x 3x 3x2x1x Free parameters
19UniversitätBremen
Volker Kühn Time-Discrete Signals and Systems
General formulation of the algorithm(1) Initialization and elimination
Gaussian Elimination (3)
Multiple-Input Multiple Output Systems
(0) (0)
( 1) ( 1), , ,
( ) ( 1) ( 1),
( 1),
, , ,
( ) ( 1) ( 1),
: , :
: 1 1
: 1
/
: 1
for to do
for to
find pivot ele
do
for to do
ment
end
end
en
d
j j
j
j ji j i n j n
j
j j ji k i k i j j k
j j ji i
j
i j
j n
j
j m
i j m
l a a
k n n
a a l a
b b b
a
l
− −
− −
−
−
−
= == −
= +=
= +
= − ⋅
= − ⋅
A A b b
( 1) ( 1), ( 1)
1 ,
: 1
1
choose values for free parameters
for downto do
end
j
j j
nj j
n j j k k jk n j n
j r
x b a xa
− −−
= +
=
= − ⋅ ⋅
∑
(2) Consistency check
(3) Back-substitution
( 1),
:
: 1
0
index of first nonzero column
if no then , break
exchange rows, so that j
j
j
jj n
n
n r j
a −
=
= −
≠
( )if 0 for some then stoprkb k r≠ >
Pivot search
20UniversitätBremen
Volker Kühn Time-Discrete Signals and Systems
Result after Elimination Step
Number of nonzero rows on left hand side: Rank of Matrix A
Solution exists only if orUnique solution if no free parameters
Infinite number of solutions if free parameters
Gaussian Elimination (4)
Multiple-Input Multiple Output Systems
1
2
(0) (0)1, 1
(1) (1)2, 2
( 1) ( 1),
( )1
( )
* * * *
0 * * *
0 0 * *
0 0 0 0
0 0 0 0
r
n
n
r rr n r
rr
rm
a b
a b
a b
b
b
− −
+
{ }rank r=A
r
m r
−
r m= ( ) ( )1 0 and r r
r mr m b b+< = = =…r n=
r n< n r−
21UniversitätBremen
Volker Kühn Time-Discrete Signals and Systems
Linear equation systemBasic idea of iterative algorithms
Start with initial estimate of the solution vectorFind improved approximation from previous approximationStop after convergence
JacobiSolve row i for unknown xi
Parallel implementation possible
Gauss-SeidelUse already updated valuesBetter convergence behavior than JacobiNo parallel implementation possible
Conjugate GradientMore complicated implementation, but usually fast convergence
Iterative Solution of Linear Equation Systems
Multiple-Input Multiple Output Systems
,1
1 for n
i j j ij
a x b i n=
= ⇔ = ≤ ≤∑Ax b
(0)x
1( 1) ( ) ( )
, ,1 1 ,
1i nk k k
i i i j j i j jj j i i i
x b a x a xa
−+
= = +
= − − ⋅
∑ ∑
1( 1) ( 1) ( )
, ,1 1 ,
1i nk k k
i i i j j i j jj j i i i
x b a x a xa
−+ +
= = +
= − − ⋅
∑ ∑
( 1)k +x ( )kx
22UniversitätBremen
Volker Kühn Time-Discrete Signals and Systems
Inverse A-1 of a square matrix A
Relation of inverse to linear equation systems
Calculation of the inverse by Gauss-Jordan methodn simultaneous linear equation systemsForward eliminationBackward elimination
Inverse exists only if has a unique solution ( A nonsingular)Condition:
Properties
Inverse Matrix
Multiple-Input Multiple Output Systems
1 1− −= =A A AA I
( ) 11 −− =A A
( ) 1 1 1− − −=AB B A
( ) ( )1 1 HH − −=A A
n n×
1− ⇒ A I U L1 1− − ⇒ U L I A
[ ] [ ]1 n = = ⇔Ax Ax AX I A I
=AX I
det 0≠A{ }rank n=A ⇔
1−= ⇔ =Ax b x A b
23UniversitätBremen
Volker Kühn Time-Discrete Signals and Systems
Every invertible matrix A can be written as the product of a lower triangular matrix L and an upper triangular matrix U
Application: Solution of linear equation systemwith constant coefficient matrix for different right hand sides
Inversion of triangular matrices easy solve and then
Calculation of LU decomposition by Gaussian eliminationForward elimination:L contains factors from the elimination steps
Direct calculation of LU decomposition (example: matrix)
Calculation order:
LU Decomposition
Multiple-Input Multiple Output Systems
1,1 1,2 1,3 1,1 1,2 1,3 1,1 1,2 1,3
2,1 2,2 2,3 2,1 2,2 2,3 2,1 1,1 2,1 1,2 2,2 2,1 1,3 2,3
3,1 3,2 3,3 3,1 3,2 3,3 3,1 1,1 3,1 1,2 3,2 2,2 3,1 1,3 3,2 2
1 0 0
1 0 0
1 0 0
a a a r r r r r r
a a a l r r l r l r r l r r
a a a l l r l r l r l r l r l r
= ⋅ = + + + + ,3 3,3r
+
=A LU
=Ly b
= =Ax LUx b
=Ux y=Ux y
[ ] 1− = ⇒ A LU I U L( 1) ( 1)
, , ,/j ji j i j j jl a a− −=
3 3×
1,1 1,2 1,3 2,1 3,1 2,2 2,3 3,2 3,3r r r l l r r l r→ → → → → → → →
24UniversitätBremen
Volker Kühn Time-Discrete Signals and Systems
Let A be hermitianpositive definite
Then A is fully characterized by alower triangular matrix L
Cholesky decomposition
Similar to LU decompositionBut computational complexity reduced by factor 2
Example: matrix
Calculation order:
Cholesky Decomposition
Multiple-Input Multiple Output Systems
H =A A0H > ∀x Ax x (0)
( 1), ,
( 1) *, , ,
( ) ( 1) *, , , ,
:
: 1
: 1
/
: 1
for to do
for to do
for to do
end
end
end
kk k k k
ki k i k k k
k ki j i j i k j k
k n
l a
i k n
l a l
j k i
a a l l
−
−
−
==
=
= +=
= += − ⋅
A A
2 * *1,1 1,1 2,1 1,1 3,1
1,1 1,2 1,32 2* * *
2,1 2,2 2,3 2,1 1,1 2,1 2,2 2,1 3,1 2,2 3,2
2 2 2* * *3,1 3,2 3,33,1 1,1 3,1 2,1 3,2 2,2 3,1 3,2 3,3
l l l l la a a
a a a l l l l l l l l
a a al l l l l l l l l
= + + + + +
1,1 2,1 3,1 2,2 3,2 3,3l l l l l l→ → → → →
H=A LL
3 3×
Algorithm
25UniversitätBremen
Volker Kühn Time-Discrete Signals and Systems
Every matrix A can be written as whereQ is a matrix with orthonormal columns
R is an upper triangular matrix
Columns of A are represented in the orthonormal base defined by Q
Illustration for the case
QR Decomposition (1)
Multiple-Input Multiple Output Systems
m n×
1
0
for
for Hi j
i j
i j
== ≠
q q
n n×
H =Q Q I⇔
m n×=A QR
2m×
1 1,1 1r=a q
2 1,2 1 2,2 2r r= +a q q2q
1q
,1
k
k i k ii
r=
=∑a q
1,2 1r q
2,2 2r q
26UniversitätBremen
Volker Kühn Time-Discrete Signals and Systems
Calculation of QR decomposition by modified Gram-Schmidt algorithmCalculate length (Euclidean norm) of a1 r1,1
Normalize a1 to have unit length q1
Projection of a2,...,an onto q1 r1,j
Subtract components of a2,...,an parallel to q1
Continue with next column
Q is computed from left to rightR is computed from top to bottomIllustration for the case
QR Decomposition (2)
Multiple-Input Multiple Output Systems
(0)
( 1),
( 1),
( 1),
( ) ( 1),
:
: 1
/
: 1
for to do
for to do
end
end
kk k k
kk k k k
H kk i k j
k kj j k i k
k n
r
r
i k n
r
r
−
−
−
−
==
=
== +=
= −
Q A
q
q q
q q
q q q
1 1,1 1r=a q
2 1,2 1 2,2 2r r= +a q q2q
1q 1,2 1r q
2,2 2r q
(1)jq
2m×
27UniversitätBremen
Volker Kühn Time-Discrete Signals and Systems
Householder reflections: reflect x onto y by multiplication with unitary matrix
Special case: create zeros in a vector
Application to QR decomposition of matrix A
QR Decomposition (3)
Multiple-Input Multiple Output Systems
(1 ) Hw= − + ⋅H I uu with and H
Hw
−= =−
x y x uu
x y u x u
=y Hx
x Huu x
Huu y[ ]|| ||
T=y x 0
m n×
: , :
: 1
( : , )
[|| || ]
, ,
( : , : ) ( : , : )
(:, : ) (:, : )
for to do
calculate
end
m
T
H
k n
k m k
w
k m k n k m k n
k m k m
= ==
==
= ⋅= ⋅
R A Q I
x R
y x 0
u H
R H R
Q Q H
Loop through all columns
Initialization
Create zeros below the maindiagonal in k-th column of R
Update unitary matrix Q
28UniversitätBremen
Volker Kühn Time-Discrete Signals and Systems
Givens rotationsLet equal an identity matrix except for
is unitary and describes a rotation
Special choices for c and s:Linear transformation
Givens rotation can create zero while changing only one other element
Example
QR Decomposition (4)
Multiple-Input Multiple Output Systems
( , , )i k θG*, ,
*, ,
cos
sin
i i k k
i k k i
g g c
g g s
= = θ =
− = = θ =
( , , )i k= θ ⋅y G x
2 2 2 2 and i i k k i kc x x x s x x x= + = − +
( , , )i k θG
⇒2 2
, 0, ,i i k k j jy x x y y x j i k= + = = ∀ ≠
31 2 (2,3, )(2,3, ) (1,2, )
* * * * * * * * * * * *
* * * * * * 0 * * 0 * *
* * * 0 * * 0 * * 0 0 *
θθ θ
= → → → =
GG GA R
3 2 1(2,3, ) (1,2, ) (2,3, )= θ θ θR G G G A 1 2 3(2,3, ) (1,2, ) (2,3, )H H H= θ θ θQ G G G
⇒
⇒
29UniversitätBremen
Volker Kühn Time-Discrete Signals and Systems
Special eigenvalue problem for arbitrary matrices
Condition for existence of nontrivial solutionsCharacteristic polynomial of degree n has to be zero
Zeros of polynomial are the eigenvalues of A with algebraic multiplicity ki
EigenvectorsSolve linear equation systems for all eigenvalues
Dimension of solution space is called geometric multiplicity gi
Eigenvectors belonging to different eigenvalues are linearly independent
Diagonalization of a matrix ADefine the matrix and the diagonal matrix
Only possible for linearly independent eigenvectors
Eigenvalues and Eigenvectors (1)
Multiple-Input Multiple Output Systems
[ ]1 n=X x x
( )− λ =A I x 0⇔
( ) ( ) ( )1
1( ) det 0lk k
lp λ = − λ = λ − λ λ − λ =A A I
iλ
n n×
( )i i− λ =A I x 0
≠x 0
( )1 i ig k≤ ≤
= λAx x
1diag( , , )n= λ λΛ …1−= ⇒ =AX XΛ X AX Λ
30UniversitätBremen
Volker Kühn Time-Discrete Signals and Systems
Some useful general properties
Properties for hermitian matricesAll eigenvalues are realEigenvectors belonging to different eigenvalues are orthogonal
Algebraic and geometric multiplicities are identical
Consequence: all eigenvectors can be chosen to be mutually orthogonalA hermitian matrix A can be diagonalized by a unitary matrix V
Eigenvalues and Eigenvectors (2)
Multiple-Input Multiple Output Systems
H H= ⇔ =V AV Λ A VΛV
*
1
,
,
,
,
Ti
Hi
i i
m mi i
i i
i i− −
→ λ
→ λα → αλ
→ λ+ β → λ + β
→ λ 1
A
A
A x
A x
A I x
X AX X x
1
1
det
trace
n
ii
n
ii
=
=
= λ
= λ
∏
∑
A
A
0
0
invertible all
positive definite all i
i
⇔ λ ≠⇔ λ >
A
A
Eigenvalue decomposition
31UniversitätBremen
Volker Kühn Time-Discrete Signals and Systems
Every matrix A of rank r can be written as
Singular values of A = square roots of nonzero eigenvalues of orUnitary matrix U contains left singular vectors of A = eigenvectors of
Unitary matrix V contains right singular vectors of A = eigenvectors of
Verification with eigenvalue decomposition
Four fundamental subspaces: the vectorsu1,...,ur span the column space of A
ur+1,...,um span the left nullspace of Av1,...,vr span the row space of A
vr+1,...,vn span the right nullspace of A
Singular Value Decomposition (SVD) (1)
Multiple-Input Multiple Output Systems
HA A HAA
m n×0H H
= =
Σ 0A UΣV U V
0 0
20H H H H H
= =
Σ 0A A VΣ U UΣV V V
0 0
20H H H H
= =
Σ 0AA UΣV VΣU U U
0 0
0 1diag( , , )r= σ σΣ …with the matrix of singular values
iσHAA
HA Am m×n n×
orthogonal
orthogonal
32UniversitätBremen
Volker Kühn Time-Discrete Signals and Systems
Illustration of the fundamental subspacesConsider linear mapping with orthogonal decomposition
Singular Value Decomposition (SVD) (2)
Multiple-Input Multiple Output Systems
right nullspace
rowsp
ace
left n
ullsp
ace
columnspace
x
rx
nx
0 n =Ax 0
r=Ax Ax
→x Ax r n= +x x x
33UniversitätBremen
Volker Kühn Time-Discrete Signals and Systems
Inverse A-1 exists only for square matrices with full rankGeneralization: (Moore-Penrose) pseudo inverse A+
Special cases for full rank matrices
Application: Least squares solution of a linear equation systemProblem: find vector x that minimizes the euclidean distance between Ax and b
Solution: project b onto the column space of A and solve Ax=bc
If no unique solution exists take solution vector with shortest length
Pseudo Inverse and Least Squares Solution (1)
Multiple-Input Multiple Output Systems
10H H−
+ + = =
Σ 0A VΣ U V U
0 00H H
= =
Σ 0A UΣV U V
0 0⇒
( ) { }
( ) { }
1
1
rank
rank
for
for
H H
H H
m
n
−
+−
== =
A AA AA
A A A A
min −x
Ax b +=x A b⇒
34UniversitätBremen
Volker Kühn Time-Discrete Signals and Systems
Illustration of the least squares solution of a linear equation system
Pseudo Inverse and Least Squares Solution (2)
Multiple-Input Multiple Output Systems
right nullspace
rowsp
ace
left n
ullsp
ace
columnspace
+=x A b
n+ =A b 0 0
c =b Ax
nb
b