MIMO Algebra

Time-Discrete Signals and SystemsAdaptation Course for Master Studies

Dr.-Ing. Volker KühnInstitute for Telecommunications and High-Frequency Techniques

Department of Communications EngineeringRoom: N2300, Phone: 0421/218-2407

[email protected]

Lecture

Thursday, 08:30 – 10:00 in N1170

Dates for exercises will be announced during lectures.

Tutors

Ronald BöhnkeRoom: N2380

Phone [email protected]

www.ant.uni-bremen.de/teaching

2UniversitätBremen

Volker Kühn Time-Discrete Signals and Systems

OutlineSummary of Laplace and Fourier TransformationZ-Transformation

Time-discrete SignalsProperties of Z-Transformation and Convergence System description by Z-transformation

Stochastic ProcessesCharacterization of stochastic processes and random variablesProbabilities, densities, distributions, moments, stationary and ergodic processesCentral limit theoremCorrelation and spectral power density (real and complex signals)System analysis for stochastic input signals (Theorem of Wiener-Khintchine)

Multiple-Input Multiple-Output SystemsSystem DescriptionLinear Algebra (eigenvalues and eigenvectors, pseudo inverse) -- briefDecompositions (QR, unitary matrices, singular value, Cholesky )Statistical representation (multivariate distributions)

Outline

3UniversitätBremen


Linear Algebra

Notations and definitionsVectors and matrices

Elementary operations, Matrix multiplicationDeterminants

Special MatricesSymmetric, orthogonal, complex, circulant, Toeplitz...

Linear equation systemsGaussian elimination, Cramer’s rule, iterative methods

Matrix factorizationsLU, Cholesky, QR (Householder, Givens, Gram-Schmidt)

Eigenvalues and eigenvectors, SVD (condition, pseudo-inverse)

Least squares, matrix inversion lemma

Multiple-Input Multiple Output Systems

4UniversitätBremen


Notations and Definitions (1)

VectorsColumn vectors (preferred): boldface lower case

Row vectors: underlined boldface lower case

MatricesBoldface capital letters

Column vectors are just matrices

Row vectors are just matrices


1

2

n

x

x

x

=

x

[ ]1,1 1,2 1, 1

2,1 2,2 2, 21 2

,1 ,2 ,

n

nn

m m m n m

a a a

a a a

a a a

= = =

a

aA a a a

a

[ ]1 2 nx x x=x

1m×1 n×

5UniversitätBremen


Notations and Definitions (2)

Some special matricesIdentity matrix and zero matrix

Diagonal, lower and upper triangular matrices


1 0 0

0 1 0

0 0 1

=

I

1

2

0 0

0 0

0 0 n

d

d

d

=

D

1,1

2,1 2,2

,1 ,2 ,

0 0

0

n n n n

l

l l

l l l

=

L

1,1 1,2 1,

2,2 2,

,

0

0 0

n

n

n n

u u u

u u

u

=

U

0 0 0

0 0 0

0 0 0

=

0

6UniversitätBremen


Let A, B, C be matrices and , be scalarsAddition and scalar multiplication are defined element-wise

PropertiesAddition commutativeAddition associative

Neutral element of addition

Inverse element of additionScalar multiplication associative

Neutral element of scalar multiplication

Scalar multiplication distributiveScalar multiplication distributive

Basic Operations and Properties


+ = +A B B A

m n×

( ) ( )+ + = + +A B C A B C

α β

1,1 1,1 1, 1,

,1 ,1 , ,

n n

m m m n m n

a b a b

a b a b

+ + + = + +

A B1,1 1,

,1 ,

n

m m n

a a

a a

α α α = α α

A

( )α + = α + αA B A B

( ) ( )αβ = α βA A

( )α + β = α + βA A A

+ =A 0 A( )+ − =A A 0

1 =A A

7UniversitätBremen


Let A be a matrix and B be a matrix

The product C=AB is a matrix with elements“row times column”

Note: number of columns of A has to equal number of rows of BEquivalent formulations of the matrix multiplication:

Matrix Multiplication (1)


m n× n p×

m p× , , ,1

n

i j i k k jk

c a b=

= ⋅∑

[ ]1,1 1, 1

1

,1 ,

n

n

m m n m

a a

a a

= = =

a

A a a

a

1,1 1, 1

1

,1 ,

p

p

n n p n

b b

b b

= = =

b

B b b

b

1, ,1 1, ,1 1 1 1 1 1

11

1, ,1 , ,

1 1

n n

k k k k pk k p n

p k kkn n

m m p mm k k m k k p

k k

a b a b

a b a b

= =

=

= =

⋅ ⋅ = = = = = ⋅ ⋅

∑ ∑∑

∑ ∑

a b a b a B

C Ab Ab a b

a b a b a B

8UniversitätBremen


Special casesm=1, n>1, p=1 (row vector times column vector)

m=1, n>1, p>1 (row vector times matrix)

m>1, n>1, p=1 (matrix times column vector)

m>1, n=1, p>1 (column vector times row vector)



1

n

k kk

c a b=

= =∑ab

1

n

k kk

a=

= =∑c aB b

1

n

k kk

b=

= =∑c Ab a

1,1 1,1 1,1 1,

,1 1,1 ,1 1,

p

m m p

a b a b

a b a b

= = ⋅

C ab

scalar

row vector

column vector

matrix

Inner or scalar product

Outer or dyadic product

Matrix-vector products

9UniversitätBremen


PropertiesMatrix multiplication distributive

Matrix multiplication distributive Mixed scalar / matrix multiplication associative

Matrix multiplication associative

Note: matrix multiplication is not commutative in generalExample



( ) ( )=AB C A BC

( )+ = +A B C AC BC

( )+ = +A B C AB AC( ) ( ) ( )α = α = αAB A B A B

2 6

1 7

=

A3 1

2 1

− − =

B15 6

1 20

=

C

6 4

11 6

=

AB7 25

5 19

− − =

BA

36 132

22 146

=

AC36 132

22 146

=

CA

≠AB BA

=AC CA

⇒

⇒

10UniversitätBremen


Transpose of a matrix

Row vectors become column vectors and vice versa

Hermitian transpose of a complex matrix

Transpose of the complex conjugate matrix

Properties

and

Transpose and Hermitian Transpose


( )T T T+ = +A B A B( )TT =A A

( )T T T=AB B A

[ ]1,1 1, 1

1

,1 ,

n

n

m m n m

a a

a a

= = =

a

A a a

a

( )* * * *1,1 1, 1,1 ,1 1

*1

* * * *,1 , 1, ,

T Hn m

TH H Hm

Hm m n n m n n

a a a a

a a a a

= = = = =

a

A A a a

a

( )H H H+ = +A B A B( )HH =A A

( )H H H=AB B A

1,1 ,1 1

1

1, ,

Tm

T T Tm

Tn m n n

a a

a a

= = =

a

A a a

a

⇒



Determinant of a matrix

Determinant of a matrix (Sarrus’ rule)

Determinant of a matrixLet the matrix Ai,j equal A without the i-th row and j-th column

Recursive definition of determinant by cofactor expansion

1,1 1,21,1 2,2 2,1 1,2

2,1 2,2

deta a

a a a aa a

= = −A

Determinants (1)


2 2×

3 3×

1,1 1,2 1,3

2,1 2,2 2,3 1,1 2,2 3,3 1,2 2,3 3,1 1,3 2,1 3,2

3,1 3,2 3,3 3,1 2,2 1,3 3,2 2,3 1,1 3,3 2,

1,1 1,2

2,1 2,2

3 1 1,1 ,, 23 2

det

a a a

a a a a a a a a a a a a

a a a a a a a a a a a

a a

a

a a

a

a

= = + +− − −

A

n n×( ) ( )1 1n n− × −

, ,1

det ( 1) detn

i ji j i j

i

a+

=

= −∑A A , ,1

det ( 1) detn

i ji j i j

j

a+

=

= −∑A A

column expansion row expansion



Fundamental propertiesLinearity in columns (rows)

Exchanging two columns (rows)Determinant of identity matrix

Some additional propertiesSymmetry in columns and rows

Zero column (row)

Two equal columns (rows)Multiple of one column (row)

Scalar multiplication

Adding two columns (rows)Determinant of matrix product

All properties valid for arbitrary matrices

Determinants (2)


det detT =A A

( )det det det= ⋅AB A B

( )det detnα = αA A

1 1 2 1 2 1 2′ ′ ′ ′α + α = α ⋅ + α ⋅a a a a a a a

2 1 1 2= −a a a adet 1=I

2 0=0 a

1 1 0=a a

1 2 1 2α = α ⋅a a a a

1 2 2 det+ α =a a a A

n n×



Full expansion of the determinant by recursive application of the definition

Sum over all n! vectors p containing permutations of 1,2,...,n

per{p} is the number of pairwise exchange operations needed for the permutation

Determinant of diagonal or triangular matrixAt least one factor is zero for all

Efficient calculation of determinantDeterminant unaffected by adding multiples of rows (columns) to rows (columns)

Transform A into triangular matrix by elementary row (column) operations

Determinants (3)


{ } { }! !

1 1

per per, ,

1 1

det ( 1) ( 1)n n

i i

n n

i p p ii i

a a= == =

= − = −∑ ∑∏ ∏p p

p p

p p p p

A

,1

detn

i ii

l=

= ∏L ,1

detn

i ii

u=

= ∏U,

1

detn

i ii

d=

= ∏D

[ ]1 2T

n≠p



System of m linear equations in n unknowns

Matrix-vector notation

Geometric interpretationsx is the intersection of m hyperplanesb is a linear combination of the column vectors

Linear Equation Systems (1)


1,1 1 1,2 2 1, 1

2,1 1 2,2 2 2, 2

,1 1 ,2 2 ,

n n

n n

m m m n n m

a x a x a x b

a x a x a x b

a x a x a x b

+ + + =+ + + =

+ + + =

1,1 1,2 1, 1 1

2,1 2,2 2, 2 2

,1 ,2 ,

n

n

m m m n n m

a a a x b

a a a x b

a a a x b

⋅ =

1,1 1,2 1, 1

2,1 2,2 2, 2

,1 ,2 ,

n

n

m m m n m

a a a b

a a a b

a a a b

=Ax b

i ib=a x

1

n

i ii

x=

=∑ a b

⇔

Extended coefficient matrix



Illustration for system (hyperplanes straight lines)



1a

2a

1 1x a

2 2x a

b

1a2a

b

1a 2ab

1 1b=a x

2 2b=a x1 1b=a x

2 2b=a x1 1b=a x

2 2b=a x

1x

2x

1x

2x

1x

2x

2 2×

intersecting straight lines parallel straight lines identical straight lines

a1, a2 linearly independent a1, a2 parallel a1, a2, b parallel

unique solution no solution infinite number of solutions



Square linear equation system Ax=b with n equations in n unkownsCramer’s rule

Let Aj equal A with the j-th column replaced by b

Then the j-th element of x is

Proof: substitute into Aj and use linearity in columns

Three possibilitiesunique solutionno solution

infinite number of solutions



1 1 1j j j n− + = A a a b a a

1

n

i ii

x=

=∑b a

det

detj

jx =A

A

det 0≠A

det 0 det 0 and for all j j= =A A

det 0 det 0 and for some j j= ≠A A



Example: system

Gaussian Elimination (1)


1,2 1,3 1

2,2 2,3 2

3,2 3,3 3

1,1 1,2 1,3 1(1) (1)2,3 2(1) (1)3,3 3

1,

1,1

(1)2,2

(2)3,

1 1,2 1,3 1(1) (1) (1)

2,1

3,1

(1)3,

2,2 2,3 2(233

)

2

0

0

0

0 0

a a b

a a b

a a b

a a a b

a b

a b

a

a

a a b

a a b

ba

a

a

a

a−

−

−

( )( )

(2) (2)3 3 3,3

(1) (1) (1)2 2 2,3 3 2,2

1 1 1,2 2 1,3 3 1,1

/

/

/

x b a

x b a x a

x b a x a x a

=

= −

= − −

(1) EliminationSubtracting multiples of rows to create zerosTransform system into upper triangular form

(2) Back-substitutionSolve for unknownsComputation in reverse order

3 3×

Pivot elements

Extension to (1): If If for some k > j

exchange rows

If for all k > jmove to next column

( 1), 0j

j ja − =( 1), 0j

k ja − ≠

( 1), 0j

k ja − =Reduced systems

2,1,1 1,12 /l aa⋅ =

3,1,1 1,13 /l aa⋅ =

(1)3,23

(1,2)

, 2 2/l aa⋅ =



Special casesAll diagonal elements nonzero

Zero row in coefficient matrix, corresponding right hand side nonzero

Zero rows in coefficient matrix, corresponding right hand sides zero



0

0 0

• ∗ ∗ ∗• ∗ ∗

• ∗

0

0 0 0

∗ ∗ ∗ ∗∗ ∗ ∗

•

0

0 0 0 0

• ∗ ∗ ∗• ∗ ∗ 0 0

0 0 0 0

• ∗ ∗ ∗• ∗ 0 0 0 0

0 0 0 0

• ∗ ∗ ∗

unique solution

no solution

0 0 0 0

0 0 0 0

0 0 0 0

infinite number of solutions

3x 2x 2x 3x 3x2x1x Free parameters



General formulation of the algorithm(1) Initialization and elimination



(0) (0)

( 1) ( 1), , ,

( ) ( 1) ( 1),

( 1),

, , ,

( ) ( 1) ( 1),

: , :

: 1 1

: 1

/

: 1

for to do

for to

find pivot ele

do

for to do

ment

end

end

en

d

j j

j

j ji j i n j n

j

j j ji k i k i j j k

j j ji i

j

i j

j n

j

j m

i j m

l a a

k n n

a a l a

b b b

a

l

− −

− −

−

−

−

= == −

= +=

= +

= − ⋅

= − ⋅

A A b b

( 1) ( 1), ( 1)

1 ,

: 1

1

choose values for free parameters

for downto do

end

j

j j

nj j

n j j k k jk n j n

j r

x b a xa

− −−

= +

=

= − ⋅ ⋅

∑

(2) Consistency check

(3) Back-substitution

( 1),

:

: 1

0

index of first nonzero column

if no then , break

exchange rows, so that j

j

j

jj n

n

n r j

a −

=

= −

≠

( )if 0 for some then stoprkb k r≠ >

Pivot search



Result after Elimination Step

Number of nonzero rows on left hand side: Rank of Matrix A

Solution exists only if orUnique solution if no free parameters

Infinite number of solutions if free parameters



1

2

(0) (0)1, 1

(1) (1)2, 2

( 1) ( 1),

( )1

( )

* * * *

0 * * *

0 0 * *

0 0 0 0

0 0 0 0

r

n

n

r rr n r

rr

rm

a b

a b

a b

b

b

− −

+

{ }rank r=A

r

m r

−

r m= ( ) ( )1 0 and r r

r mr m b b+< = = =…r n=

r n< n r−



Linear equation systemBasic idea of iterative algorithms

Start with initial estimate of the solution vectorFind improved approximation from previous approximationStop after convergence

JacobiSolve row i for unknown xi

Parallel implementation possible

Gauss-SeidelUse already updated valuesBetter convergence behavior than JacobiNo parallel implementation possible

Conjugate GradientMore complicated implementation, but usually fast convergence

Iterative Solution of Linear Equation Systems


,1

1 for n

i j j ij

a x b i n=

= ⇔ = ≤ ≤∑Ax b

(0)x

1( 1) ( ) ( )

, ,1 1 ,

1i nk k k

i i i j j i j jj j i i i

x b a x a xa

−+

= = +

= − − ⋅

∑ ∑

1( 1) ( 1) ( )

, ,1 1 ,

1i nk k k

i i i j j i j jj j i i i

x b a x a xa

−+ +

= = +

= − − ⋅

∑ ∑

( 1)k +x ( )kx



Inverse A-1 of a square matrix A

Relation of inverse to linear equation systems

Calculation of the inverse by Gauss-Jordan methodn simultaneous linear equation systemsForward eliminationBackward elimination

Inverse exists only if has a unique solution ( A nonsingular)Condition:

Properties

Inverse Matrix


1 1− −= =A A AA I

( ) 11 −− =A A

( ) 1 1 1− − −=AB B A

( ) ( )1 1 HH − −=A A

n n×

1− ⇒ A I U L1 1− − ⇒ U L I A

[ ] [ ]1 n = = ⇔Ax Ax AX I A I

=AX I

det 0≠A{ }rank n=A ⇔

1−= ⇔ =Ax b x A b



Every invertible matrix A can be written as the product of a lower triangular matrix L and an upper triangular matrix U

Application: Solution of linear equation systemwith constant coefficient matrix for different right hand sides

Inversion of triangular matrices easy solve and then

Calculation of LU decomposition by Gaussian eliminationForward elimination:L contains factors from the elimination steps

Direct calculation of LU decomposition (example: matrix)

Calculation order:

LU Decomposition


1,1 1,2 1,3 1,1 1,2 1,3 1,1 1,2 1,3

2,1 2,2 2,3 2,1 2,2 2,3 2,1 1,1 2,1 1,2 2,2 2,1 1,3 2,3

3,1 3,2 3,3 3,1 3,2 3,3 3,1 1,1 3,1 1,2 3,2 2,2 3,1 1,3 3,2 2

1 0 0

1 0 0

1 0 0

a a a r r r r r r

a a a l r r l r l r r l r r

a a a l l r l r l r l r l r l r

= ⋅ = + + + + ,3 3,3r

+

=A LU

=Ly b

= =Ax LUx b

=Ux y=Ux y

[ ] 1− = ⇒ A LU I U L( 1) ( 1)

, , ,/j ji j i j j jl a a− −=

3 3×

1,1 1,2 1,3 2,1 3,1 2,2 2,3 3,2 3,3r r r l l r r l r→ → → → → → → →



Let A be hermitianpositive definite

Then A is fully characterized by alower triangular matrix L

Cholesky decomposition

Similar to LU decompositionBut computational complexity reduced by factor 2

Example: matrix

Calculation order:

Cholesky Decomposition


H =A A0H > ∀x Ax x (0)

( 1), ,

( 1) *, , ,

( ) ( 1) *, , , ,

:

: 1

: 1

/

: 1

for to do

for to do

for to do

end

end

end

kk k k k

ki k i k k k

k ki j i j i k j k

k n

l a

i k n

l a l

j k i

a a l l

−

−

−

==

=

= +=

= += − ⋅

A A

2 * *1,1 1,1 2,1 1,1 3,1

1,1 1,2 1,32 2* * *

2,1 2,2 2,3 2,1 1,1 2,1 2,2 2,1 3,1 2,2 3,2

2 2 2* * *3,1 3,2 3,33,1 1,1 3,1 2,1 3,2 2,2 3,1 3,2 3,3

l l l l la a a

a a a l l l l l l l l

a a al l l l l l l l l

= + + + + +

1,1 2,1 3,1 2,2 3,2 3,3l l l l l l→ → → → →

H=A LL

3 3×

Algorithm



Every matrix A can be written as whereQ is a matrix with orthonormal columns

R is an upper triangular matrix

Columns of A are represented in the orthonormal base defined by Q

Illustration for the case

QR Decomposition (1)


m n×

1

0

for

for Hi j

i j

i j

== ≠

q q

n n×

H =Q Q I⇔

m n×=A QR

2m×

1 1,1 1r=a q

2 1,2 1 2,2 2r r= +a q q2q

1q

,1

k

k i k ii

r=

=∑a q

1,2 1r q

2,2 2r q



Calculation of QR decomposition by modified Gram-Schmidt algorithmCalculate length (Euclidean norm) of a1 r1,1

Normalize a1 to have unit length q1

Projection of a2,...,an onto q1 r1,j

Subtract components of a2,...,an parallel to q1

Continue with next column

Q is computed from left to rightR is computed from top to bottomIllustration for the case



(0)

( 1),

( 1),

( 1),

( ) ( 1),

:

: 1

/

: 1

for to do

for to do

end

end

kk k k

kk k k k

H kk i k j

k kj j k i k

k n

r

r

i k n

r

r

−

−

−

−

==

=

== +=

= −

Q A

q

q q

q q

q q q

1 1,1 1r=a q

2 1,2 1 2,2 2r r= +a q q2q

1q 1,2 1r q

2,2 2r q

(1)jq

2m×



Householder reflections: reflect x onto y by multiplication with unitary matrix

Special case: create zeros in a vector

Application to QR decomposition of matrix A



(1 ) Hw= − + ⋅H I uu with and H

Hw

−= =−

x y x uu

x y u x u

=y Hx

x Huu x

Huu y[ ]|| ||

T=y x 0

m n×

: , :

: 1

( : , )

[|| || ]

, ,

( : , : ) ( : , : )

(:, : ) (:, : )

for to do

calculate

end

m

T

H

k n

k m k

w

k m k n k m k n

k m k m

= ==

==

= ⋅= ⋅

R A Q I

x R

y x 0

u H

R H R

Q Q H

Loop through all columns

Initialization

Create zeros below the maindiagonal in k-th column of R

Update unitary matrix Q



Givens rotationsLet equal an identity matrix except for

is unitary and describes a rotation

Special choices for c and s:Linear transformation

Givens rotation can create zero while changing only one other element

Example



( , , )i k θG*, ,

*, ,

cos

sin

i i k k

i k k i

g g c

g g s

= = θ =

− = = θ =

( , , )i k= θ ⋅y G x

2 2 2 2 and i i k k i kc x x x s x x x= + = − +

( , , )i k θG

⇒2 2

, 0, ,i i k k j jy x x y y x j i k= + = = ∀ ≠

31 2 (2,3, )(2,3, ) (1,2, )

* * * * * * * * * * * *

* * * * * * 0 * * 0 * *

* * * 0 * * 0 * * 0 0 *

θθ θ

= → → → =

GG GA R

3 2 1(2,3, ) (1,2, ) (2,3, )= θ θ θR G G G A 1 2 3(2,3, ) (1,2, ) (2,3, )H H H= θ θ θQ G G G

⇒

⇒



Special eigenvalue problem for arbitrary matrices

Condition for existence of nontrivial solutionsCharacteristic polynomial of degree n has to be zero

Zeros of polynomial are the eigenvalues of A with algebraic multiplicity ki

EigenvectorsSolve linear equation systems for all eigenvalues

Dimension of solution space is called geometric multiplicity gi

Eigenvectors belonging to different eigenvalues are linearly independent

Diagonalization of a matrix ADefine the matrix and the diagonal matrix

Only possible for linearly independent eigenvectors

Eigenvalues and Eigenvectors (1)


[ ]1 n=X x x

( )− λ =A I x 0⇔

( ) ( ) ( )1

1( ) det 0lk k

lp λ = − λ = λ − λ λ − λ =A A I

iλ

n n×

( )i i− λ =A I x 0

≠x 0

( )1 i ig k≤ ≤

= λAx x

1diag( , , )n= λ λΛ …1−= ⇒ =AX XΛ X AX Λ



Some useful general properties

Properties for hermitian matricesAll eigenvalues are realEigenvectors belonging to different eigenvalues are orthogonal

Algebraic and geometric multiplicities are identical

Consequence: all eigenvectors can be chosen to be mutually orthogonalA hermitian matrix A can be diagonalized by a unitary matrix V

Eigenvalues and Eigenvectors (2)


H H= ⇔ =V AV Λ A VΛV

*

1

,

,

,

,

Ti

Hi

i i

m mi i

i i

i i− −

→ λ

→ λα → αλ

→ λ+ β → λ + β

→ λ 1

A

A

A x

A x

A I x

X AX X x

1

1

det

trace

n

ii

n

ii

=

=

= λ

= λ

∏

∑

A

A

0

0

invertible all

positive definite all i

i

⇔ λ ≠⇔ λ >

A

A

Eigenvalue decomposition



Every matrix A of rank r can be written as

Singular values of A = square roots of nonzero eigenvalues of orUnitary matrix U contains left singular vectors of A = eigenvectors of

Unitary matrix V contains right singular vectors of A = eigenvectors of

Verification with eigenvalue decomposition

Four fundamental subspaces: the vectorsu1,...,ur span the column space of A

ur+1,...,um span the left nullspace of Av1,...,vr span the row space of A

vr+1,...,vn span the right nullspace of A

Singular Value Decomposition (SVD) (1)


HA A HAA

m n×0H H

= =

Σ 0A UΣV U V

0 0

20H H H H H

= =

Σ 0A A VΣ U UΣV V V

0 0

20H H H H

= =

Σ 0AA UΣV VΣU U U

0 0

0 1diag( , , )r= σ σΣ …with the matrix of singular values

iσHAA

HA Am m×n n×

orthogonal

orthogonal



Illustration of the fundamental subspacesConsider linear mapping with orthogonal decomposition

Singular Value Decomposition (SVD) (2)


right nullspace

rowsp

ace

left n

ullsp

ace

columnspace

x

rx

nx

0 n =Ax 0

r=Ax Ax

→x Ax r n= +x x x



Inverse A-1 exists only for square matrices with full rankGeneralization: (Moore-Penrose) pseudo inverse A+

Special cases for full rank matrices

Application: Least squares solution of a linear equation systemProblem: find vector x that minimizes the euclidean distance between Ax and b

Solution: project b onto the column space of A and solve Ax=bc

If no unique solution exists take solution vector with shortest length

Pseudo Inverse and Least Squares Solution (1)


10H H−

+ + = =

Σ 0A VΣ U V U

0 00H H

= =

Σ 0A UΣV U V

0 0⇒

( ) { }

( ) { }

1

1

rank

rank

for

for

H H

H H

m

n

−

+−

== =

A AA AA

A A A A

min −x

Ax b +=x A b⇒



Illustration of the least squares solution of a linear equation system

Pseudo Inverse and Least Squares Solution (2)


right nullspace

rowsp

ace

left n

ullsp

ace

columnspace

+=x A b

n+ =A b 0 0

c =b Ax

nb

b

Documents

MIMO Algebra