Overview Principal Component Analysis Independent ... · Principal Component Analysis & Independent Component Analysis Overview ... NNM T T T M N xx NM MN xx

Pattern Recognition for VisionFall 2004

Principal Component Analysis & Independent Component Analysis

Overview

Principal Component Analysis•Purpose•Derivation•Examples

Independent Component Analysis•Generative Model•Purpose•Restrictions•Computation•Example

Literature


Notation & Basics

Vector Matrix

, , Dot product written as matrix product

Product of a row vector with

T N

a

∈

uUu v u v R

u

22

scalar as matrix product, and not

squared normRules for matrix multiplication:

( ) ( )( )( )

T

T T T

a

= =

= =+ = +

=

u

u u u u

UVW UV W U VWU V W UW VWUV V U


Principal Component Analysis (PCA)

PurposeFor a set of samples of a random vector

,discover or reduce the dimensionality and identify meaningful variables.

N∈x R

1u

2u 1u

, : ,p N p N= × <y Ux U



PCA by Variance Maximization

AuBu

2 2A B

σ σ>u u

{ } { }{ } { }

{ }

1

1

1

1

2 21 1

21 1 1 1

21 1

1

Find the vector ,such that the variance of the data along this direction is maximized:

( ) , 0, 1

( )( ) ,

,

The solution is the eigenvector of wi

T

T T T T

T T

E E

E E

E

σ

σ

σ

= = =

= =

= =

u

u

u

u

u x x u

u x x u u xx u

u Cu C xx

e C

1

12

1 1 1 1 1 1 1

th the largesteigenvalue .

, T

λ

λ λ σ λ= = ⇔ =uCe e e Ce



{ }

For a given , find orthonormal basis vectors such that the variance of the data along these vectors is maximally large, under the constraint of decorrelation:

( )( ) 0, 0

The sol

i

T T Ti n i n

p N p

E n p

<

= = ≠

u

u x u x u u

{ } { }

1 1 2 2 1 2

orthogonal

ution are the eigenvectors of ordered according to decreasing eigenvalues :

, ,..., , ...

Proof of decorrelation for eigenvectors:

( )( )

p p p

T T T T T Ti n i n i n i n nE E

λ

λ λ λ

λ

= = = > >

= = = =

C

u e u e u e

e x e x e xx e e Ce e e 0

PCA by Variance Maximization



PCA by Mean Square Error Compression

x

x̂

ku

{ } ( )2

1

For a given , find orthonormal basis vectors such that ˆthe between and its projection into the subspace spanned

by the orthonormal basis vectors is mimimum:

ˆ ˆ, ,p

Ti k k k

k

p N pmse

p

mse E=

<

= − =∑

x x

x x x u x u u ,Tk m k mδ=u



{ }

{ }

{ }

{ }( )

2

2

1 scalar

2

1 1 1

2 2 2

1 1

2 2

1

1

ˆ ( )

2 ( ) ( ) ( )

2 ( ) ( )

( )

trace ,

pT

k kk

p p pT T T T T

k k n n k kk n k

p pT T

k kk k

pT

kk

pTk k

k

mse E E

E E E

E E E

E E

=

= = =

= =

=

=

= − = −

= − +

= − +

= −

= −

∑

∑ ∑ ∑

∑ ∑

∑

∑

x x x u x u

x x u x u x u u u x u

x x u x u

x x u

C u Cu C { }TE= xx



( ) { }1

maximize

trace ,P

T Tk k

k

mse E=

= − =∑C u Cu C xx

1

Solution to minimizing is any (orthonormal) basis of the subspace spanned by the first eigenvectors

,..., of .p

msep

e e C

( )1 1

tracep N

k kk k p

mse λ λ= = +

= − =∑ ∑C

11

The is the sum of the eigenvalues corresponding to

the discarded eigenvectors ,..., of : N

P N kk p

mse

mse λ+= +

= ∑e e C


Principal Component Analysis (PCA)How to determine the number of principal components ?p

{ } { }{ } { } { } { }

1,1 1,

,1 ,

2

0

Linear signal model with unknown number of signals:

,

Signal have 0 mean and are uncorrelated, is white noise:

,

( )

p

N N p

i

T Tn

T T T T

p N

a aN p

a a

s

E E

E E E E

σ

=

<

= + = ×

= =

= = + +

x As n A

n

ss I nn I

C xx As As nn Asn

{ } { } 2

21 2 1 2... ...

cut off when eigenvalues become constants

T T T Tn

p p p N n

E E

d d d d d d

σ

σ+ +

= + = +

> > > > = = = =

→

A ss A nn AA I



Computing the PCA

{ }1

1

1

Given a set of samples ,..., of a random vector calculate mean and covariance.

1 ,

1 ( )( )

Compute eigenvecotors of e.g. with QR algorithm

M

M

ii

MT

i ii

M

M

=

=

= → −

= − −

∑

∑

x x x

µ x x x µ

C x µ x µ

C



Computing the PCA

1,1 1,

,1 ,

If the number of samples is smaller than the dimensionality of :

, , : , :

( ) ( ),

MT T

N N M

T

T

T

MN

x xN M M N

x x

λ

λ

λλ λ

= = × × =

′ ′ ′=

′ ′ ′=′ ′= =

x

B C BB B B

BB e eB Be eBB Be Bee Be 2 2

Reducing complexity from( ) to ( ) O N O M

→



Examples

Eigenfaces for face recognition (Turk&Pentland):

Training:-Calculate the eigenspace for all faces in the training database-Project each face into the eigenspace feature reduction

Classification:-Projec

→

t new face into eigenspace-Nearest neighbor in the eigenspace



Examples cont.

Feature reduction/extraction

Original Reconstruction with 20 PC

http://www.nist.gov/


Independent Component Analysis (ICA)

Generative model

( )( )

1

1

1

1,1 1,

1, ,

,1 ,

Noise free, linear signal model:

,..., Observed variables

,..., latent signals, independent components

unknown mixing matrix, ( ,..., )

N

i ii

TN

TN

NT

i i N i

N N N

s

x x

s s

a aa a

a a

=

= =

=

=

= =

∑x As a

x

s

A a


Independent Component Analysis (ICA)Task

For the linear, noise free signal model, compute and given the measurements .

A sx

1 2 3

1 2 3

Blind source separation :separate the three original signals

, , and from their mixtures , , and .

s s sx x x

Figure by MIT OCW.

S3

S2

S1

X1

X2

X3

S1

S 1

S1



Restrictions

{ } { } { } { }

1 2 1 1 2 2

1 1 2 2 1 1 2 2

1.) Statistical independenceThe signals must be statistically independent:

( , ,..., ) ( ) ( )... ( )Independent variables satisfy:

( ) ( )... ( ) ( ) ( ) ... ( )

for any

i

N N N

N N N N

sp s s s p s p s p s

E g s g s g s E g s E g s E g s

g

=

=

{ }

{ } { }

1

1 1 2 2 1 1 2 2 1 2 1 2

1 1 1 1 2 2 2 2 1 1 2 2

( )

( ) ( ) ( ) ( ) ( , )

( ) ( ) ( ) ( ) ( ) ( )

i s L

E g s g s g s g s p s s ds ds

g s p s ds g s p s ds E g s E g s

∞ ∞

−∞ −∞

∞ ∞

−∞ −∞

∈

=

= =

∫ ∫∫ ∫



Restrictions

{ } { } { } { }

{ } { } { }

1 1 2 2 1 1 2 2

Statistical independence cont.

( ) ( )... ( ) ( ) ( ) ... ( )Independence includes uncorrelatedness:

( )( ) ( ) ( ) 0

N N N N

i i n n i i n n

E g s g s g s E g s E g s E g s

E s s E s E sµ µ µ µ

=

− − = − − =



Restrictions

2 21 2

1 2 2 21 2 1 2

2.) Nongaussian componentsThe components must have a nongaussian distributionotherwise there is no unique solution.Example:given and two gaussian signals:

1( , ) exp( )2 2 2

gene

is

s sp s sπσ σ σ σ

= − −

A

2 21 1 2

1 22

Scaling matrix

rate new signals1/ 0 1( , ) exp( )exp( )

0 1/ 2 2 2s sp s s

σσ π

′ ′ ′ ′ ′= ⇒ = − −

S

s s

1σ

2σ

11



Restrictions

2 21 2

1 2

Rotation matrix

Nongaussian components cont.

under rotation the components remain independent:cos sin 1, ( , ) exp( )exp( )sin cos 2 2 2

combine whitening and rotatio

s sp s sϕ ϕϕ ϕ π

′′ ′′ ′′ ′ ′′ ′′= = − − − R

s s

1

1

n :

is also a solution to the ICA problem.

−

−

=

′′= =

B RSx As AB sAB



Restrictions

3.) Mixing matrix must be invertibleThe number of independent components is equal to the number of observerd variables.Which means that there are no redundant mixtures.

In case mixing matrix is not invertible apply PCA on measurements first to remove redundancy.



Ambiguities

{ }1 1

2

1.) Scale1( )( )

Reduce ambiguity by enforcing 1

2.) OrderWe cannot determine an order of the independant components

N N

i i i i ii i i

i

s s

E s

αα= =

= =

=

∑ ∑x a a



Computing ICA

1

1

2

a) Minimizing mutual information: ˆˆ

ˆ ˆˆMutual information: ( ) ( ) ( )

is the differential etnropy:

ˆ ˆ ˆ ˆ( ) ( ) log ( ( ))

ˆ is always nonnegative and 0 only if the areindepende

N

ii

i

I H s H

H

H p p d

I s

−

=

=

= −

= −

∑

∫

s A x

s s

s s s s

1

nt.ˆ ˆIteratively modify such that ( ) is minimized.I−A s



Computing ICA cont.

b) Maximizing Nongaussianity1

introduce and :From central limit theorem:

is more gaussian than any of the and becomes least gaussian if .

Iteratively modify such that the "gaussianity"of is m

T T T

Ti

i

T

y y

y sy s

y

−=

= = =

=

=

s A xb b x b As q s

q s

b

1

inimized. When a local minimum is reached, is a row vector of .T −b A


PCA Applied to Faces

1x

Mx

1,1x

,1Nx

1,Mx

,N Mx

1u

2u

Each pixel is a feature, each face image a point in the feature space.Dimension of feature vector is given by the size of the image.

1x

2x

3x

1x

Mx

1

are the eigenvectors which can berepresented as pixel images in the originalcooridinate system ...

i

Nx x

u

…


ICA Applied to Faces

…

1,1x

1,Mx

,1Nx

,N Mx

Nx1x

Now each image corresponds to a particular observed variable measured over time (M samples). N is the number of images.

( )( )

1

1

1

,..., Observed variables

,..., latent signals, independent components

N

i ii

TN

TN

s

x x

s s

=

= =

=

=

∑x As a

x

s

S1

S1


PCA and ICA for Faces

Features for face recognition

Image removed due to copyright considerations. See Figure 1 in: Baek, Kyungim et. al. "PCA vs. ICA: A comparison on the FERET data set." International Conference of Computer Vision, Pattern Recognition, and Image Processing, in conjunction with the 6th JCIS. Durham, NC, March 8-14 2002, June 2001.

Documents

Overview Principal Component Analysis Independent ... · Principal Component Analysis & Independent Component Analysis Overview ... NNM T T T M N xx NM MN xx