Lec 13: Low Dimension Embedding · PDF file Laplacian Eigenmap direct data embedding without explicit projection input data -> affinity graph -> graph Laplacian eigenvectors - > embedding

Spring 2020: Venu: Haag 315, Time: M/W 4-5:15pm

ECE 5582 Computer Vision Lec 13: Low Dimension Embedding

Zhu Li Dept of CSEE, UMKC

Office: FH560E, Email: [email protected], Ph: x 2346. http://l.web.umkc.edu/lizhu

Z. Li: ECE 5582 Computer Vision, 2020 p.1

slides created with WPS Office Linux and EqualX LaTex equation editor

Outline

 Recap:  Part I

 Linear Algebra Refresher  SVD and Principal Component Analysis (PCA)  Laplacian Eigen Map (LEM)  Stochastic Neigborhood Embedding (SNE)

Z. Li: ECE 5582 Computer Vision, 2020 p.2

Handcrafted Feature Pipeline

 An image retrieval pipeline (hand crafted features)

p.3

Image Formation

Feature Computing

Feature Aggregation

Classification

Knowledge /Data Base

Z. Li: ECE 5582 Computer Vision, 2020

Homography, Color space

Color histogram Filtering, Edge Detection HoG, Harris Detector, LoG Scale space, SIFT

Bow VLAD Fisher Vector Supervector

TPR, FPR, Precision, Recall, mAP

kNN, Bayesian SVM, Kernel Machine

Vector and Matrix Notations

 Vector

Matrix

p.4Z. Li: ECE 5582 Computer Vision, 2020

Vector Products

 Inner Product

Outer Product

p.5Z. Li: ECE 5582 Computer Vision, 2020

Matrix-Vector Product

 y=Ax

 So, y is a linear combination of basis {ak} with weights from x

p.6Z. Li: ECE 5582 Computer Vision, 2020

Matrix Product

 C=AB

 Associative:  ABC = (AB)C = A(BC)

Distributive:  A(B+C) = AB + AC

p.7

A: nxp B: pxm A: nxm=

Z. Li: ECE 5582 Computer Vision, 2020

Outer Product/Kron

 Vector outer product:

Example

Z. Li: ECE 5582 Computer Vision, 2020 p.8

Matrix Transpose

 Transpose

p.9Z. Li: ECE 5582 Computer Vision, 2020

Matrix Trace and Determinant

 Trace:Tr(A): only for nxn square matrix

 Determinant: Det(A):  The size of volumes spanned by A,  All possible linear combinations of a1 and a2

p.10

Det(A) = |2-9| = 7;

Z. Li: ECE 5582 Computer Vision, 2020

Eigen Values and Eigen Vectors

 Definition: for nxn matrix A:

 In Matlab: [P, V]=eig(A);

p.11Z. Li: ECE 5582 Computer Vision, 2020

Eigen Vectors of Symmetric Matrix

 If square matrix A: nxn is symmetric A=AT

Then its Eigen Values are real, and Eigen Vectors are othonormal:

� = �X�� where S is a diagonal matrix with eigen values of A.

Application: solution to the Quadratic form maximization:

will be the largest eigen value, and x* will be the corresponding eigen vector of A.

p.12Z. Li: ECE 5582 Computer Vision, 2020

SVD  for non square matrix: A mxn:

p.13Z. Li: ECE 5582 Computer Vision, 2020

� = ���

��

Σ

SVD as Signal Expansion The Singular Value Decomposition (SVD) of an nxm matrix A, is,

 Where the diagonal of S are the eigen values of AAT, [��,��,…,  ��], called “singular values”  U are eigenvectors of AAT, and V are eigen vectors of ATA,

the outer product of uiviT, are basis of A in reconstruction:

p.14

� = �X�� =� � ������

�

A(mxn) = U(mxm) S(mxn) V(nxn)

The 1st order SVD approx. of A is:

�� ∗��: , 1� ∗��: , 1��

Z. Li: ECE 5582 Computer Vision, 2020

SVD approximation of an image

 Very easy… function [x]=svd_approx(x0, k) dbg=0; if dbg x0= fix(100*randn(4,6)); k=2; end [u, s, v]=svd(x0); [m, n]=size(s); x = zeros(m, n); sgm = diag(s); for j=1:k x = x + sgm(j)*u(:,j)*v(:,j)'; end

p.15Z. Li: ECE 5582 Computer Vision, 2020

SVD for Separable Filtering

 Take LoG filter for eg.  h=fspecial('LoG', 11, 2.0);  [u,s,v]=svd(h); h1=s(1,1)*u(:,1)*v(:,1)';

Z. Li: ECE 5582 Computer Vision, 2020 p.16

h1 is 1-SVD approx of LoG

Many implications for Deep networks acceleration !

Norm Vector Norm: Length of the vector  Euclidean Norm (L2 Norm): norm(x, 2)

 Lp norm:

Matrix Norm: Forbenius Norm

p.17Z. Li: ECE 5582 Computer Vision, 2020

Quadratic Form

 Quadratic form f(x)=xTAx in R:

Positive Definite (PD):  For non-zero x, xTAx > 0

Positive Semi-Definite (PSD):  For non-zero x, xTAx >= 0

 Indefinite:  Exists x1, x2 non zero, but x1TAx1 >0, while x2TAx2 < 0;

p.18Z. Li: ECE 5582 Computer Vision, 2020

Matrix Calculus

 Gradient of f(A):

 Matrix Gradient Properties

p.19Z. Li: ECE 5582 Computer Vision, 2020

Hessian of f(X)

 For function:�:�� →�

 Gradient & Hessian of Quadratic Form: f(x)= xTAx

p.20Z. Li: ECE 5582 Computer Vision, 2020

PCA -Dimension Reduction

 A typical image retrieval pipeline

Z. Li: ECE 5582 Computer Vision, 2020 p.21

Image Formation

Feature Computing

Feature Aggregation

Classification

Knowledge /Data Base

e.g, dense SIFT: 12000 x 128 e.g, Fisher Vector: k=64, d=128

Rd -> Rp

Outline

 Recap:  Part I

 Linear Algebra Refresher  SVD and Principal Component Analysis (PCA)  Laplacian Eigen Map (LEM)  Stochastic Neigborhood Embedding (SNE)

Z. Li: ECE 5582 Computer Vision, 2020 p.22

Principal Component Analysis

 The formulation:  for data points {x1, x2,…, } in Rn, find a lower dimensional

representation in Rm, via a projection W,: mxn, s.t., the energy of the data is preserved

Z. Li: ECE 5582 Computer Vision, 2020 p.23

PCA solution

 Take the Lagrangian of the problem

 Take the derivative w.r.t. to w, and KKT condition gives us,

 This is an Eigen problem, finding projection s.t. it is just a scaling along the scatter matrix eigen vectors.

Z. Li: ECE 5582 Computer Vision, 2020 p.24

X� = ��

PCA – how to compute

 PCA via SVD on the Covariance matrix

Z. Li: ECE 5582 Computer Vision, 2020 p.25

S: covariance, nxn

2d Data

2.5 3 3.5 4 4.5 5 5.5 6 6.5 7 7.5 -2

0

2

4

6

8

10

Z. Li: ECE 5582 Computer Vision, 2020 p.26

Principal Components

-5 -4 -3 -2 -1 0 1 2 3 4 5 -5

-4

-3

-2

-1

0

1

2

3

4

5

1st principal vector

2nd principal vector

Gives best axis to project Minimum RMS

error Principal vectors

are orthogonal

Z. Li: ECE 5582 Computer Vision, 2020 p.27

PCA on HoGs

 Matlab Implementation of PCA:  [A, s, eig_values]=princomp(hogs);

Z. Li: ECE 5582 Computer Vision, 2020 p.28

HoG basis function

PCA Application in Aggregation

 SIFT aggregation  Usually a PCA is

done on SIFT features, to reduce the dimension from 128 to say 24, 32.  Then a GMM is

trained in R32 space, for FV encoding

 Homework-2 Aggregation  Fisher Vector

Aggregation of SIFT

Z. Li: ECE 5582 Computer Vision, 2020 p.29

load../../dataset/cdvs_sift_aggregation_te st_data.mat;

[n_sift, kd_sift]=size(gd_sift_cdvs); offs = randperm(n_sift); offs = offs(1:200*2^10); % PCA [A1, s1, lat1]=princomp(double(gd_sift_cdvs(off s,:)));

figure(41); hold on; grid on; stem(lat1, '.'); title('sift pca eigen values');

SIFT PCA

Eigen values

 That is why we have kd=[24, 32 48] for SIFT GMM in FV aggregation

Z. Li: ECE 5582 Computer Vision, 2020 p.30

SIFT PCA Basis Functions

 Capturing max variation directions

Z. Li: ECE 5582 Computer Vision, 2020 p.31

Visualizing SIFT in lower dimensional space

 Project SIFTs from 2 images to 2D space

Z. Li: ECE 5582 Computer Vision, 2020 p.32

Laplacian Eigen Map

 Directly compute an embedding {yk} from input {xk} in RD without the explicit projection model A, s.t. Y=AX  Objective function

where the nxn affinity matrix W reflects the data points relationship in the original space X.

Z. Li: ECE 5582 Computer Vision, 2020 p.33

M. Belkin and P. Niyogi. Laplacian Eigenmaps and spectral techniques for embedding and clustering. In Advances in Neural Information Processing Systems, volume 14, pages 585–591, Cambridge, MA, USA, 2002. The MIT Press

Graph Laplacian

Graph Laplacian: L= D - W

p.34Z. Li: ECE 5582 Computer Vision, 2020

Laplacian Eigenmap

 Minimizing the following

Is equivalent to

Where D is the degree matrix (diagonal) with

Z. Li: ECE 55