Nonlinear Dimensionality Reduction Approach

(ISOMAP)2006. 2. 28

Young Ki BaikComputer Vision Lab.

Seoul National University

Nonlinear Dimensionality Reduction Approach (ISOMAP)

Computer Vision Lab. SNU

References

A global geometric framework for nonlinear dimensionality reduction

J. B. Tenenbaum, V. De Silva, J. C. Langford (Science 2000)

LLE and Isomap Analysis of Spectra and Colour Images

Dejan Kulpinski (Thesis 1999)

Out-of-Sample Extensions for LLE, Isomap, MDS, Eigenmaps, and Spectral Clustering

Yoshua Bengio et.al. (TR 2003)

Nonlinear Dimensionality Reduction Approach (ISOMAP)

Computer Vision Lab. SNU

Contents

IntroductionPCA and MDS ISOMAPConclusion

Nonlinear Dimensionality Reduction Approach (ISOMAP)

Computer Vision Lab. SNU

Dimensionality Reduction

The goalThe meaningful low-dimensional structures hidden in their high-dimensional observations.

Classical techniquesPCA (Principle Component Analysis)

– preserves the variance

MDS (MultiDimensional Scaling)

- preserves inter-point distance

ISOMAP

LLE (Locally Linear Embedding)

Nonlinear Dimensionality Reduction Approach (ISOMAP)

Computer Vision Lab. SNU

Linear Dimensionality Reduction

PCAFinds a low-dimensional embedding of the data points that best preserves their variance as measured in the high-dimensional input space.

MDSFinds an embedding that preserves the inter-point distances, equivalent to PCA when the distances are Euclidean.

Nonlinear Dimensionality Reduction Approach (ISOMAP)

Computer Vision Lab. SNU

Linear Dimensionality Reduction

MDSdistances

Relation

ijd

)()( 2ji

Tjiij xxxxd

221 ijdA

matrix centering theis H , HAHB

)()( xxxxb jT

iij T

T XX(HX)(HX)Bthen

Nonlinear Dimensionality Reduction Approach (ISOMAP)

Computer Vision Lab. SNU

Nonlinear Dimensionality Reduction

Many data sets contain essential nonlinear structures that invisible to PCA and MDSResort to some nonlinear dimensionality reduction approaches.

Nonlinear Dimensionality Reduction Approach (ISOMAP)

Computer Vision Lab. SNU

ISOMAP

Example of Non-linear structure (Swiss roll)Only the geodesic distances reflect the true low-dimensional geometry of the manifold.

ISOMAP (Isometric feature Mapping)Preserves the intrinsic geometry of the data.Uses the geodesic manifold distances between all pairs.

Nonlinear Dimensionality Reduction Approach (ISOMAP)

Computer Vision Lab. SNU

ISOMAP (Algorithm Description)

Step 1Determining neighboring points within a fixed radius based on the input space distance .These neighborhood relations are represented as a weighted graph G over the data points.

Step 2Estimating the geodesic distances between all pairs of points on the manifold by computing their shortest path distances in the graph G.

Step 3Constructing an embedding of the data in d-dimensional Euclidean space Y that best preserves the manifold’s geometry.

jid ,X

jidG ,

Nonlinear Dimensionality Reduction Approach (ISOMAP)

Computer Vision Lab. SNU

Step 1Determining neighboring points within a fixed radius based on the input space distance .

# ε-radius # K-nearest neighbors

These neighborhood relations are represented as a weighted graph G over the data points.

ISOMAP (Algorithm Description)

jid ,X

ε

K=4

i j

k

jid ,X

kid ,X

Nonlinear Dimensionality Reduction Approach (ISOMAP)

Computer Vision Lab. SNU

ISOMAP (Algorithm Description)

Step 2

Estimating the geodesic distances between all pairs of points on the manifold by computing their shortest path distances in the graph G.

Can be done using Floyd’s algorithm or Dijkstra’s algorithm

jidG ,

)},(),( ),,(min{),( N1,2,...,k

othewise ),(ji, gneighborin ),(),(

jkdkidjidjidfor

jidjidjid

GGGG

G

G

ij

k jkdG , kidG ,

Nonlinear Dimensionality Reduction Approach (ISOMAP)

Computer Vision Lab. SNU

ISOMAP (Algorithm Description)

Step 3Constructing an embedding of the data in d-dimensional Euclidean space Y that best preserves the manifold’s geometry.Minimize the cost function:

)()()(

),(),(

),(

12.121

NN

GG

jiY

IDID

andjidjiD

yyjiDwhere

2)()(LYG DDE

Solution: take top d eigenvectors of the

matrix )( GD

Nonlinear Dimensionality Reduction Approach (ISOMAP)

Computer Vision Lab. SNU

Experimental results

# FACE # Hand writing : face pose and illumination : bottom loop and top

arch

MDS : open triangles

Isomap : filled circles

Nonlinear Dimensionality Reduction Approach (ISOMAP)

Computer Vision Lab. SNU

Discussion

Isomap handles non-linear manifold.

Isomap keeps the advantages of PCA and MDS.

Non-iterative procedure

Polynomial procedure

Guaranteed convergence

Isomap represents the global structure of a data set within a single coordinate system.

Nonlinear Dimensionality Reduction Approach (ISOMAP)

Computer Vision Lab. SNU