Impact of Regularization on Spectral Clustering

Antony Joseph* and Bin Yu#

* Walmart Research Lab in San Francisco(formerly UCB and LBNL)

# Departments of Statistics and EECSUC Berkeley

Workshop on Spectral Algorithms, Simons Inst, Oct. , 2014

Monday, November 3, 2014

spectral clusteringin

graphs

Berkeley Drosophila Genome Project (BDGP)(The fruit fly project)

Overview

collaborators :

• Siqi Wu, UC Berkeley

• Erwin Frise, Lawrence Berkeley Lab

• Ann Hammonds, Lawrence Berkeley Lab

• Sue Celniker, Lawrence Berkeley Lab

spectral clusteringin

graphs

Berkeley Drosophila Genome Project (BDGP)(The fruit fly project)

Overview

collaborators :

• Siqi Wu, UC Berkeley

• Erwin Frise, Lawrence Berkeley Lab

• Ann Hammonds, Lawrence Berkeley Lab

• Sue Celniker, Lawrence Berkeley Lab

A Graph

Social network people

The fruit fly project pixels/points in the embryo template

Context Nodes

3Monday, November 3, 2014

The fruit fly project (Berkeley Drosophila Genome Project)

Drosophila(fruit fly)

Widely studied :• genetic mechanism similar to humans • easy to maintain in the lab• short life cycle • ...

Image dataset from the fruit fly project

tailless

• Over 100,000 stained embryo images (over 7000 genes)

• the interaction between different genes

• the genes required for development of various organs.

Goals: Contribute to the understanding of ...

‘Fate’ map in early embryos

Lohs-Schardin et. al (’70), Hartenstein et. al. (‘85)

Laser ablation experiments in embryos in early stages of development

‘Fate’ map in early embryos

hind gut

anterior mid-gut

dorsal epidermis

ventral neurogenic region

procephalic neurogenic region

pharynx

mesoderm

esophagus

Lohs-Schardin et. al (’70), Hartenstein et. al. (‘85)

Laser ablation experiments in embryos in early stages of development

Do genes explain the `fate’ map?

.... early stage gene expression images

Discovery of fate map &

communities on graphs

Nodes : pixels/points in the embryo

Edge if lot of genes are co-expressed at the two nodes

fate map

Edge between node i and node j

= . . . . . .

90-th percentile

τ = τ =

Take K = 8

Comparing unregularized vs. regularized SC

τ = τ =

Take K = 8dorsal epidermis

mesoderm

hind gut

anterior mid-gut

pharynx

esophagus

Communities

people like minded people

pixels/points in embryo area of future organs

Nodes Communities

Finding communities

Notion of (two) communities

Finding communities

Notion of (two) communities

Methods

Spectral clustering (Fiedler (’73), Donath & Hoffman (’73), ...)

Modularity (Newman & Girvan (‘03)), Latent space methods (Hoff et. al. (’02))Profile-likelihood (Bickel & Chen (’09)), Pseudo-Likelihood (Amini et. al. (’13)),

Spectral Clustering

Notation

Adjacency matrix:(symmetric binary)

Number of nodes: n

A ∈ Rn×n

Aij = Aji =

�1, (i, j)

Notation

Number of nodes: n

A ∈ Rn×n

Aij = Aji =

�1, (i, j)

Each row/column of A associated with a node

Notation

Number of nodes: n

A ∈ Rn×n

Aij = Aji =

�1, (i, j)

Degree matrix:(diagonal)

D ∈ Rn×n

Dii =�

Spectral Clustering

(Normalizedsymmetric Laplacian matrix)

= − / − /

Spectral clustering deals with the eigenvectors of the matrix :

Spectral Clustering

(Normalizedsymmetric Laplacian matrix)

= − / − /

Spectral clustering deals with the eigenvectors of the matrix :

Other matrices used ...

D −A

D−1 A ( Normalized random walk Laplacian)

(Unnormalized Laplacian)

(Adjacency matrix)

Illustration of SC

First eigenvector

0.052 0.05 0.048 0.046 0.044 0.042 0.04 0.038 0.0360.08

Illustration of SC

First eigenvector

0.052 0.05 0.048 0.046 0.044 0.042 0.04 0.038 0.0360.08

Illustration of SC

cluster

First eigenvector

SC for finding K clusters (Shi and Malik (00), Ng et. al (’02))

n×K V K L

SC for finding K clusters (Shi and Malik (00), Ng et. al (’02))

n×K V K L

Popularity of spectral clustering

• Computational advantage :

-requires eigenvector decomposition which is very fast

Theoretical backing :

- relaxation of various cut-based measures

(Hagen & Kahng (’92), Shi & Malik (’00), Ng et al, (’02))

- Stochastic Block Model and its extensions

(McSherry (‘01), Rohe. et. al (‘11), Chaudhari et. al. (’12), Sussman (’12),

Fishkind (’11))

Performance of spectral clustering improves greatly through regularization

Regularization proposed by Amini, Chen, Bickel and Levina (AoS, 2013)

Aτ = A+τ

n11�, τ > 0.

Lτ Aτ

Vτ = K Lτ

Alternative forms of regularization proposed and analyzed in Chaudhuri et. al (2012), Qin & Rohe (’13)

Aτ = A+τ

n11�, τ > 0.

Lτ Aτ

Vτ = K Lτ

Stochastic Block Model

Stochastic Block Model (SBM) (Holland et. al (’83))

(i, j) Pij

Stochastic Block Model (SBM) (Holland et. al (’83))

SBM with two blocks

=� �

(i, j) Pij

sample.4

Analysis of regularization for the SBM(Focus on K =2)

first sample eigenvector first sample eigenvector

k-means success : 100%k-means success : 87%

Recap : Regularized spectral clustering

Aτ = A+τ

n11�, τ > 0.

Lτ = D−1/2τ AτD

−1/2τ

Vτ = Lτ

Population level quantities

A = P=

Pτ = P + τn11

Lpopτ

Pτ = P + τn11

Lpopτ

Recall:

Vτ n× 2

Pτ = P + τn11

Lpopτ

Vτ V popτ

1,τ 2,τ

Pτ = P + τn11

Lpopτ

Vτ V popτ

first sample eigenvector

0.052 0.05 0.048 0.046 0.044 0.042 0.04 0.0380.1

τ =max = , max ∈ � ,τ − ,τ�

� ,τ − ,τ�

0.052 0.05 0.048 0.046 0.044 0.042 0.04 0.0380.1

τ =max = , max ∈ � ,τ − ,τ�

� ,τ − ,τ�

τ =Lτ Lpop

� 1,τ − 2,τ�

τ =Lτ Lpop

� 1,τ − 2,τ�

τ =Lτ Lpop

� 1,τ − 2,τ�

τ �√ � τ − τ �

µ ,τ

Implication of matrix perturbation theory (Davis - Kahan) :

τ =Lτ Lpop

� 1,τ − 2,τ�

τ �√ � τ − τ �

µ ,τ

(µ2,τ τ)

τ =Lτ Lpop

� 1,τ − 2,τ�

Implication of concentration of Laplacian (Oliveira (’10)):

with high probability� τ − τ � � min

�√

, + τ, ,

( , + τ)

��log

τ � log ,

τ �√ � τ − τ �

µ ,τ

τ =Lτ Lpop

� 1,τ − 2,τ�

Improvements using extension of techniques in Balakrishnan et. al. (’11).

τ �√ � τ − τ �

µ ,τ

τ =Lτ Lpop

� 1,τ − 2,τ�

τ = dn

Result (SBM with two blocks):

dn �√n log n

Summary:

τ = dn

Result (SBM with two blocks):

dn �√n log n

Choice of regularization parameter

� τ − τ �µ ,τ

Recall: trade-offs dictated by

� τ − τ �µ ,τ

� τ − ˆτ �µ̂ ,τ

Estimates based on estimated SBM (or degree corrected SBM)

� τ − τ �µ ,τ

� τ − ˆτ �µ̂ ,τ

� τ − τ �µ ,τ

� τ − ˆτ �µ̂ ,τ

ˆτ , µ̂ , τ

C1, C2

ˆτ , µ̂ , τ

C1, C2

p1, p2 q C1 C2

e.g. p̂1 = C1

ˆτ , µ̂ , τ

p̂2q̂

C1, C2

p1, p2 q C1 C2

e.g. p̂1 = C1

ˆτ , µ̂ , τ

p̂2q̂

C1, C2

p1, p2 q C1 C2

e.g. p̂1 = C1

P̂ L̂popτ µ̂2,τ L̂pop

Example

τ = 0 τ = 18

Example

k-means success : 94%k-means success : 75%

τ = 0 τ = 18

Political blog data

source : Adamic & Glance (’05)

0 50 100 150 200 250 300 350 4000

Histogram of degrees

source : Adamic & Glance (’05)

Political blogs data set

Unregularized Spectral Clustering

/4638Monday, November 3, 2014

second eigenvector discriminates these from the remaining

/4638Monday, November 3, 2014

third eigenvector

Regularized SC for political blogs datasetfir

τ = .

nodesnodes

Regularized SC for political blogs dataset

13% of misclassified nodes for regularizedcompared to 48% for unregularized

τ = .

nodesnodes

τ = τ =

Take K = 8

Comparing unregularized vs. regularized Spectral Clustering (SC)

τ = τ =

Take K = 8dorsal epidermis

mesoderm

hind gut

anterior mid-gut

pharynx

esophagus

Comparing unregularized vs. regularized Spectral Clustering (SC)

Summary

• Theoretical upper bound under SBM shows “bias-variance”-like trade-off while the amount of regularization increases in SC

• Theoretical analysis motivates practically useful scheme (using SBM or degree-corrected SBM) to select regularization parameter in RSC.

Promising results in fruitfly image segmentation

Paper at (2014 rev): http://arxiv.org/pdf/1312.1733.pdf

Ongoing/future directions

The BDGP project (with Antony Joseph, Siqi Wu, Ann Hammonds, Sue Celniker, Erwin Frise)

• Fast algorithm for computing the data-driven choice of regularization parameter• Role of regularization in other scenarios, such as hierarchical clusters• Regularization parameter choice for continuous data

Spectral Clustering (with Antony Joseph)

• Analysis of gene interactions in different regions of early stage embryos• Extension of analysis to later stage embryos

Impact of Regularization on Spectral Clustering · Impact of Regularization on Spectral Clustering...

Documents

A Tutorial on Spectral Clustering · ulrike.luxburg@tuebingen.mpg.de ... This tutorial is set up as a self-contained introduction to spectral clustering. We derive spectral

Clustering Applications Reminder Applications Spectral Clustring Assignment Clustering

Spectral Curvature Clustering for Hybrid Linear …gchen/papers/thesis_gchen.pdf · Spectral Curvature Clustering for Hybrid Linear ... Spectral Curvature Clustering for Hybrid Linear

Spectral Regularization Algorithms for Learning Large ...jmlr.csail.mit.edu/papers/volume11/mazumder10a/mazumder10a.pdf · MATRIX COMPLETION BY SPECTRAL REGULARIZATION In this paper

Spectral Clustering and Semi-Supervised learning using Evolving Similarity Graphsposeidon.csd.auth.gr/.../2015/Spectral_clustering_ASC.pdf · 2016-05-21 · Spectral Clustering and

Spectral clustering via half-quadratic optimization · Spectral clustering generates the new representation (i.e., the spectral representation) of original data through matrix spectral

Dimensionality Reduction for Spectral Clusteringproceedings.mlr.press/v15/niu11a/niu11a.pdf · 2018-07-14 · Dimensionality Reduction for Spectral Clustering for spectral clustering

Spectral Curvature Clustering for Hybrid Linear Modeling · Our algorithm, Spectral Curvature Clustering (SCC), combines Govindu’s frame-work [19] and Ng et al.’s spectral clustering

Constrained Clustering via Spectral Regularization

Spectral Clustering of Synchronous Spike Trains

Spectral clustering

Spectral Regularization and its Applications in Quantum ... Spectral Regularization and its Applications in Quantum Field Theory Settore Scientiﬁco Disciplinare FIS/02 Dottorando

A Tutorial on Spectral Clustering - Max Planck Society1].… · A Tutorial on Spectral Clustering Ulrike von Luxburg Abstract. In recent years, spectral clustering has become one

Learning Eigenfunctions: Links with Spectral Clustering ...lisa/pointeurs/snowbird_eigenfn_talk.pdf · Link between Spectral Clustering and Eigenfunctions Equivalence between eigenvectors

l1-spectral clustering algorithm: a spectral clustering

Spectral Graph Clustering

Spectral Clustering and Label Fusion For 3D Tissue ... · Spectral Clustering and Label Fusion For 3D Tissue Classiﬁcation: ... SPECTRAL CLUSTERING AND LABEL FUSION ... [Shi and

Zeta function regularization of the spectral determinant and

Spectral Clustering - sml.csa.iisc.ac.in

Network clustering via spectral projections