PRML 9.1-9.2: K-means Clustering & Mixtures of Gaussians

PRML 9.1-9.2

K-means Clustering &

Mixtures of Gaussians July 16, 2014

by Shinichi TAMURA

Mixtures of Gaussians K-means Clustering

Today's topics

1.  K-means Clustering 1.  Clustering Problem 2.  K-means Clustering 3.  Application for Image Compression

2.  Mixtures of Gaussians 1.  Introduction of latent variables 2.  Problem of ML estimates 3.  EM-algorithm for Mixture of Gaussians

July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA


Today's topics





Today's topics





Clustering Problem

An unsupervised machine learning problem Divide data in some group (=cluster) where ü  similar data > same group ü  dissimilar data > different group



Clustering Problem

Divide data in some group (=cluster) where ü  similar data > same group ü  dissimilar data > different group



Clustering Problem


MinimizeN!

n=1

!xn " µk(n)!2



Clustering Problem


MinimizeN!

n=1

!xn " µk(n)!2

Center of the cluster



Clustering Problem

Given data set and # of cluster K Let be cluster representative and be assignment indicator ( ), Here, J is called “distortion measure”.

X = {x1, . . . ,xN}

µk rnkrnk = 1 if x ! Ck

Minimize J =N!

n=1

K!

k=1

rnk!xn " µk!2



Today's topics





Today's topics





K-means Clustering

How to solve that?



K-means Clustering

How to solve that? and are dependent each other

> No closed form solution µk rnk



K-means Clustering

How to solve that? and are dependent each other

> No closed form solution Use iterative algorithm !

µk rnk



K-means Clustering

Strategy and can't be updated simultaneously µk rnk



K-means Clustering

Strategy and can't be updated simultaneously

> Update them one by one

µk rnk



K-means Clustering

Update of (assignment) Since each can be determined independently, J will be minimum if they are assigned to the nearest .

rnk

xn

µk



K-means Clustering

Update of (assignment) Since each can be determined independently, J will be minimum if they are assigned to the nearest . Therefore,

rnk

xn

µk

rnk =

!1 if k = arg minj !xn " µj!2,

0 otherwise.



K-means Clustering

Update of (parameter estimation) Optimal is obtained by setting derivative 0. µk

µk

!

!µk

N!

n=1

K!

k!=1

rnk!!xn " µk!!2 = 0.

#$ 2N!

n=1

rnk(xn " µk) = 0.

! µk ="N

n=1 rnkxn"Nn=1 rnk

=1

Nk

!

xn!Ck

xn.



K-means Clustering


µk

!

!µk

N!

n=1

K!

k!=1

rnk!!xn " µk!!2 = 0.

#$ 2N!

n=1

rnk(xn " µk) = 0.

! µk ="N

n=1 rnkxn"Nn=1 rnk

=1

Nk

!

xn!Ck

xn.



K-means Clustering


µk

!

!µk

N!

n=1

K!

k!=1

rnk!!xn " µk!!2 = 0.

#$ 2N!

n=1

rnk(xn " µk) = 0.

! µk ="N

n=1 rnkxn"Nn=1 rnk

=1

Nk

!

xn!Ck

xn.

Mean of the cluster



K-means Clustering


µk

!

!µk

N!

n=1

K!

k!=1

rnk!!xn " µk!!2 = 0.

#$ 2N!

n=1

rnk(xn " µk) = 0.

! µk ="N

n=1 rnkxn"Nn=1 rnk

=1

Nk

!

xn!Ck

xn.

Mean of the cluster



K-means Clustering


µk

!

!µk

N!

n=1

K!

k!=1

rnk!!xn " µk!!2 = 0.

#$ 2N!

n=1

rnk(xn " µk) = 0.

! µk ="N

n=1 rnkxn"Nn=1 rnk

=1

Nk

!

xn!Ck

xn.

Mean of the cluster

is the mean of the cluster Cost function J corresponds to the sum of inner-class variance!

µk



K-means Clustering


µk

!

!µk

N!

n=1

K!

k!=1

rnk!!xn " µk!!2 = 0.

#$ 2N!

n=1

rnk(xn " µk) = 0.

! µk ="N

n=1 rnkxn"Nn=1 rnk

=1

Nk

!

xn!Ck

xn.

Mean of the cluster

is the mean of the cluster Cost function J corresponds to the sum of inner-class variance!

µk



K-means Clustering

K-means algorithm 1. Initialize , 2. Repeat following two steps until converge

i) Assign each to closest ii) Update to the mean of the cluster

µk rnk

xn µk

µk



K-means Clustering



µk rnk

xn µk

µk

E step



K-means Clustering



µk rnk

xn µk

µk

M step



K-means Clustering

Convergence property Both steps never increase J, so we can obtain better result in every iteration.



K-means Clustering

Convergence property Both steps never increase J, so we can obtain better result in every iteration. Since is finite, algorithm converge after finite iterations.

rnk



K-means Clustering

Demo of algorithm



K-means Clustering

Demo of algorithm

E step July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA


K-means Clustering

Demo of algorithm M step



K-means Clustering

Demo of algorithm



K-means Clustering




K-means Clustering

Demo of algorithm



K-means Clustering




K-means Clustering

Demo of algorithm



K-means Clustering




K-means Clustering

Demo of algorithm



K-means Clustering

Demo of algorithm



K-means Clustering

Demo of algorithm



K-means Clustering

Calculation performance E step ... Comparison of every data point

and every cluster mean > O(KN)

µk

xn



K-means Clustering



µk

xn

Not good



K-means Clustering



µk

xn

Not good Improve with kd-tree, triangle inequality...etc



K-means Clustering



µk

xn



K-means Clustering



M step ... Calculation of mean for every cluster > O(N)

µk

xn



K-means Clustering

Here, two variation will be introduced: 1.  On-line version 2.  General dissimilarity



K-means Clustering




K-means Clustering

[Variation] 1. On-line version The case where one datum is observed at once.



K-means Clustering


> Apply Robbins-Monro algorithm



K-means Clustering


> Apply Robbins-Monro algorithm µnew

k = µoldk + !n(xn ! µold

k ).



K-means Clustering




k ).Learning rate



K-means Clustering




k ).Learning rate Decrease with iteration



K-means Clustering




K-means Clustering

[Variation] 2. General dissimilarity Euclidian distance is not ü  appropriate to categorical data, etc. ü  robust to outlier.



K-means Clustering

[Variation] 2. General dissimilarity Euclidian distance is not ü  appropriate to categorical data, etc. ü  robust to outlier. > Use general dissimilarity measure V(x,x!)



K-means Clustering


E step ... No difference July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA


K-means Clustering


M step ... Not assured J is easy to minimize July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA


K-means Clustering

[Variation] 2. General dissimilarity To make M-step easy, restrict to the vector chosen from > A solution can be obtained by finite number of comparison

µk

{xn}



K-means Clustering

[Variation] 2. General dissimilarity To make M-step easy, restrict to the vector chosen from > A solution can be obtained by finite number of comparison

µk

{xn}

µk = arg minxn

!

xn!!Ck

V(xn,xn!)



Today's topics





Today's topics





Application for Image Compression

K-means algorithm can be applied to Image Compression and Segmentation




K-means algorithm can be applied to Image Compression and Segmentation Basic Idea Treat similar pixel as same one





Original data





Cluster center





Cluster center (pallet / code-book vector)




K-means algorithm can be applied to Image Compression and Segmentation Basic Idea Treat similar pixel as same one = so called “vector quantization”




Demo




Demo




Demo




Compression rate Original image...24N bits

(N=# of pixels)





(N=# of pixels) Compressed image... 24K+N log2K bits

(K=# of pallet)





(N=# of pixels) Compressed image... 24K+N log2K bits

(K=# of pallet) 16.7% if N~1M, K=10



Today's topics





Today's topics





Today's topics





Introduction of Latent Variable

In K-means, all assignments are equal, “all or nothing”.

Treated same July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA



In K-means, all assignments are equal, “all or nothing”. Is these “hard” assignment appropriate?

Treated same July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA



In K-means, all assignments are equal, “all or nothing”. Is these “hard” assignment appropriate? > Want introduce "soft" assignment

Treated same Probabilistic




Introduce random variable z, having 1-of-K representation > Control unobserved “states”





Once state is determined,





Once state is determined, x is drawn from Gaussian of the state

p(x|zk = 1) = N (x|µk,!k).





Once state is determined, x is drawn from Gaussian of the state

p(x|zk = 1) = N (x|µk,!k).

x

z

Graphical representation July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA



Here the distribution over x is p(x) =

!

z

p(z)p(x|z)

=K!

k=1

p(zk = 1)p(x|zk = 1)

=K!

k=1

!kN (x|µk,!k).





!

z

p(z)p(x|z)

=K!

k=1

p(zk = 1)p(x|zk = 1)

=K!

k=1

!kN (x|µk,!k).

z is 1-of-K rep.





!

z

p(z)p(x|z)

=K!

k=1

p(zk = 1)p(x|zk = 1)

=K!

k=1

!kN (x|µk,!k).





!

z

p(z)p(x|z)

=K!

k=1

p(zk = 1)p(x|zk = 1)

=K!

k=1

!kN (x|µk,!k).

Gaussian Mixtures ! July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA



Estimate (or “explain”) x came from which state

!(zk) ! p(zk = 1|x) =p(zk = 1)p(x|zk = 1)!j p(zj = 1)p(x|zj = 1)

="kN (x|µk,!k)!j "jN (x|µj ,!j)

.







.







.







.Posteriors






.Posteriors Priors







.Posteriors Priors Likelihood




Estimate (or “explain”) x came from which state This value is also called “responsibilities”



.




Example of Gaussian Mixtures

(a)

0 0.5 1

0

0.5

1(b)

0 0.5 1

0

0.5

1 (c)

0 0.5 1

0

0.5

1

No state info Coloured by true state

Coloured by responsibility



Today's topics





Today's topics





Problems of ML estimates

ML estimates of mixtures of Gaussians have two problems:

i.  Presence of Singularities ii.  Identifiability









i) Presence of Singularities What if a mean collides with a data point? !j,m µj = xm




i) Presence of Singularities What if a mean collides with a data point? Likelihood can be however large by

!j,m µj = xm

!j ! 0

L !

!

" 1!j

+#

k !=j

pk,m

$

%&

n!=m

!

" 1!j

exp

'" (xn " µj)2

2!2j

(+

#

k !=j

pk,n

$

%

#$.





!j,m µj = xm

!j ! 0

L !

!

" 1!j

+#

k !=j

pk,m

$

%&

n!=m

!

" 1!j

exp

'" (xn " µj)2

2!2j

(+

#

k !=j

pk,n

$

%

#$.! "





!j,m µj = xm

!j ! 0

L !

!

" 1!j

+#

k !=j

pk,m

$

%&

n!=m

!

" 1!j

exp

'" (xn " µj)2

2!2j

(+

#

k !=j

pk,n

$

%

#$.! "





!j,m µj = xm

!j ! 0

L !

!

" 1!j

+#

k !=j

pk,m

$

%&

n!=m

!

" 1!j

exp

'" (xn " µj)2

2!2j

(+

#

k !=j

pk,n

$

%

#$.! " ! 0





!j,m µj = xm

!j ! 0

L !

!

" 1!j

+#

k !=j

pk,m

$

%&

n!=m

!

" 1!j

exp

'" (xn " µj)2

2!2j

(+

#

k !=j

pk,n

$

%

#$.! " > 0





!j,m µj = xm

!j ! 0

L !

!

" 1!j

+#

k !=j

pk,m

$

%&

n!=m

!

" 1!j

exp

'" (xn " µj)2

2!2j

(+

#

k !=j

pk,n

$

%

#$.




i) Presence of Singularities It doesn't occur in single Gaussian.

L ! 1!N

j

!

n!=m

exp

"" (xn " µj)2

2!2j

#

#0.





L ! 1!N

j

!

n!=m

exp

"" (xn " µj)2

2!2j

#

#0.! "





L ! 1!N

j

!

n!=m

exp

"" (xn " µj)2

2!2j

#

#0.! " ! 0





L ! 1!N

j

!

n!=m

exp

"" (xn " µj)2

2!2j

#

#0.




i) Presence of Singularities It doesn't occur in single Gaussian. It doesn't occur in Bayesian approach either.

L ! 1!N

j

!

n!=m

exp

"" (xn " µj)2

2!2j

#

#0.









ii) Identifiability Optimal solutions are not unique: If we have a solution, there are (K!-1) other equivalent solution.




ii) Identifiability Optimal solutions are not unique: If we have a solution, there are (K!-1) other equivalent solution. Matters when interpret, but does not matter when model only



Today's topics





Today's topics





EM-algorithm for Gaussian Mixtures

The conditions of ML are obtained by where

!

!µk

L = 0,

!

!!k

L = 0,

!

!"k

!L + #

"#j "j ! 1

$%= 0.

L(!,µ,!) =!N

n=1 ln"!K

k=1 !kN (xn|µk,!k)#




The conditions of ML where

µk =1

Nk

N!

n=1

!n(zk)xn,

!k =1

Nk

N!

n=1

!n(zk)(xn ! µj)(xn ! µj)T,

"k =Nk

N,

Nk =!N

n=1 !n(zk)




The conditions of ML where

µk =1

Nk

N!

n=1

!n(zk)xn,

!k =1

Nk

N!

n=1

!n(zk)(xn ! µj)(xn ! µj)T,

"k =Nk

N,

Nk =!N

n=1 !n(zk)

!n(zk) appeared




Recall that

!n(zk) ="kN (xn|µk,!k)!j "jN (xn|µj ,!j)

.




Recall that


.

Parameters appeared




Recall that


.

Parameters appeared = No closed form solution




Recall that Again, use iterative algorithm!


.

Parameters appeared = No closed form solution




EM algorithm for Gaussian Mixtures 1. Initialize parameters 2. Repeat following two steps until converge

i) Calculate ii) Update parameters


.







.

E step







.

M step




Demo of algorithm




Demo of algorithm




Demo of algorithm




Demo of algorithm




Demo of algorithm




Comparison with K-means

EM for Gaussian Mixtures

K-means Clustering July 16, 2014 PRML 9.1-9.2 Shinichi TAMURA


Today's topics





Today's topics





Today's topics




Technology

PRML 9.1-9.2: K-means Clustering & Mixtures of Gaussians