Embed Size (px)
Citation preview
Machine Learning for Data Science (CS4786)Lecture 5
Gaussian Mixture Models
Course Webpage :http://www.cs.cornell.edu/Courses/cs4786/2017fa/
Clustering demo
Two elongated ellipses
Iris dataset: Flowers
Iris-Setosa Iris-versicolor Iris-virginica
Smiley dataset
Wisconsin Breast Cancer dataset
K-means: pitfalls
K-means: pitfalls
K-means: pitfalls
K-means: pitfalls
K-means: pitfalls
K-means: pitfalls
K-means: pitfalls
K-means: pitfalls
K-means: pitfalls
K-means: pitfalls
K-means: pitfalls
K-means: pitfalls
• Looks for spherical clusters
• Of same radius
• And with roughly equal number of points
• Can we design algorithm that can address these shortcomings?
K-means: pitfalls
Ellipsoid
⌃ =1
|Cj |X
t2Cj
(xt � rj)(xt � rj)>
rjx
(x� rj)>⌃�1(x� rj)
t
K-MEANS CLUSTERING
For all j ∈ [K], initialize cluster centroids r
0j randomly and set m = 1
Repeat until convergence (or until patience runs out)1 For each t ∈ {1, . . . ,n}, set cluster identity of the point
cm(xt) = argminj∈[K]
�xt − r
m−1j �
2 For each j ∈ [K], set new representative as
r
mj = 1�Cm
j � �xt∈Cmj
xt
3 m← m + 1
ELLIPSOIDAL CLUSTERING
For all j ∈ [K], initialize cluster centroids r
0j and ellipsoids ⌃0
jrandomly and set m = 1Repeat until convergence (or until patience runs out)
1 For each t ∈ {1, . . . ,n}, set cluster identity of the point
cm(xt) = argminj∈[K]
(xt − r
m−1j )� �⌃m−1�−1 (xt − r
m−1j )
2 For each j ∈ [K], set new representative as
r
mj = 1�Cm
j � �xt∈Cmj
xt ⌃m = 1�Cj� �t∈Cj
(xt − r
mj )(xt − r
mj )�
3 m← m + 1
K-means: pitfalls
• Looks for spherical clusters
• Of same radius
• And with roughly equal number of points
HARD GAUSSIAN MIXTURE MODEL
For all j ∈ [K], initialize cluster centroids r
0j , ellipsoids ⌃0
j andinitial proportions ⇡0 randomly and set m = 1Repeat until convergence (or until patience runs out)
1 For each t ∈ {1, . . . ,n}, set cluster identity of the point
cm(xt) = argminj∈[K]
(xt − r
m−1j )� �⌃m−1�−1 (xt − r
m−1j ) − log(⇡m−1
j )2 For each j ∈ [K], set new representative as
r
mj = 1�Cm
j � �xt∈Cmj
xt ⌃m = 1�Cj� �t∈Cj
(xt − r
mj )(xt − r
mj )� ⇡m
j = �Cmj �
n
3 m← m + 1
Multivariate Gaussian• Two parameters:
• Mean
• Covariance matrix of size dxd
µ 2 Rd
⌃
p(x;µ,⌃) = (2⇡)
d/2det(⌃)
�1/2exp
✓�1
2
(x� µ)
>⌃
�1(x� µ)
◆
Multivariate Gaussian• Two parameters:
• Mean
• Covariance matrix of size dxd
µ 2 Rd
⌃
32
10
x
-1
exp(-(10 x2 +y2)/2)
-2-3-3
-2
-1
0
y
1
2
3
1
0.8
0.6
0.4
0.2
0
p(x;µ,⌃) = (2⇡)
d/2det(⌃)
�1/2exp
✓�1
2
(x� µ)
>⌃
�1(x� µ)
◆
HARD GAUSSIAN MIXTURE MODEL
For all j ∈ [K], initialize cluster centroids r
0j , ellipsoids ⌃0
j andinitial proportions ⇡0 randomly and set m = 1Repeat until convergence (or until patience runs out)
1 For each t ∈ {1, . . . ,n}, set cluster identity of the point
cm(xt) = argminj∈[K]
p(xt, rm−1j , ⌃m−1) × ⇡m(j)
2 For each j ∈ [K], set new representative as
r
mj = 1�Cm
j � �xt∈Cmj
xt ⌃m = 1�Cj� �t∈Cj
(xt − r
mj )(xt − r
mj )� ⇡m
j = �Cmj �
n
3 m← m + 1
(SOFT) GAUSSIAN MIXTURE MODEL
For all j ∈ [K], initialize cluster centroids r
0j and ellipsoids ⌃0
jrandomly and set m = 1Repeat until convergence (or until patience runs out)
1 For each t ∈ {1, . . . ,n}, set cluster identity of the point
Qmt (j) = p(xt, r
m−1j , ⌃m−1) × ⇡m(j)
2 For each j ∈ [K], set new representative as
r
mj = ∑
nt=1 Qt(j)xt∑n
t=1 Qt(j) ⌃m = ∑nt=1 Qt(j)(xt − r
mj )(xt − r
mj )�
∑nt=1 Qt(j)
⇡mj = ∑
nt=1 Qt(j)
n
3 m← m + 1
