Embed Size (px)
Citation preview
1
Page 1
EstimasiEstimasi Prob. Density Function Prob. Density Function dengandengan EMEM
SumberSumber: :
--Forsyth & Ponce Chap. 7Forsyth & Ponce Chap. 7
--StandfordStandford Vision & ModelingVision & Modeling
Probability Density EstimationProbability Density Estimation
• Parametric Representations• Non-Parametric Representations• Mixture Models
2
Page 2
MetodeMetode estimasiestimasi NonNon--parametricparametric
• Tanpa asumsi apapun tentang distribusi• Estimasi sepenuhnya bergantung ada DATA• cara mudah menggunakan: Histogram
HistogramsHistograms
Diskritisasi, lantas ubah dalam bentuk batang:
3
Page 3
HistogramsHistograms
• Butuh komputasi banyak, namun sangat umumdigunakan• Dapat diterapkan pada sembarang bentukdensitas (arbitrary density)
HistogramsHistograms
Permasalahan:
• Higher dimensional Spaces:
- jumlah batang (bins) yg. Exponential - jumlah training data yg exponential- Curse of Dimensionality
• size batang ? Terlalu sedikit: >> kasar
Terlalu banyak: >> terlalu halus
4
Page 4
PendekatanPendekatan secarasecara prinsipprinsip::
• x diambil dari ‘unknown’ p(x)• probabiliti bahwa x ada dalam region R adalah:
VxpdxxpPR
)(')'( ≈= ∫
PendekatanPendekatan secarasecara prinsipprinsip::
VxpdxxpPR
)(')'( ≈= ∫
N
KP =
• x diambil dari ‘unknown’ p(x)• probabiliti bahwa x ada dalam region R adalah:
5
Page 5
PendekatanPendekatan secarasecara prinsipprinsip::
VxpdxxpPR
)(')'( ≈= ∫
N
KP =
NV
Kxp ≈⇒ )(
• x diambil dari ‘unknown’ p(x)• probabiliti bahwa x ada dalam region R adalah:
PendekatanPendekatan secarasecara prinsipprinsip::
NV
Kxp ≈⇒ )(
Dengan Fix VTentukan K
Dengan Fix KTentukan V
Metoda Kernel-Based K-nearestneighbor
6
Page 6
MetodaMetoda KernelKernel--Based:Based:
NV
Kxp ≈⇒ )(
Parzen Window:
<
=otherwise 0
2/1|u| 1)( juH
MetodaMetoda KernelKernel--Based:Based:
NV
Kxp ≈⇒ )(
Parzen Window:
<
=otherwise 0
2/1|u| 1)( juH
∑=
−=N
nnxxHK
1
)(
7
Page 7
MetodaMetoda KernelKernel--Based:Based:
NV
Kxp ≈⇒ )(
Parzen Window:
<
=otherwise 0
2/1|u| 1)( juH
∑=
−=N
nnxxHK
1
)(∑=
−=N
nnd
xxHNh
xp1
)(1
)(
MetodaMetoda KernelKernel--Based:Based:
NV
Kxp ≈⇒ )(
Gaussian Window:
−−= ∑
=2
2
12/2 2
||||exp
)2(
11)(
h
xx
hNxp n
N
ndπ
8
Page 8
MetodaMetoda KernelKernel--Based:Based:
KK--nearestnearest--neighbor:neighbor:
NV
Kxp ≈⇒ )(
Kembankan V sampai dia mencapai K points.
9
Page 9
KK--nearestnearest--neighbor:neighbor:
KK--nearestnearest--neighbor:neighbor:
Klasifikasi secara Bayesian :
VN
KCxp
k
kk =)|(
NV
Kxp =)(
N
NCp k
k =)(
10
Page 10
KK--nearestnearest--neighbor:neighbor:
Klasifikasi secara Bayesian :
VN
KCxp
k
kk =)|(
NV
Kxp =)(
N
NCp k
k =)(
K
KxCp k
k =)|(
“aturan klasifikasi k-nearest-neighbour ”
Probability Density EstimationProbability Density Estimation
• Parametric Representations• Non-Parametric Representations• Mixture Models (Model Gabungan)
11
Page 11
MixtureMixture--Models (Model Models (Model GabunganGabungan):):
Gaussians:
- Mudah- Low Memory- Cepat- Good Properties
Non-Parametric:
- Umum- Memory Intensive- Slow
Mixture Models
CampuranCampuran fungsifungsi Gaussian (mixture of Gaussian (mixture of Gaussians):Gaussians):
x
p(x)
Jumlah dari Gaussians tunggal
12
Page 12
CampuranCampuran fungsifungsi Gaussian:Gaussian:
x
p(x)
Jumlah dari Gaussians tunggal
Keunggulan: Dapat mendekati bentuk densitassembarang (Arbitrary Shape)
CampuranCampuran fungsifungsi Gaussian:Gaussian:
x
p(x)
Generative Model: z
1 2 3P(j)
p(x|j)
13
Page 13
CampuranCampuran fungsifungsi Gaussian:Gaussian:
x
p(x)
∑=
=M
j
jPjxpxp1
)()|()(
−−=
2
2
2/2 2
||||exp
)2(
1)|(
jd
j
xjxp
σµ
πσ
CampuranCampuran fungsifungsi Gaussian:Gaussian:
Maximum Likelihood:
∑=
−=−=N
nnxpLE
1
)(lnln
14
Page 14
CampuranCampuran fungsifungsi Gaussian:Gaussian:
Maximum Likelihood:
∑=
−=−=N
nnxpLE
1
)(lnln
0=∂∂
k
E
µ
E
kµ
CampuranCampuran fungsifungsi Gaussian:Gaussian:
Maximum Likelihood:
∑=
−=−=N
nnxpLE
1
)(lnln
0=∂∂
k
E
µ ∑
∑
=
==⇒N
nn
n
N
nn
j
xjP
xxjP
1
1
)|(
)|(µ
15
Page 15
CampuranCampuran fungsifungsi Gaussian:Gaussian:
∑
∑
=
==N
nn
n
N
nn
j
xjP
xxjP
1
1
)|(
)|(µ
CampuranCampuran fungsifungsi Gaussian:Gaussian:
∑
∑
=
==N
nn
n
N
nn
j
xjP
xxjP
1
1
)|(
)|(µ
∑=
= M
kn
nn
kPkxp
jPjxpxjP
1
)()|(
)()|()|(
16
Page 16
CampuranCampuran fungsifungsi Gaussian:Gaussian:
∑
∑
=
==N
nn
n
N
nn
j
xjP
xxjP
1
1
)|(
)|(µ
−−=
2
2
2/2 2
||||exp
)2(
1)|(
j
jn
dj
n
xjxp
σ
µ
πσ
∑=
= M
kn
nn
kPkxp
jPjxpxjP
1
)()|(
)()|()|(
CampuranCampuran fungsifungsi Gaussian:Gaussian:
∑
∑
=
==N
nn
n
N
nn
j
xjP
xxjP
1
1
)|(
)|(µ
−−=
2
2
2/2 2
||||exp
)2(
1)|(
j
jn
dj
n
xjxp
σ
µ
πσ
∑=
= M
kn
nn
kPkxp
jPjxpxjP
1
)()|(
)()|()|(
17
Page 17
CampuranCampuran fungsifungsi Gaussian:Gaussian:
Maximum Likelihood:
∑=
−=−=N
nnxpLE
1
)(lnln
0=∂∂
k
E
µ
E
kµ
Tidak adasolusi pendek !
CampuranCampuran fungsifungsi Gaussian:Gaussian:
Maximum Likelihood:
∑=
−=−=N
nnxpLE
1
)(lnln
EGradient Descent
18
Page 18
CampuranCampuran fungsifungsi Gaussian:Gaussian:
Maximum Likelihood:
∑=
−=−=N
nnxpLE
1
)(lnln
),...,,,...,,,...,( 111 MMMk
fE
αασσµµµ
=∂∂
CampuranCampuran fungsifungsi Gaussian:Gaussian:
Optimasi secara Gradient Descent:
• Complex Gradient Function(highly nonlinear coupled equations)
• Optimasi sebuah Gaussian tergantung dari seluruh
campuran lainnya.
19
Page 19
CampuranCampuran fungsifungsi Gaussian:Gaussian:
x
p(x)
-> Dengan strategi berbeda:
Observed Data:
CampuranCampuran fungsifungsi Gaussian:Gaussian:
x
p(x)
Observed Data:
Densitas yg dihasilkan
20
Page 20
CampuranCampuran fungsifungsi Gaussian:Gaussian:
x
p(x)
yVariabel Hidden
1 2
Observed Data:
CampuranCampuran fungsifungsi Gaussian:Gaussian:
x
p(x)
yVariabel Hidden
1 2
1 1 1111 1 2 2 2222 2 yUnobserved:
Observed Data:
21
Page 21
ContohContoh populerpopuler ttgttg. Chicken and Egg . Chicken and Egg Problem:Problem:
x
p(x)
1 1 1111 1 2 2 2222 2 yAnggapkita tahu
Max.LikelihoodUtk. Gaussian #1
Max.LikelihoodUtk. Gaussian #2
Chicken+Egg Problem:Chicken+Egg Problem:
x
p(x)
1 1 1111 1 2 2 2222 2 y
Anggapkita tahu
P(y=1|x) P(y=2|x)
22
Page 22
Chicken+Egg Problem:Chicken+Egg Problem:
x
p(x)
1 1 1111 1 2 2 2222 2 y
Tapi yg ini kitatidak tau samasekali ?
?
Chicken+Egg Problem:Chicken+Egg Problem:
x
p(x)
1 1 1111 1 2 2 2222 2 yCoba pura2 tahu
23
Page 23
Clustering:Clustering:
x
1 1 1111 1 2 2 2222 2 yTebakan benar ?
K-mean clustering / Basic Isodata
PengelompokanPengelompokan (Clustering):(Clustering):
Procedure: Basic Isodata
1. Choose some initial values for the meansLoop: 2. Classify the n samples by assigning them to the class
of the closest mean.3. Recompute the means as the average of the samples
in their class.4. If any mean changed value, go to Loop;
otherwise, stop.
Mµµ ,...,1
24
Page 24
IsodataIsodata: : InisialisasiInisialisasi
1µ
2µ
IsodataIsodata: : MenyatuMenyatu (Convergence)(Convergence)
1µ
2µ
25
Page 25
IsodataIsodata: : BeberapaBeberapa permasalahanpermasalahan
DitebakDitebak Eggs / Eggs / TerhitungTerhitung ChickenChicken
x
p(x)
1 1 1111 1 2 2 2222 2 yDisini kita berada
Max.LikelihoodUtk. Gaussian #1
Max.LikelihoodUtk. Gaussian #2
26
Page 26
GaussianAproximasiGaussianAproximasi ygyg. . baikbaik
x
p(x)
• Namun tidak optimal! • Permasalahan: Highly overlapping Gaussians
Expectation Maximization (EM)Expectation Maximization (EM)
• EM adalah formula umum dari problem seperti “Chicken+Egg” (Mix.Gaussians, Mix.Experts, Neural Nets, HMMs, Bayes-Nets,…)
• Isodata: adalah contoh spesifik dari EM
• General EM for mix.Gaussian: disebut Soft-Clustering
• Dapat konvergen menjadi Maximum Likelihood
27
Page 27
IngatIngat rumusanrumusan iniini ?:?:
∑
∑
=
==N
nn
n
N
nn
j
xjP
xxjP
1
1
)|(
)|(µ
−−=
2
2
2/2 2
||||exp
)2(
1)|(
j
jn
dj
n
xjxp
σ
µ
πσ
∑=
= M
kn
nn
kPkxp
jPjxpxjP
1
)()|(
)()|()|(
Soft Chicken and Egg Problem:Soft Chicken and Egg Problem:
x
p(x)
P(1|x)0.1 0.3 0.7 0.1 0.01 0.0001
0.99 0.99 0.99 0.5 0.001 0.00001
∑
∑
=
==N
nn
n
N
nn
j
xjP
xxjP
1
1
)|(
)|(µ
28
Page 28
Soft Chicken and Egg Problem:Soft Chicken and Egg Problem:
x
p(x)
P(1|x)
∑
∑
=
==N
nn
n
N
nn
j
xjP
xxjP
1
1
)|(
)|(µ
0.1 0.3 0.7 0.1 0.01 0.0001
0.99 0.99 0.99 0.5 0.001 0.00001
Anggap kitatahu:
Weighted Mean of Data
Soft Chicken and Egg Problem:Soft Chicken and Egg Problem:
x
p(x)
P(1|x)
∑
∑
=
==N
nn
n
N
nn
j
xjP
xxjP
1
1
)|(
)|(µ
0.1 0.3 0.7 0.1 0.01 0.0001
0.99 0.99 0.99 0.5 0.001 0.00001
Step-2:Hitung ulangposteriors
29
Page 29
LangkahLangkah prosedurprosedur EM:EM:
Procedure: EM
1. Choose some initial values for the meansE-Step: 2. Compute the posteriors for each class and each
sample: M-Step: 3. Re-compute the means as the weighted average
of their class:
4. If any mean changed value, go to Loop; otherwise, stop.
Mµµ ,...,1
)|( nxjP
∑
∑
=
==N
nn
n
N
nn
j
xjP
xxjP
1
1
)|(
)|(µ
EM EM dandan Gaussian mixtureGaussian mixture
),(maxarg )1()( −= ii Q θθθθ
∑
∑
=
−
=
−
=N
n
in
N
nn
in
ij
xjp
xxjp
1
)1(
1
)1(
)(
),|(
),|(
θ
θµ
30
Page 30
EM EM dandan Gaussian mixtureGaussian mixture
),(maxarg )1()( −= ii Q θθθθ
∑
∑
=
−
=
− −−=∑
N
n
in
N
n
Tijn
ijn
in
ij
xjp
xxxjp
1
)1(
1
)()()1(
)(
),|(
))()(,|(
θ
µµθ
EM EM dandan Gaussian mixtureGaussian mixture
),(maxarg )1()( −= ii Q θθθθ
∑=
−=N
n
in
ij xjp
N 1
)1()( ),|(1
θα
31
Page 31
ContohContoh--contohcontoh EM:EM:
Training Samples
ContohContoh--contohcontoh EM:EM:
Training Samples Initialization
32
Page 32
ContohContoh--contohcontoh EM:EM:
Training Samples End Result of EM
ContohContoh--contohcontoh EM:EM:
Training Samples Density Isocontours
33
Page 33
ContohContoh--contohcontoh EM:EM:
Color Segmentation
ContohContoh--contohcontoh EM:EM:
Layered Motion
Yair Weiss
