Upload
charla-shepherd
View
216
Download
0
Tags:
Embed Size (px)
Citation preview
14 October, 201014 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud)LRI Seminar 2010 (Univ. Paris-Sud) 11
Statistical performance analysis by Statistical performance analysis by loopy belief propagation in loopy belief propagation in
probabilistic image processing probabilistic image processing Kazuyuki TanakaKazuyuki Tanaka
Graduate School of Information Sciences,Graduate School of Information Sciences,Tohoku University, JapanTohoku University, Japan
http://www.smapip.is.tohoku.ac.jp/~kazu/http://www.smapip.is.tohoku.ac.jp/~kazu/
CollaboratorsD. M. Titterington (University of Glasgow, UK)
M. Yasuda (Tohoku University, Japan)S. Kataoka (Tohoku University, Japan)
22
IntroductionIntroduction
Bayesian network is originally one of the methods for probabilistic inferences in artificial intelligence. Some probabilistic models for information processing are also regarded as Bayesian networks. Bayesian networks are expressed in terms of products of functions with a couple of random variables and can be associated with graphical representations. Such probabilistic models for Bayesian networks are referred to as Graphical Model.
14 October, 201014 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud)LRI Seminar 2010 (Univ. Paris-Sud)
33
Probabilistic Image Processing by Bayesian NetworkProbabilistic Image Processing by Bayesian Network
Probabilistic image processing systems are formulated on square grid graphs. Averages, variances and covariances of the Bayesian network are approximately computed by using the belief propagation on the square grid graph.
14 October, 201014 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud)LRI Seminar 2010 (Univ. Paris-Sud)
10 March, 201010 March, 2010 IW-SMI2010 (Kyoto)IW-SMI2010 (Kyoto) 44
MRF, Belief Propagation and MRF, Belief Propagation and Statistical PerformanceStatistical Performance
Nishimori and Wong (1999): Physical Review ENishimori and Wong (1999): Physical Review EStatistical Performance Estimation for MRFStatistical Performance Estimation for MRF
(Infinite Range Ising Model and Replica Theory)(Infinite Range Ising Model and Replica Theory)
Is it possible to estimate the performance of belief Is it possible to estimate the performance of belief propagation statistically?propagation statistically?
Tanaka and Morita (1995): Physics Letters ATanaka and Morita (1995): Physics Letters ACluster Variation Method for MRF in Image ProcessingCluster Variation Method for MRF in Image Processing
CVM= Generalized Belief Propagation (GBP)CVM= Generalized Belief Propagation (GBP)
Geman and Geman (1986): IEEE Transactions on PAMIGeman and Geman (1986): IEEE Transactions on PAMIImage Processing by Image Processing by Markov Random Fields (MRF)Markov Random Fields (MRF)
14 October, 201014 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud)LRI Seminar 2010 (Univ. Paris-Sud) 55
OutlineOutline
1.1. IntroductionIntroduction2.2. Bayesian Image Analysis by Gauss Markov Bayesian Image Analysis by Gauss Markov
Random FieldsRandom Fields3.3. Statistical Performance Analysis for Gauss Statistical Performance Analysis for Gauss
Markov Random FieldsMarkov Random Fields4.4. Statistical Performance Analysis in Binary Statistical Performance Analysis in Binary
Markov Random Fields by Loopy Belief Markov Random Fields by Loopy Belief PropagationPropagation
5.5. Concluding RemarksConcluding Remarks
14 October, 201014 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud)LRI Seminar 2010 (Univ. Paris-Sud) 66
OutlineOutline
1.1. IntroductionIntroduction2.2. Bayesian Image Analysis by Gauss Markov Bayesian Image Analysis by Gauss Markov
Random FieldsRandom Fields3.3. Statistical Performance Analysis for Gauss Statistical Performance Analysis for Gauss
Markov Random FieldsMarkov Random Fields4.4. Statistical Performance Analysis in Binary Statistical Performance Analysis in Binary
Markov Random Fields by Loopy Belief Markov Random Fields by Loopy Belief PropagationPropagation
5.5. Concluding RemarksConcluding Remarks
Image Restoration by Bayesian Statistics
Original Image
14 October, 2010 7LRI Seminar 2010 (Univ. Paris-Sud)
Image Restoration by Bayesian Statistics
Original Image
Degraded Image
Transmission
Noise
14 October, 2010 8LRI Seminar 2010 (Univ. Paris-Sud)
Image Restoration by Bayesian Statistics
Original Image
Degraded Image
Transmission
Noise
Estimate
14 October, 2010 9LRI Seminar 2010 (Univ. Paris-Sud)
10
Image Restoration by Bayesian Statistics
PriorProcessn Degradatio
Posterior
}Image OriginalPr{}Image Original|Image DegradedPr{
}Image Degraded|Image OriginalPr{
Bayes Formula
Original Image
Degraded Image
Transmission
Noise
Estimate
Posterior
14 October, 2010 10LRI Seminar 2010 (Univ. Paris-Sud)
11
Image Restoration by Bayesian Statistics
PriorProcessn Degradatio
Posterior
}Image OriginalPr{}Image Original|Image DegradedPr{
}Image Degraded|Image OriginalPr{
Assumption 1: Original images are randomly generated by according to a prior probability.
Bayes Formula
Original Image
Degraded Image
Transmission
Noise
Estimate
Posterior
14 October, 201011LRI Seminar 2010 (Univ. Paris-Sud)
7 July, 2010 12
Image Restoration by Bayesian Statistics
PriorProcessn Degradatio
Posterior
}Image OriginalPr{}Image Original|Image DegradedPr{
}Image Degraded|Image OriginalPr{
Bayes Formula
Assumption 2: Degraded images are randomly generated from the original image by according to a conditional probability of degradation process.
Original Image
Degraded Image
Transmission
Noise
Estimate
Posterior
12LRI Seminar 2010 (Univ. Paris-Sud)14 October, 2010
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 13
Bayesian Image Analysis
Prior Probability
Ejiji
Ejiji fffffF
},{
2
},{
2 )(2
1exp)(
2
1exp}|Pr{
Assumption: Prior Probability consists of a product of functions defined on the neighbouring pixels.
Prior
Likelihood
Posterior
}Image OriginalPr{
}Image Original|Image DegradedPr{
}Image Degraded|Image OriginalPr{
0
14
Bayesian Image Analysis
Prior Probability
Ejiji
Ejiji fffffF
},{
2
},{
2 )(2
1exp)(
2
1exp}|Pr{
Assumption: Prior Probability consists of a product of functions defined on the neighbouring pixels.
Prior
Likelihood
Posterior
}Image OriginalPr{
}Image Original|Image DegradedPr{
}Image Degraded|Image OriginalPr{
0
14 October, 2010 14LRI Seminar 2010 (Univ. Paris-Sud)
15
Bayesian Image Analysis
0005.0 0030.00001.0
Patterns by MCMC.
Prior Probability
Ejiji
Ejiji fffffF
},{
2
},{
2 )(2
1exp)(
2
1exp}|Pr{
Assumption: Prior Probability consists of a product of functions defined on the neighbouring pixels.
Prior
Likelihood
Posterior
}Image OriginalPr{
}Image Original|Image DegradedPr{
}Image Degraded|Image OriginalPr{
0
14 October, 2010 15LRI Seminar 2010 (Univ. Paris-Sud)
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 16
Bayesian Image Analysis
Assumption: Degraded image is generated from the original image by Additive White Gaussian Noise.
Prior
Likelihood
Posterior
}Image OriginalPr{
}Image Original|Image DegradedPr{
}Image Degraded|Image OriginalPr{
Viii fg
fFgG
22
)(2
1exp
},|Pr{
0V:Set of all the pixels
17
Bayesian Image Analysis
Assumption: Degraded image is generated from the original image by Additive White Gaussian Noise.
Prior
Likelihood
Posterior
}Image OriginalPr{
}Image Original|Image DegradedPr{
}Image Degraded|Image OriginalPr{
14 October, 2010 17LRI Seminar 2010 (Univ. Paris-Sud)
Viii fg
fFgG
22
)(2
1exp
},|Pr{
18
Bayesian Image AnalysisAssumption: Degraded image is generated from the original image by Additive White Gaussian Noise.
Prior
Likelihood
Posterior
}Image OriginalPr{
}Image Original|Image DegradedPr{
}Image Degraded|Image OriginalPr{
Viii fg
fFgG
22
)(2
1exp
},|Pr{
0
14 October, 2010 18LRI Seminar 2010 (Univ. Paris-Sud)
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 19
Bayesian Image Analysis
f
g
}|Pr{ fF
},|Pr{ fFgG
gOriginal
Image
Degraded Image
Prior Probability
Posterior Probability
Degradation Process
Eji
jiVi
ii ffgf
gG
fFfFgG
gGfF
},{
222
)(2
1)(
2
1exp
},|Pr{
},Pr{},|Pr{
},,|Pr{
Estimate
20
Bayesian Image Analysis
f
g
}|Pr{ fF
},|Pr{ fFgG
gOriginal
Image
Degraded Image
Prior Probability
Posterior Probability
Degradation Process
Eji
jiVi
ii ffgf
gG
fFfFgG
gGfF
},{
222
)(2
1)(
2
1exp
},|Pr{
},Pr{},|Pr{
},,|Pr{
SmoothingData Dominant
Estimate
14 October, 2010 20LRI Seminar 2010 (Univ. Paris-Sud)
11 March, 2010 IW-SMI2010 (Kyoto) 21
Bayesian Image Analysis
f
g
}|Pr{ fF
},|Pr{ fFgG
gOriginal
Image
Degraded Image
Prior Probability
Posterior Probability
Degradation Process
Eji
jiVi
ii ffgf
gG
fFfFgG
gGfF
},{
222
)(2
1)(
2
1exp
},|Pr{
},Pr{},|Pr{
},,|Pr{
SmoothingData Dominant
Bayesian Network Estimate
14 October, 2010 21LRI Seminar 2010 (Univ. Paris-Sud)
gzdgGzFz
ghf
�
12
},,|Pr{
),,(ˆ
CI
22
Bayesian Image Analysis
f
g
}|Pr{ fF
},|Pr{ fFgG
gOriginal
Image
Degraded Image
Prior ProbabilityPosterior Probability
Degradation Process
Eji
jiVi
ii ffgf
gG
fFfFgG
gGfF
},{
222
)(2
1)(
2
1exp
},|Pr{
},Pr{},|Pr{
},,|Pr{
Field Random Markov Gauss
),(
iF
SmoothingData Dominant
Bayesian Network Estimate
14 October, 2010 22LRI Seminar 2010 (Univ. Paris-Sud)
otherwise,0
},{,1
|,|
Eji
Vjii
ji C
14 October, 201014 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud)LRI Seminar 2010 (Univ. Paris-Sud) 2323
Image Restorations by Image Restorations by Gaussian Markov Random Fields Gaussian Markov Random Fields
and Conventional Filtersand Conventional Filters
2||
ˆ||
||
1MSE
Vi
ffV
MSE
Gauss Markov Random Field
315
Lowpass Filter
(3x3) 388
(5x5) 413
Median Filter
(3x3) 486
(5x5) 445
(3x3) Lowpass(3x3) Lowpass (5x5) Median(5x5) MedianGauss Markov Gauss Markov Random FieldRandom Field
Original ImageOriginal Image Degraded ImageDegraded Image
RestoredRestoredImageImage VV: Set of all : Set of all
the pixelsthe pixels
Field Random Markov Gauss
),(
iF
14 October, 201014 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud)LRI Seminar 2010 (Univ. Paris-Sud) 2424
OutlineOutline
1.1. IntroductionIntroduction2.2. Bayesian Image Analysis by Gauss Markov Bayesian Image Analysis by Gauss Markov
Random FieldsRandom Fields3.3. Statistical Performance Analysis for Gauss Statistical Performance Analysis for Gauss
Markov Random FieldsMarkov Random Fields4.4. Statistical Performance Analysis in Binary Statistical Performance Analysis in Binary
Markov Random Fields by Loopy Belief Markov Random Fields by Loopy Belief PropagationPropagation
5.5. Concluding RemarksConcluding Remarks
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 25
Statistical Performance by Sample Average of Numerical Experiments
f
Original Images
26
Statistical Performance by Sample Average of Numerical Experiments
1g
f
2g
3g
4g
5g
Original Images Noi
se P
r{G
|F=f,}
ObservedData
14 October, 2010 26LRI Seminar 2010 (Univ. Paris-Sud)
11 March, 2010 IW-SMI2010 (Kyoto) 27
Statistical Performance by Sample Average of Numerical Experiments
1g
f
2g
3g
4g
5g
1h
2h
3h
4h
4h
Pos
teri
or P
rob
abil
ity
Pr{F
|G=g,
,}
Original Images Noi
se P
r{G
|F=f,}
Estimated ResultsObserved
Data
14 October, 2010 27LRI Seminar 2010 (Univ. Paris-Sud)
28
Statistical Performance by Sample Average of Numerical Experiments
1g
f
2g
3g
4g
5g
1h
2h
3h
4h
4h
Pos
teri
or P
rob
abil
ity
Pr{F
|G=g,
,}
Sample Average of Mean Square Error
Original Images Noi
se P
r{G
|F=f,}
5
1
2||||5
1),|,(
nnhffD
Estimated ResultsObserved
Data
14 October, 2010 28LRI Seminar 2010 (Univ. Paris-Sud)
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 29
Statistical Performance Estimation
gdfFgGfghV
fD
},|Pr{),,(1
),|,(2
ggh
12
),,(
CI
g
g
Additive White Gaussian Noise
},|Pr{ fFgG
f
},,|Pr{ gGfF
Posterior Probability
RestoredImage
Original Image Degraded Image
},|Pr{ fFgG
Additive White Gaussian Noise
otherwise,0
},{,1
|,|
Eji
Vjii
ji C
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 30
Statistical Performance Estimation for Gauss Markov Random Fields
T22
242
22
2
2
2
||2
2
2
)(||
1
)(Tr
1
2
1exp
2
11
},|Pr{),,(1
),|,(
fI
fVV
gdfggfV
gdfFgGghfV
fD
V
C
C
CI
I
CI
I
otherwise
Eji
Vjii
ji
,0
},{,1
|,|
C
0
200
400
600
0 0.001 0.002 0.003
=40
0
200
400
600
0 0.001 0.002 0.003
=40
),|,( fD
),|,( fD
OutlineOutline
1.1. IntroductionIntroduction2.2. Bayesian Image Analysis by Gauss Markov Bayesian Image Analysis by Gauss Markov
Random FieldsRandom Fields3.3. Statistical Performance Analysis for Gauss Statistical Performance Analysis for Gauss
Markov Random FieldsMarkov Random Fields4.4. Statistical Performance Analysis in Binary Statistical Performance Analysis in Binary
Markov Random Fields by Loopy Belief Markov Random Fields by Loopy Belief PropagationPropagation
5.5. Concluding RemarksConcluding Remarks
14 October, 201014 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud)LRI Seminar 2010 (Univ. Paris-Sud) 3131
32
Marginal Probability in Belief Propagation
1 3 4
,,,,,PrPr 43212F F F F
N
N
FFFFFF gGgG
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud)
In order to compute the marginal probability Pr{F2|G=g},we take summations over all the pixels except the pixel 2.
33
Marginal Probability in Belief Propagation
1 3 4F F F FN
2
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud)
1 3 4
,,,,,PrPr 43212F F F F
N
N
FFFFFF gGgG
34
Marginal Probability in Belief Propagation
1 3 4F F F FN
2 2
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud)
1 3 4
,,,,,PrPr 43212F F F F
N
N
FFFFFF gGgG
In the belief propagation, the marginal probability Pr{F2|G=g} is approximately expressed in terms of the messages from the neighbouring region of the pixel 2.
35
Marginal Probability in Belief Propagation
3 4F F FN
1 2
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud)
In order to compute the marginal probability Pr{F1,F2|G=g},we take summations over all the pixels except the pixels 1 and2.
3 4
,,,,,Pr,Pr 432121F F F
N
N
FFFFFFF gGgG
36
Marginal Probability in Belief Propagation
3 4F F FN
1 2 1 2
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud)
3 4
,,,,,Pr,Pr 432121F F F
N
N
FFFFFFF gGgG
In the belief propagation, the marginal probability Pr{F1,F2|G=g} is approximately expressed in terms of the messages from the neighbouring region of the pixels 1 and 2.
Belief Propagation in Probabilistic Image Processing
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 37
Belief Propagation in Probabilistic Image Processing
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 38
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 39
Image Restorations by Image Restorations by Gaussian Markov Random Fields Gaussian Markov Random Fields and Conventional Filtersand Conventional Filters
MSE
Gauss MRF (Exact) 315
Gauss MRF (Belief Propagation)
327
Lowpass Filter
(3x3) 388
(5x5) 413
Median Filter
(3x3) 486
(5x5) 445
Belief PropagationExact
Original Image Degraded Image
RestoredRestoredImageImage
VV: Set of all : Set of all the pixelsthe pixels
Field Random Markov Gauss
),(
iF
2ˆ||
1MSE
Vi
ii ffV
40
Gray-Level Image Restoration(Spike Noise)
Original Image
MSE:135MSE: 217
MSE: 371 MSE: 523
MSE: 244
MSE: 3469
MSE: 2075
Degraded Image
Belief Propagation Lowpass Filter Median Filter
MSE: 395
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud)
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 41
Binary Image Restoration byLoopy Belief Propagation
42
Statistical Performance by Sample Average of Numerical Experiments
1g
f
2g
3g
4g
5g
1h
2h
3h
4h
4h
Pos
teri
or P
rob
abil
ity
Pr{F
|G=g,
,}
Sample Average of Mean Square Error
Original Images Noi
se P
r{G
|F=f,}
5
1
2||||5
1),|,(
nnhffD
Estimated ResultsObserved
Data
14 October, 2010 42LRI Seminar 2010 (Univ. Paris-Sud)
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 43
Statistical Performance Estimation for Binary Markov Random Fields
gdfFgGfghV
fD
},|Pr{),,(1
),|,(2
||211 1 1 },{
222
22
1 2 ||
)(2
1)(
2
1explog
)(2
1exp
Vz z z Eji
jiVi
ii
Viii
dgdgdgzzgz
fg
V
It can be reduced to the calculation of the average of free energy with respect to locally non-uniform external fields g1, g2,…,g|V|.
Free Energy of Ising Model with Random External Fields
1ifLight intensities of the original image can be regarded as spin states of ferromagnetic system.
== >
== >
Eji },{
Eji
jiVi
ii ffgf
gGfF
},{
222
)(2
1)(
2
1exp
},,|Pr{
Statistical Performance Estimation for Binary Markov Random Fields
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 44
},|Pr{)(),,(||
1
},|Pr{),,(1
),|,(
2
2
iiiiij
ijijVi
iiij
iji fFgGghfddgV
gdfFgGfghV
fD
},|Pr{)(
tanh)tanh(tanh
)(
*
}/{
}/{2
1
}{\
jjjjijk
jkjk
ijljl
iij
Vi ijkjkiijij
fFgG
g
ddg
ijij
ii
ggh
2sgn),,(
gdfFgGfghV
fD
},|Pr{),,(1
),|,(2
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 45
0
0.1
0.2
0.3
0.4
0 0.2 0.4 0.6 0.8 1
Statistical Performance Estimation for Markov Random Fields
0
200
400
600
0 0.001 0.002 0.003
=40=1
),|,( fD
Spin Glass Theory in Statistical MechanicsLoopy Belief Propagation
0.8
0.6
0.4
0.2
),|,( fD
Multi-dimensional Gauss Integral Formulas
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 46
Statistical Performance Estimation for Markov Random Fields
gdfFgGghfV
fD
},|Pr{),,(1
),|,(2
OutlineOutline
1.1. IntroductionIntroduction2.2. Bayesian Image Analysis by Gauss Markov Bayesian Image Analysis by Gauss Markov
Random FieldsRandom Fields3.3. Statistical Performance Analysis for Gauss Statistical Performance Analysis for Gauss
Markov Random FieldsMarkov Random Fields4.4. Statistical Performance Analysis in Binary Statistical Performance Analysis in Binary
Markov Random Fields by Loopy Belief Markov Random Fields by Loopy Belief PropagationPropagation
5.5. Concluding RemarksConcluding Remarks
14 October, 201014 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud)LRI Seminar 2010 (Univ. Paris-Sud) 4747
14 October, 201014 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud)LRI Seminar 2010 (Univ. Paris-Sud) 4848
SummarySummary
Formulation of probabilistic model for image Formulation of probabilistic model for image processing by means of conventional statistical processing by means of conventional statistical schemes has been summarized.schemes has been summarized.Statistical performance analysis of probabilistic Statistical performance analysis of probabilistic image processing by using Gauss Markov Random image processing by using Gauss Markov Random Fields has been shown.Fields has been shown.One of extensions of statistical performance One of extensions of statistical performance estimation to probabilistic image processing with estimation to probabilistic image processing with discretediscrete states has been demonstrated.states has been demonstrated.
Image Impainting by Gauss MRF and LBP
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 49
Gauss MRFand LBP
Our framework can be extended to erase a scribbling.
14 October, 201014 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud)LRI Seminar 2010 (Univ. Paris-Sud) 5050
ReferencesReferences1. K. Tanaka and D. M. Titterington: Statistical Trajectory of Approximate EM
Algorithm for Probabilistic Image Processing, Journal of Physics A: Mathematical and Theoretical, vol.40, no.37, pp.11285-11300, 2007.
2. M. Yasuda and K. Tanaka: The Mathematical Structure of the Approximate Linear Response Relation, Journal of Physics A: Mathematical and Theoretical, vol.40, no.33, pp.9993-10007, 2007.
3. K. Tanaka and K. Tsuda: A Quantum-Statistical-Mechanical Extension of Gaussian Mixture Model, Journal of Physics: Conference Series, vol.95, article no.012023, pp.1-9, January 2008
4. K. Tanaka: Mathematical Structures of Loopy Belief Propagation and Cluster Variation Method, Journal of Physics: Conference Series, vol.143, article no.012023, pp.1-18, 2009
5. M. Yasuda and K. Tanaka: Approximate Learning Algorithm in Boltzmann Machines, Neural Computation, vol.21, no.11, pp.3130-3178, 2009.
6. S. Kataoka, M. Yasuda and K. Tanaka: Statistical Performance Analysis in Probabilistic Image Processing, Journal of the Physical Society of Japan, vol.79, no.2, article no.025001, 2010.