Kazuyuki Tanaka Graduate School of Information Sciences, Tohoku University

1

Physical Fluctuomatics5th and 6th Probabilistic information processing

by Gaussian graphical model

Kazuyuki TanakaGraduate School of Information Sciences, Tohoku University

[email protected]://www.smapip.is.tohoku.ac.jp/~kazu/

Physical Fluctuomatics (Tohoku University)

Physical Fluctuomatics (Tohoku University) 2

Textbooks

Kazuyuki Tanaka: Introduction of Image Processing by Probabilistic Models, Morikita Publishing Co., Ltd., 2006 (in Japanese) , Chapter 7.K. Tanaka: Statistical-mechanical approach to image processing (Topical Review), Journal of Physics A: Mathematical and General, vol.35, no.37, pp.R81-R150, 2002, Section 4.


Contents

1. Introduction2. Probabilistic Image

Processing3. Gaussian Graphical Model4. Statistical Performance

Analysis5. Concluding Remarks


Contents





Markov Random Fields for Image Processing

S. Geman and D. Geman (1986): IEEE Transactions on PAMIImage Processing for Markov Random Fields (MRF) (Simulated Annealing, Line Fields)

J. Zhang (1992): IEEE Transactions on Signal ProcessingImage Processing in EM algorithm for Markov Random Fields (MRF) (Mean Field Methods)

Markov Random Fields are One of Probabilistic Methods for Image processing.


Markov Random Fields for Image Processing

In Markov Random Fields, we have to consider not only the states with high probabilities but also ones with low probabilities.In Markov Random Fields, we have to estimate not only the image but also hyperparameters in the probabilistic model.We have to perform the calculations of statistical quantities repeatedly.

Hyperparameter Estimation

Statistical Quantities

Estimation of Image

We can calculate statistical quantities by adopting the Gaussian graphical model as a prior probabilistic model and by using Gaussian integral formulas.


Purpose of My TalkReview of formulation of probabilistic model for image processing by means of conventional statistical schemes.Review of probabilistic image processing by using Gaussian graphical model (Gaussian Markov Random Fields) as the most basic example.

K. Tanaka: Statistical-Mechanical Approach to Image Processing (Topical Review), J. Phys. A: Math. Gen., vol.35, pp.R81-R150, 2002.

Section 2 and Section 4 are summarized in the present talk.


Contents





Bayes Formula and Bayesian Network

Posterior Probability

}Pr{

}Pr{}|Pr{}|Pr{

B

AABBA

Bayes Rule

Prior Probability

Event B is given as the observed data.Event A corresponds to the original information to estimate. Thus the Bayes formula can be applied to the estimation of the original information from the given data.

A

B

Bayesian Network

Data-Generating Process


Image Restoration by Probabilistic Model

Original Image

Degraded Image

Transmission

Noise

Likelihood Marginal

PriorLikelihood

Posterior

}ageDegradedImPr{

}Image OriginalPr{}Image Original|Image DegradedPr{

}Image Degraded|Image OriginalPr{

Assumption 1: The degraded image is randomly generated from the original image by according to the degradation process. Assumption 2: The original image is randomly generated by according to the prior probability.

Bayes Formula


Image Restoration by Probabilistic Model

Degraded

Image

i

fi: Light Intensity of Pixel iin Original Image

),( iii yxr

Position Vector

of Pixel i

gi: Light Intensity of Pixel iin Degraded Image

i

Original

Image

The original images and degraded images are represented by f = (f1,f2,…,f|V|) and g = (g1,g2,…,g|V|), respectively.


Probabilistic Modeling of Image Restoration

PriorLikelihood

Posterior



Vi

ii fg ),(}|Pr{

}Image Original|Image DegradedPr{

fFgG

fg

Random Fieldsfi

gi

fi

gi

or

Assumption 1: A given degraded image is obtained from the original image by changing the state of each pixel to another state by the same probability, independently of the other pixels.

),...,,( Image Degraded of Field Random

),...,,( Image Original of Field Random

||21

||21

V

V

GGG

FFF

G

F

),...,,( Image Degraded ofVector State

),...,,( Image Original ofVector State

||21

||21

V

V

ggg

fff

g

f


Probabilistic Modeling of Image Restoration

PriorLikelihood

Posterior



Eij

ji ff ),(}Pr{

}Image OriginalPr{

fF

f

Random Fields

Assumption 2: The original image is generated according to a prior probability. Prior Probability consists of a product of functions defined on the neighbouring pixels.

i j

Product over All the Nearest Neighbour Pairs of Pixels

),...,,( Image Original ofVector State

),...,,( Image Original of Field Random

||21

||21

V

V

fff

FFF

f

F


Bayesian Image Analysis

Eji

jiVi

ii ffgf},{

),(),(

Pr

PrPrPr

gG

fFfFgGgGfF

fg

fF Pr fFgG Pr gOriginalImage

Degraded Image

Prior Probability


Degradation Process

Image processing is reduced to calculations of averages, variances and co-variances in the posterior probability.

E ： Set of all the nearest neighbour pairs of pixels

V ： Set of All the pixels


Estimation of Original Image

We have some choices to estimate the restored image from posterior probability. In each choice, the computational time is generally exponential order of the number of pixels.

}|Pr{maxargˆ gG iiz

i zFfi

2}|Pr{minargˆ zgGzF dzf iiii

}|Pr{maxargˆ gGzFfz

Thresholded Posterior Mean (TPM) estimation

Maximum posterior marginal (MPM) estimation

Maximum A Posteriori (MAP) estimation

if

ii fF\

}|Pr{}|Pr{f

gGfFgG

(1)

(2)

(3)


Contents





Bayesian Image Analysis by Gaussian Graphical Model

Ejiji ff

Z },{

2

PR 2

1exp

)(

1|Pr

fF

0005.0 0030.00001.0

Patterns are generated by MCMC.

Markov Chain Monte Carlo Method

PriorProbability

,if

E:Set of all the nearest-neighbour pairs of pixels

V:Set of all the pixels

||21},{

2PR 2

1exp)( V

Ejiji dzdzdzzzZ



2,0~ Nfg ii

Viii gf 2

22 2

1exp

2

1,Pr

fFgG

Histogram of Gaussian Random Numbers

,, ii gf

Degraded image is obtained by adding a white Gaussian noise to the original image.

Degradation Process is assumed to be the additive white Gaussian noise.

V: Set of all the pixels

Original Image f Gaussian Noise n

Degraded Image g


Bayesian Image Analysis

Eji

jiVi

ii ffgf},{

),(),(

Pr

PrPrPr

gG

fFfFgGgGfF

fg

fF Pr fFgG Pr gOriginalImage

Degraded Image

Prior Probability


Degradation Process

Image processing is reduced to calculations of averages, variances and co-variances in the posterior probability.

E ： Set of all the nearest neighbour pairs of pixels

V ： Set of All the pixels


Bayesian Image Analysis ,, ji gf

Ejiji

Viii

Eji

ff

jiVi

gf

ii

ffgf

ffgf

jiii

},{

},{

),(

2

),(

2

2

),(),(

2

1exp

2

1exp

},|Pr{

}|Pr{},|Pr{},,|Pr{

gG

fFfFgGgGfF

A Posteriori Probability


Statistical Estimation of Hyperparameters

zzFzFgG

zgGzFgG

d

d

}|Pr{},|Pr{

},|,Pr{},|Pr{

},|Pr{max arg)ˆ,ˆ(

,

gG

f g

Marginalized with respect to F

}|Pr{ fF },|Pr{ fFgG gOriginal Image

Marginal Likelihood

Degraded ImageVy

x

},|Pr{ gG

Hyperparameters , a s are determined so as to maximize the marginal likelihood Pr{G=g|a,s} with respect to a, .s

zgGzFzf d }ˆ,ˆ,|Pr{ˆ


Bayesian Image Analysis ,, ji gf

TT2

POS

},{

22

2POS

2

1))((

2

1exp

,,

1

2

1

2

1exp

,,

1

},|Pr{

}|Pr{},|Pr{},,|Pr{

fCfgfgfg

g

gG

fFfFgGgGfF

Z

ffgfZ Eji

jiVi

ii

A Posteriori Probability

Gaussian Graphical Model

||21

TT2POS

2

1))((

2

1exp,, VdzdzdzZ zCzgzgzg

otherwise0

},{1

4

Eji

ji

ji C

|V|x|V| matrix

),...,,(

),...,,(

),...,,(

||21

||21

||21

V

V

V

zzz

ggg

fff

z

g

f


Average of Posterior Probability

)0,00())()((2

1exp)(

||

||21T2

V

V ,,dzdzdz

mzCI mz mz

gCI

I

CI

IzCI

CI

Iz

CI

IzCI

CI

Izz

gGzFzF

2

||21

T

22

22

||21

T

22

22

||21

2

1exp

2

1exp

},,|Pr{,

V

V

V

dzdzdzgg

dzdzdzgg

dzdzdz

E

Gaussian Integral formula


gCI

I

zgGzFzf

2

},,|Pr{ˆ

d

Bayesian Image Analysis by Gaussian Graphical Model ,, ii gf

otherwise0

},{1

4

Eji

ji

ji C

Multi-Dimensional Gaussian Integral Formula

Ejiji

Viii ffgf

},{

22

2 2

1

2

1exp},,|Pr{

gGfF


Average of the posterior probability can be calculated by using the multi-dimensional Gauss integral Formula

|V|x|V| matrix

E:Set of all the nearest-neghbour pairs of pixels



Statistical Estimation of Hyperparameters

)()2(

),,(}|Pr{},|Pr{},|Pr{

PR2/||2

POS

Z

Zd

V

gzzFzFgGgG

f g

Marginalized with respect to F

}|Pr{ fF },|Pr{ fFgG gOriginal Image

Marginal Likelihood

Degraded ImageVy

x

},|Pr{ gG

||21

TT2

POS

2

1))((

2

1exp

,,

Vdzdzdz

Z

zCzgzgz

g

||21

TPR

2

1exp VdzdzdzZ zCz


Calculations of Partition Function

AmzA mz

det

)2()( )(

2

1exp

||

||21T

V

Vdzdzdz

T22

||2

||21

T

22

22

T2

||21T

2

T

22

22

POS

2

1exp

)det(

)2(

2

1exp

2

1exp

2

1

2

1exp

),,(

gCI

Cg

CI

gCI

IzCIg

CI

Iz

gCI

Cg

gCI

Cgg

CI

IzCIg

CI

Iz

g

V

V

V

dzdzdz

dzdzdz

Z

(A is a real symmetric and positive definite matrix.)Gaussian Integral formula

CCzz

det

)2(

2

1exp)(

||

||

||21 V

V

VPR dzdzdzZ


Exact expression of Marginal Likelihood in Gaussian Graphical Model

T

22 2

1exp

det2

det},|Pr{ g

CI

Cg

CI

CgG

V

gCI

If

2ˆ

Multi-dimensional Gauss integral formula

Vy

x

},|Pr{max argˆ,ˆ,

gG We can construct an exact EM algorithm.

)()2(

),,(}|Pr{},|Pr{},|Pr{

PR2/||2

POS

Z

Zd

V

gzzFzFgGgG



1T

22

2

11||

1

11

1Tr

||

1

g

CI

Cg

CI

C

ttVtt

t

Vt

T22

242

2

2

11

11

||

1

11

1Tr

||

1g

CI

Cg

CI

I

tt

tt

Vtt

t

Vt

0

0.0002

0.0004

0.0006

0.0008

0.001

0 20 40 60 80 100 t

t

g f̂

gCI

Im

2)()()(

ttt

2)(minarg)(ˆ tmztf ii

zi

i

Iteration Procedure in Gaussian Graphical Model

f̂

g


Image Restoration by Markov Random Field Model and Conventional Filters

2ˆ||

1MSE

Viii ff

V

MSE

Statistical Method 315

Lowpass Filter

(3x3) 388

(5x5) 413

Median Filter

(3x3) 486

(5x5) 445

(3x3) Lowpass (5x5) MedianMRF

Original Image Degraded Image

RestoredImage



Contents





Performance Analysis

1y

x

2y

3y

4y

5y

1x̂

2x̂

3x̂

4x̂

5x̂

Pos

teri

or P

rob

abil

ity

Estimated ResultsObserved

Data

Sample Average of Mean Square Error

SignalA

dd

itiv

e W

hit

e G

auss

ian

Noi

se

5

1

2||ˆ||5

1][MSE

nnxx


Statistical Performance Analysis

ydxXyYxyhV

x

},|Pr{),,(1

),|(MSE2

),,( yh

gy

Additive White Gaussian Noise

},|Pr{ xXyY

x

},,|Pr{ yYxX


Restored Image

Original Image Degraded Image

},|Pr{ xXyY

Additive White Gaussian Noise


Statistical Performance Analysis

ydxXyYxyV

ydxXyYxyhV

x

Pr1

Pr,,1

),|MSE(

212

2

CI

y

xdyxPxyh

12

)|(,,

CI

)2

1exp(

2

1exp,Pr

22

22

yx

yxxXyYVi

ii

otherwise,0

},{,1

,4

Eji

Vji

ji C


Statistical Performance Estimation for Gaussian Markov Random Fields

T22

242

22

2

22

||

22

242T

22

||

22

2T

22

||

22

2T

22

||

22T

22

||

2

2

2

T

2

2

2

22

||2

2

2

2

22

||2

22

22

||2

2

2

)(||

1

)(Tr

1

2

1exp

2

1

)(

1

2

1exp

2

1)(

)(

1

2

1exp

2

1

)()(

1

2

1exp

2

1)(

)()(

1

2

1exp

2

1)()(

1

2

1exp

2

1)(

1

2

1exp

2

1)(

1

2

1exp

2

11

},|Pr{),,(1

),|(MSE

xI

xVV

ydxyxxV

ydxyxyxV

ydxyxxyV

ydxyxyxyV

ydxyxxyxxyV

ydxyxxyV

ydxyxxxyV

ydyxxyV

ydxXyYxyhV

x

V

V

V

V

V

V

V

V

C

C

CI

I

CI

C

CI

C

CI

C

CI

I

CI

C

CI

I

CI

C

CI

I

CI

C

CI

I

CI

I

CI

I

CI

I

= 0

otherwise,0

},{,1

,4

Eji

Vji

ji C


Statistical Performance Estimation for Gaussian Markov Random Fields

xI

xVV

ydyxxyV

ydxXyYxyhV

x

V

22

242T

22

2

22

||

2

2

2

2

)(||

1

)(Tr

1

2

1exp

2

11

},|Pr{),,(1

),|(MSE

C

C

CI

I

CI

I

otherwise,0

},{,1

,4

Eji

Vji

ji C

0

200

400

600

0 0.001 0.002 0.003a

s=40

0

200

400

600

0 0.001 0.002 0.003

s=40

a

),|(MSE x ),|(MSE x


Contents





Summary

Formulation of probabilistic model for image processing by means of conventional statistical schemes has been summarized.

Probabilistic image processing by using Gaussian graphical model has been shown as the most basic example.

References

K. Tanaka: Introduction of Image Processing by Probabilistic Models, Morikita Publishing Co., Ltd., 2006 (in Japanese) .

K. Tanaka: Statistical-Mechanical Approach to Image Processing (Topical Review), J. Phys. A, 35 (2002).

A. S. Willsky: Multiresolution Markov Models for Signal and Image Processing, Proceedings of IEEE, 90 (2002).



Problem 5-1: Derive the expression of the posterior probability Pr{F=f|G=g,a,s} by using Bayes formulas Pr{F=f|G=g,a,s}=Pr{G=g|F=f,s}Pr{F=f,a}/Pr{G=g|a,s}. Here Pr{G=g|F=f,s} and Pr{F=f,a} are assumed to be as follows:

Viii gf 2

22 2

1exp

2

1,Pr

fFgG

Ejiji ff

Z },{

2

PR 2

1exp

)(

1|Pr

fF

||21},{

2PR 2

1exp)( V

Ejiji dzdzdzzzZ

Ejiji

Viii ffgf

Z },{

22

2POS 2

1

2

1exp

,,

1},,|Pr{

ggGfF

||21

},{

22

2PR 2

1

2

1exp V

Ejiji

Viii dzdzdzzzgzZ

[Answer]


Problem 5-2: Show the following equality.

TT2

},{

22

2

2

1))((

2

1

2

1

2

1

fCfgfgf

Eji

jiVi

ii ffgf

otherwise0

},{1

4

Eji

ji

ji C


Problem 5-3: Show the following equality.

T

2

T

22

22

TT2

2

1

2

1

2

1))((

2

1

gCI

Cg

gCI

IzCIg

CI

Iz

zCzgzgz


Problem 5-4: Show the following equalities by using the multi-dimensional Gaussian integral formulas.

T22

||2

||21},{

22

2POS

2

1exp

)det(

)2(

2

1

2

1exp,,

gCI

Cg

CI

g

V

VEji

jiVi

ii dzdzdzffgfZ

Cdet

)2(

2

1exp)(

||

||

||21},{

2

V

V

VEji

jiPR dzdzdzffZ


Problem 5-5: Derive the extremum conditions for the following marginal likelihood Pr{G=g| ,a s} with respect to the hyperparameters a and s.

T

22 2

1exp

det2

det},|Pr{ g

CI

Cg

CI

CgG

V

T

22

2

||

1Tr

||

11g

CI

Cg

CI

C

VV

T

22

242

2

22

||

1Tr

||

1g

CI

Cg

CI

I

VV

[Answer]


Problem 5-6: Derive the extremum conditions for the following marginal likelihood Pr{G=g| ,a s} with respect to the hyperparameters a and s.

T

22 2

1exp

det2

det},|Pr{ g

CI

Cg

CI

CgG

V

T

22

2

||

1Tr

||

11g

CI

Cg

CI

C

VV

T

22

242

2

22

||

1Tr

||

1g

CI

Cg

CI

I

VV

[Answer]


Problem 5-7: Make a program that generate a degraded image by the additive white Gaussian noise. Generate some degraded images from a given standard images by setting s=10,20,30,40 numerically. Calculate the mean square error (MSE) between the original image and the degraded image.

Histogram of Gaussian Random Numbers Fi -Gi~N(0,402)

Original Image Gaussian Noise Degraded Image

Sample Program: http://www.morikita.co.jp/soft/84661/

2||

1MSE

Viii gf

V

K.Tanaka: Introduction of Image Processing by Probabilistic Models, Morikita Publishing Co., Ltd., 2006 .


Problem 5-8: Make a program of the following procedure in probabilistic image processing by using the Gaussian graphical model and the additive white Gaussian noise.

1T

22

2

11||

1

11

1Tr

||

1

g

CI

Cg

CI

C

ttVtt

t

Vt

T22

242

2

2

11

11

||

1

11

1Tr

||

1g

CI

Cg

CI

I

tt

tt

Vtt

t

Vt

gCI

Im

2)()()(

ttt

2)(minarg)(ˆ tmztf ii

zi

i

Algorithm: Repeat the following procedure until convergence

Sample Program: http://www.morikita.co.jp/soft/84661/

K.Tanaka: Introduction of Image Processing by Probabilistic Models, Morikita Publishing Co., Ltd., 2006 .

Documents

Kazuyuki Tanaka Graduate School of Information Sciences, Tohoku University