Tokyo Institute of Technology, Japan

Yu Nishiyama and Sumio Watanabe

Theoretical Analysis of Accuracy of Gaussian Belief Propagation

Background

Belief propagation (BP)

The algorithm which computesmarginal distributions efficiently

),,,,()( 21

1

dxxx

ii xxxpxpdi

Marginal distribution:

requires huge computational cost.

1d

Variety of Research Areas

(i) Probabilistic inference for AI

(ii) Error correcting code (LDPC, Turbo codes)

(iii) Code division multiple access (CDMA)

ex.

(iv) Probabilistic image processing

000101 000111 000101correctingnoise

degradeimage

restoredimage

restore

Properties of BP & Loopy BP (LBP)

Tree-structured target distribution

　 Exact marginal probabilities

Loop-structured target distribution

Convergence?Approximate marginal probabilities

Y.　Weiss,”Correctness　of　belief　propagation　in　graphical　models　witharbitrary　topology”,　Neural Computation　13(10),　2173-2200,　2001.

T.　Heskes,　”On　the　Uniqueness　of　Loopy　Belief　Propagation　Fixed　Points”,Neural Computation　16(11),　2379-2414,　2004.

Ex.

PurposeWe analytically clarify the accuracy of LBP when the target distribution is a Gaussian distribution. 　

What is the conditions for LBP convergence?

How close is the LBP solution to the true marginal distributions?

K.　Tanaka,　H.　Shouno,　M.　Okada,　“Accuracy　of　the　Bethe　approximation　for　hyperparameter　estimation　in　probabilistic　image　processing”,J.phys.　A,　Math.　Gen.,　vol.37,　　no.36,　　pp.8675-8696,　2004.

In Probabilistic image processing

Table of Contents

・ BP　＆ LBP

・ Gaussian　Distribution

・Main　Results

(i)　Single　Loop

(ii)　Graphs　with　Multi-loops

・ Conclusion

Graphical Models

Bij

jiij xxWZ

p}{

),(1

)(x

Target distribution

1x2x

4x 5x

6x

12W

23W

34W

35W56W

Marginal distribution

3\ }{

33 ),(1

)(x Bij

jiij xxWZ

xpx

53 ,\ }{

5335 ),(1

),(xx Bij

jiij xxWZ

xxpx

3x

BP ＆ LBP

ix jxijW

)()()(

iiNk

ikii xMxp

)()(),(

\)(\)(jjk

ijNkij

jiNkiikjiij xMWxMxxp

Marginal　distribution

ix

1kx

2kx

)(iN)(iN )( jN

1kx 1k

x

How are messages decided?)( jji xM

)( jji xM Messages are　decided　by　the　fixed-points

of　a　message　update　rule:

.)(),()(}\{)(

)()1(

jiNk

itikji

xijj

tji xMxxWxM

i

1x 2x

4x 3x

)(21

tM

)(23

tM

)1(42

tM

)(43

tM

)(41

tM )1(24

tM

)1(32

tM

)1(34

tM

)(21

tM

)(24

tM

)(34

tM

)1(23

tM )(32

tM

)1(43

tM

)1(21

tM

)(14

tM )1(41

tM

)(12

tM )1(12

tM

)(41

tM )(42

tM )(42

tM

)(43

tM

)1(14

tM

)(23

tM

)(24

tM If　it　converges 1x 2x

4x 3x

*12M*

21M

*32M

*23M

*43M*

34M

*41M

*14M *

24M

*42M

a　fixed-point

Gaussian Distribution

jxix

mx

lx)(tjiM

)(tijM

)(tilM

)(timM

)(til

)(tij

)(tim

)(tji

Messages: ),0()( )()( tjij

tji NxM

}\{)(

)(

2)1(

~~

jiNk

tikii

ijii

tji s

ss

Update　rule:

Target　distribution:　 ),( SN 0 (　　　　　Inverse　covariance　matrix):S

}\{)(

)()1( )(),()(jiNk

itikji

xijj

tji xMxxWxM

i

Fixed-Points of Messages 1x

2x

3x

dx

Single　loop

When　a　Gaussian　distribution　forms　a　single　loop,　the　fixed-points　of　messages　are　given　by　　

,2 1,1

2,12,11,1,*1

ii

iiiiiiiiii

Dss),,0()( *

11*

1 iiiii NxM

Theorem1　

,det4)1()(det 123122 SsssSD d

d

,2 1,1

2,12,11,1,*1

ii

iiiiiiiiii

Dss),,0()( *

11*

1 iiiii NxM

where　　 }{ , ji are　the　cofactors.　　

LBP Solution

Theorem　2　

The　solution　of　LBP　is　given　by　　

,det

4)1(1

det 12312*

S

sssS dd

iii

),,0( **

ii Np

),,( *1,

*1, iiii SNp 0 ,

1,1

1,1,

1,,

1,

*1,

ii

iiii

iiii

ii

ii Es

sE

S

where .2

det1,1,1,

iiiiii s

DSE

Intuitive Understanding

S

sssS dd

iii det

4)1(1

det 12312*

LBP　Solution True

ii

S

det

12s 12s1x 2x

4x

23s41s1x 2x

4x

41s023 s

Loop Tree

Loopy　Belief　Propagation Belief　Propagation

3x3x34s 34s

Accuracy of LBP

.det

4)1( 12312

S

sss dd

The　Kullback-Leibler　(KL)　distances　are　calculated　as　

,1log2

11

2

1

2

1)||( * ii ppKL

where　is　given　by　

Solution　of　LBP　　　True　marginaldensity　　　

.1det*

ii

i

SConvergence　condition　is　 1 since　

Theorem　3　

},,,1{ di

Graphs with Multi-Loops

1x2x

3x

4x 5x

6x

Multi-loops　

How　about　the　graphshaving　arbitrary　structures?　

We　clarify　the　LBP　solution　at　small　covariances.

We　derive　the　expansions　　 ,sw.　r.　t. where　inverse　　.diagonaloffdiagonal sSSS covariance　matrix　is　

A Fixed-Point of Inverse Variances

A　fixed-point　of　inverse　variances　satisfies　the　following　system　of　equations:

,4

12

2||22

iNj ij

ji

i

iii

sssN }.,,1{ di

The　solution　of　the　system　is　expanded　as

),()( 422

)(1

sOss

sss

jj

jid

ijiii

}.,,1{ di

Theorem　4　

Comparison with true inverse variances

Expansions　of　LBP　solution　are

),()( 422

)(1

sOss

sss

jj

jid

ijiii

True　inverse　variances　are

}.,,1{ di

),(det 432

2

)(1

sOsss

ss

Si

jj

jid

ijii

ii

}.,,1{ di ],})(){()([3

3131iioiidod

iii SStrSStrs

Accuracy of LBP

Theorem　5　

The　Kullback-Leibler　(KL)　distances　are　expanded　as

),(4

)||( 762

2* sOs

sppKL

ii

iii

Solution　of　LBP　　　True　marginaldensity　　　

where　

],})(){()([3

3131iioiidod

iii SStrSStrs

}{ i are　

},,,1{ di

}.,,1{ di

Conclusion We analytically clarified the accuracy of

LBP in a Gaussian distribution.

(i)　For　a　single　loop,　　we　revealed　the　parameter　that　　　　　　　determines　the　accuracy　of　LBP　and　the　condition　

that　　　　　　tells　us　when　LBP　converges.　　　(ii)　For　arbitrary　structures,　　we　revealed　the　

expansions　of　　　　　LBP　solution　at　small　covariances　and　the　accuracy.　　

　　　　These　fundamental　results　contribute　to　understanding　the　theoretical　properties　underlying　LBP.