Mean Field Asymptotic in Statistical Learning . Apr 12

Lecture 21 .

Derivation of AMP .

I.

① From Belief propagation to message passing algorithm

wrong intuition : the beliefs are approximately Gaussian .

Intuition : In the update rule , only the mean and vom of input beliefs are important .

Input beliefs can be approximated by Gaussian disc .

in the update rule .

Real belief is still non- Gaussian.

Def ( Message passing algorithm) mean and variance of beliefs.

{ Misa ,Vita

,mita.si ,

Jhansi }aeF,iev

,WoE IR

. Messages .

ktl

Update rule : Calculating { misa .viii. niiiii

,Tia } using { mki-a.vika.miii.ii.si } .

I- E

pika Cx ; ) = Fi exp f- }

.

density of N(mita , Vika)→a

fat, exit =#exp f - C×izF÷÷ }.

density of Nlniaki , i'Isi)21T Takei

ri'alxi) & 4iki) Telma Fb it" ) Non- backtracking

÷÷:÷÷:÷÷i÷÷÷÷:÷÷÷÷÷:÷:::÷÷÷i:÷÷¥¥÷÷Extract marginals :

rikki ) & 4- Giblets ; f iGi )

( Pita ,Taka. } input beliefs

,

assumed to be Gaussian

{ rita ,Taji } output beliefs , possibly non - Gaussian

② Example : LASSO with finite temperature p

k

⑦ ri 's:'(Xi ) a expl - Bakit } x exp { - E,a7÷ }a expf - p talxil ) ) .

Pls 'iIa5'= Eat 's.it '

:÷÷÷÷÷:÷÷÷÷÷÷÷÷.¥:÷:c.Vitti = Varxinaep.a.oi.in

, s . (Xi )

where ICP .x. 0.5 ) d. exp { - PC 1- Nxt ) )

.

⑦ r:*. .sn:"::i÷÷÷÷÷÷%This is a Gaussian integration , which gives a Gaussian density .

a exp C - } .2J ktla→i

Aai-ma→i=Ya-¥iAajm'K

Aut - Iasi = ¥. Aaj'

Y'Eat f-

④ Simplification as p → • .

( This step is unnecessary if D= finite)

'ping Ex - acs.ao.si ( x )

= arg;nin{ + Nxt}= 710 ; d 3) Soft thresholding .

hi:b B. Varxnxip.x.gs, [ X] = f 5 ,if 101 > as

,

o, if 101 Cds

.

= S - n' ( O 's RS ).

We are short of alphabets . Abuse notation :

Mita ← him. Mika .

nia.

-

← him. niaki .

Vika ← times. P - Vita,

Tati← limo P - vats..

.

(Sita)" = TEA X 's.it '

(Si'

Oita = Ffa Fist' ni 's.si

mittal = 71 Oita ; dSita)÷÷÷÷:i÷i:÷i÷÷:÷÷z r k

Aai - Va-si= ¥. Aaj

'

Y'Ia t l

r k k

change of variable : zE→ ;= Aai -

ma → i= Ya - Fi Aaj mj→a

Takes ; E Aki - Jka.si = Fei Aaj'

Vjkatl

Assume Aai niid Unit ( { Ifn } ).

Can replace Aai by ht .

K Fa Abi Zb ( t k⇒ Oi→a =boilE t / 2b i

i EV

beta a EF

initial = moi'saids a) there are

ZE-si = Ya - 7¥. Aaj m a2x nxd messages,,±=µ⇐,,⇒viii

'

= Sit.

- n' ( Oita ,xsiia )

.

Takei = In ¥. YES . + I

③ From message passing to approximate message passing .

a÷÷÷:÷i..:÷:::::÷¥:::A crude approximation : throw away non-backtracking property .

O ,"=FAbizbk.lt#

k it iz

-1nF 'Hb mii.my/miiiaa a

mill = M ( Oik ; RSF ) miiiaiffmiisiZE = Ya - F Aaj mjks:::i÷÷:÷÷÷÷siTk = In E ujk + I ⇒ Tk = Lhjvjktla j

simplifying this givesT

mktl = M ( Mkt Azk ; ask) End.::.::÷÷÷i÷÷÷i::

Doesn't give a correct approx .

Needs a correction .

The equations for S,V and T are fine

.

The equations for O , m , Z are off by OG ) .

Derivation of AMP .

④ Replace Vika by Uk, Sita by Sk

.

Oi a = TEA Abi Zb ⑥

:÷÷::i÷÷÷÷i:÷3kt ' = skxtn.IQ, y

' ( Mkt Azk ; ask) ; t I

④ Lee Oita = Oi't S Oita

Misa = Mik t Smita

Zaki ; = Zak t 8 2- a'Ii

⑨ : Oita = F Abi Zbti - Aai Zaks i--

OF SO ;-1

Oik = F Abi Zbii = F. Abi Zbkt BE Abiszbi

{ SO.

= - Aai Za -5. = - Aai ( Zaktszasi ) K - Aai Zak

⑤ : Miki'= 7 ( Oita ; ask) x y Loi

'

; ask) -127 coikixsklsoika-

fmi"

= qq.is , ask ,

mi"" Smita'

Smita = an ( oik ; ask )SOiEa .

② : Zaks ; = Ya - F Aby' Mj# a t Haim a

--

zka SZa

{ Zak = Ya - jZAbjmj→ka= Ya - F.Abjmjk - F. Abjsmjta

S Zita = Aai mia.= Aai (Mit Smita ) re Aai Mik .

⇒ no::÷÷::÷÷÷÷÷÷*nii*iZak = ya - F Abjmjk t F. Abj 27 ( Oj"

; d 3k ) Zaki

= Ya - F Abj Mj t th F. 27 lojk ; a 3k) Zak ,

⇒ ::÷::::÷:÷÷:n.⇒3kt ' = Skx In Iq, y' ( Oj

"

; ask) t 1

This is a version of the AMP for LASSO