Spin Glasses and Information Processingpavithra/pavithranHomepage/Notes_Pavithran_S_Iyer... · Spin Glasses and Information Processing Pavithran S Iyer Guide: Prof. V.V Sreedhar Chennai

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography

Spin Glasses and Information Processing

Pavithran S IyerGuide: Prof. V.V Sreedhar

Chennai Mathematical Institute

April 25, 2011

Pavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing


1 Overview

2 Information TheoryCommunication problemError correcting codesShannon Heartely theorem

3 Disordered spin systemsIntroductionReason for correspondenceSpin glass physics

4 Implications of the correspondenceSK ModelREMConvolution Codes

5 Questions

6 Bibliography



Outline

Work described - papers by N. Sourlas and a book by NishimoriLooking at:

correspondences

Error correcting code ⇔ Spin Hamiltonian

Signal to noise ⇔ J20J2

Maximum likelihood Decoding ⇔ Find a ground stateError probability per bit ⇔ Ground state magnetizationSequence of most probable symbols ⇔ magnetization at T = 1Convolutional Codes ⇔ One dimentional spin glasses

Viterbi decoding ⇔ T = 0 Transfer matrix algorithmBCJR decoding ⇔ T = 1 Transfer matrix algorithmGallager LDPC codes ⇔ Diluted p-spin ferromagnetsTurbo Codes ⇔ Coupled spin chainsZero error threshold ⇔ Phase transition pointBelief propagation algorithm ⇔ Iterative solution of TAP equations



Outline

correspondences

Error correcting code ⇔ Spin Hamiltonian

Signal to noise ⇔ J20J2

Maximum likelihood Decoding ⇔ Find a ground stateError probability per bit ⇔ Ground state magnetizationSequence of most probable symbols ⇔ magnetization at T = 1Convolutional Codes ⇔ One dimentional spin glasses

Viterbi decoding ⇔ T = 0 Transfer matrix algorithmBCJR decoding ⇔ T = 1 Transfer matrix algorithmGallager LDPC codes ⇔ Diluted p-spin ferromagnetsTurbo Codes ⇔ Coupled spin chainsZero error threshold ⇔ Phase transition pointBelief propagation algorithm ⇔ Iterative solution of TAP equations



Communication problem

Communication Problem

−→ usual formulation - message from Alice to Bob

Alice transmits encoded input - (gaussian) channel inflicts error - Bob tries to recoverfrom the errorStatistical formulation: Bob’s perspective - given an output, maximizes his guess ofthe input being correct. Maximizing quantity: P

(J in|Jout

)- called the posterior

probability.Maximum Aposteriori Probability or MAP decoding: compute conditional probabilitiesusing baye’s theorem, assign J in

i = 1 if P (Ji = 1|Jout) > P (−1|Jout) and J ini = −1

otherwise. Maximum information about the (Alice) input which can be transmittedacross the channel to Bob = channel capacity C.Input (signal) power S & Noise (power) = N , then

important quantities

SN

= signal to noise ratio and for a gaussian channel, channel capacity

C =1

2log2

(1 +SN

).





−→ usual formulation - message from Alice to Bob


(J in|Jout






SN


C =1

2log2

(1 +SN

).






(J in|Jout






SN


C =1

2log2

(1 +SN

).





Statistical formulation: Bob’s perspective - given an output, maximizes his guess ofthe input being correct. Maximizing quantity: P

(J in|Jout






SN


C =1

2log2

(1 +SN

).





Statistical formulation: Bob’s perspective - given an output, maximizes his guess ofthe input being correct. Maximizing quantity: P

(J in|Jout






SN


C =1

2log2

(1 +SN

).



Error correcting codes

Error correcting code

Not all encodings can assure recovery from error - only certain codes called errorcorrecting codes.Crux: add redundant bits to input message - majority of bits are unaffected by error -original message can be retrieved.Redundancy is undesirable - slow rate of information transmission.

Rate of transmission

Rate of information =# bits for encoding (ignoring error)

# bits used for encoding with error



















Shannon Heartely theorem

A theorem

Aim: Maximum rate. upperbound ?

Shannon Heartely or Noisy channel coding theorem

The rate of an error correcting code cannot exceeed the channel capacity for noiselesstransmission. R ≤ C.

The aim of every ECC is to go as close to the bound. This bound is a theoreticalmaximum.



Shannon Heartely theorem

A theorem

Aim: Maximum rate. upperbound ?

Shannon Heartely or Noisy channel coding theorem

The rate of an error correcting code cannot exceeed the channel capacity for noiselesstransmission. R ≤ C.

The aim of every ECC is to go as close to the bound. This bound is a theoreticalmaximum.



Introduction

A disordered spin system

Ising model H = −J∑

i<j SiSj . For any choice of J - exactly 2 ground states.Bad information storage structures - can only store two units of information. Needplenty of ground states.Spin glass - lot of equilibrium states.

Naively: put Jij = SiSj . J - local to a pair of sites - link variable:

{+1 : Ferro

−1 : Antiferro

Finding ground states of a simple spin glass is difficult[1] - main reason - link variablesare random - frustration causing many degenerate ground states.Note: would not happen if all Jij = some constant.



Introduction





{+1 : Ferro

−1 : Antiferro




Introduction





{+1 : Ferro

−1 : Antiferro




Introduction





{+1 : Ferro

−1 : Antiferro




Introduction





{+1 : Ferro

−1 : Antiferro




Reason for correspondence

Interesting correspondance

Going back - maximizing posterior probability P(J in|Jout) - using baye’s theoremrelates it to P(Jout |J in).After some algebra, we find: P(J ij |Jout) = exp

(u∑

n Mnk1...kn

J ink1. . . J in

kn−∑

i hiJini

)→

exponent strikingly similar to the hamiltonian of a spin glass - minimize this ⇒ find aground state of the spin glasswe have a spin glass where

{J ini

}play the role of spins and Mk1...kn contain link

variables for n-spin interactions.Instead of finding ground states, demontrating encoding & decoding, we look at moresimilarities between SG physics and ECC.All the above are only for ising spins.






(u∑

n Mnk1...kn

J ink1. . . J in

kn−∑

i hiJini

)→


{J ini








(u∑

n Mnk1...kn

J ink1. . . J in

kn−∑

i hiJini

)→


{J ini








(u∑

n Mnk1...kn

J ink1. . . J in

kn−∑

i hiJini

)→


{J ini








(u∑

n Mnk1...kn

J ink1. . . J in

kn−∑

i hiJini

)→


{J ini





Spin glass physics

Physics of spin glass

Spins on a lattice - total energy invariant under local transformations

Simple example: SK model



Spin glass physics


Spins on a lattice - total energy invariant under local transformations

Phases: Paramagnetic, Ferromagnetic,Spin glass

Order Parameters:

Phase m q

Ferro > 0 > 0SG 0 > 0

Para 0 0

In SG phase, m = 0 - random alignmentbut q 6= 0 - key difference fromparamagnetic phase. Over large time, spinsat two lattice sites will be correlated[1].




Spin glass physics



Order Parameters:

Phase m q

Ferro > 0 > 0SG 0 > 0

Para 0 0





Spin glass physics



Order Parameters:

Phase m q

Ferro > 0 > 0SG 0 > 0

Para 0 0


Information cannot be encoded in paramagnetic phase - high random fluctuationsdestroy spin-spin correlation.




Spin glass physics


Aim: Look at SG phase and overlap.Phase transitions in a disordered spin system - using Ginsburg Landau theory -expanding the free energy about critical points (small order parameter).




Spin glass physics


Aim: Look at SG phase and overlap.


H =∑

i<j JijSiSj . Distribution of links, Jij - given by

P(Jij) =

√N

2πJ2exp

{− N

2J2

(Jij −

J0N

)2}



Spin glass physics



H =∑

i<j JijSiSj . Distribution of links, Jij - given by

P(Jij) =

√N

2πJ2exp

{− N

2J2

(Jij −

J0N

)2}

Free energy: every sample - one realization of disorder - n replicas of the system -average over all values of Jij - gaussian distribution



Spin glass physics



After some algebra - free energy ≡ f (m, {qαβ},T , J, J0). Equations of state -determine value for order parameters in terms of J, J0,T .



Spin glass physics



Infinite range model - H =∑

i1<i2<···<irSi1Si2 · · · Sin .

Similar calculations as SK model yield free energy & existence of spin glass phase atfull RSB.Aim is to demonstrate shannon heartely theorem - take a system for which 1RSB issufficient - calculation convenience.



Spin glass physics



REM

r →∞ limit of Infinite range model - probability of a state only depends on energy &independent distribution of energy states1 RSB is enough → exact calculations confirm with 1RSB results

Magnetic phases:Phases Condition

P ↔ SG Tc =J

2√

ln 2SG ↔ F j0 = J

√ln 2

P ↔ F j0 =J2

4T+ T ln 2

Calculations for overlap - M=1 inferromagnetic phase -j0/J >

√ln 2. Hence, error free

decoding.



Implications of this correspondence

Transistion from binary to ±1 since spin & link variables take ±1 values. Symbolu → (−1)u.raw input ⇔ spin orientations, code symbols ⇔ link variables in ground statesignal amplitude ⇔ J0 and noise amplitude ⇔ J.word MAP decoding ⇔ finding ground state.error probability per bit ⇔ ground state magnetization.sequence of most probable bits ⇔ magnetization at T = 1.Error free decoding ⇔ overlap: M = 1Hence we have some correspondances.




Transistion from binary to ±1 since spin & link variables take ±1 values. Symbolu → (−1)u.Information processing using Spin glass → raw input ⇔ spin orientations, encoded intocode symbols ⇔ link variables in ground state by introducing interactions.signal amplitude ⇔ J0 and noise amplitude ⇔ J.word MAP decoding ⇔ finding ground state.error probability per bit ⇔ ground state magnetization.sequence of most probable bits ⇔ magnetization at T = 1.Error free decoding ⇔ overlap: M = 1Hence we have some correspondances.




raw input ⇔ spin orientations, code symbols ⇔ link variables in ground state

signal amplitude ⇔ J0 and error operationTemp−−−→rise ?

gaussian disorder - variance of J2 -

noise amplitude ⇔ J.word MAP decoding ⇔ finding ground state.error probability per bit ⇔ ground state magnetization.sequence of most probable bits ⇔ magnetization at T = 1.Error free decoding ⇔ overlap: M = 1Hence we have some correspondances.




raw input ⇔ spin orientations, code symbols ⇔ link variables in ground statesignal amplitude ⇔ J0 and noise amplitude ⇔ J.State of spin glass corresponding to output has variance J2

0 + J2 in link variables.word MAP decoding ⇔ finding ground state.error probability per bit ⇔ ground state magnetization.sequence of most probable bits ⇔ magnetization at T = 1.Error free decoding ⇔ overlap: M = 1Hence we have some correspondances.




raw input ⇔ spin orientations, code symbols ⇔ link variables in ground statesignal amplitude ⇔ J0 and noise amplitude ⇔ J.Decoding ⇒ finding an assignment for J in which maximizes P

(J in|Jout

)- same

assigmnet would minimize − ln P(J in|Jout

)≡ H → ground state

word MAP decoding ⇔ finding ground state.error probability per bit ⇔ ground state magnetization.sequence of most probable bits ⇔ magnetization at T = 1.Error free decoding ⇔ overlap: M = 1Hence we have some correspondances.




raw input ⇔ spin orientations, code symbols ⇔ link variables in ground statesignal amplitude ⇔ J0 and noise amplitude ⇔ J.word MAP decoding ⇔ finding ground state.error probability per bit ⇔ ground state magnetization.sequence of most probable bits ⇔ magnetization at T = 1.Error free decoding ⇔ overlap: M = 1Hence we have some correspondances.




raw input ⇔ spin orientations, code symbols ⇔ link variables in ground statesignal amplitude ⇔ J0 and noise amplitude ⇔ J.word MAP decoding ⇔ finding ground state.error probability per bit ⇔ ground state magnetization.Average value of a bit at index i :

〈τi 〉P =

∑i τiP (τi |Jout)∑i P (τi |Jout)

−−−−−−−−−−−−−−−→P(τi |Jout)=e−H[Jout ,{τi}]

∑i τie

−H[Jout ,{τi}]∑i e−H[Jout ,{τi}]

. Putting ln P ∼ Z ,

we see 〈τi 〉P = magnetization with β = 1.sequence of most probable bits ⇔ magnetization at T = 1.Error free decoding ⇔ overlap: M = 1Hence we have some correspondances.




raw input ⇔ spin orientations, code symbols ⇔ link variables in ground statesignal amplitude ⇔ J0 and noise amplitude ⇔ J.word MAP decoding ⇔ finding ground state.error probability per bit ⇔ ground state magnetization.sequence of most probable bits ⇔ magnetization at T = 1.Measuring decoding performance: overlap: overlap of original & decoded message M.Suppose original bit: ξi & MAP decoded bit: ξ̂i = sign〈σi 〉, then:

M = Trξ∑

J P(J)ξsign〈σi 〉 ⇔ Hamming Distance =

{1 both are same

−1 when inverted.

Error free decoding ⇔ overlap: M = 1Hence we have some correspondances.




raw input ⇔ spin orientations, code symbols ⇔ link variables in ground statesignal amplitude ⇔ J0 and noise amplitude ⇔ J.word MAP decoding ⇔ finding ground state.error probability per bit ⇔ ground state magnetization.sequence of most probable bits ⇔ magnetization at T = 1.Error free decoding ⇔ overlap: M = 1Hence we have some correspondances.



SK Model

SK Model

For SK Model:

Phase diagram - for the SK model.



REM

Random Energy Model

Error free decoding in ferromagnetic phasefor T =. Rate: Totally N sites used forencoding and

(Nr

)possible sites. In the

r →∞ limit[?]: R =r !

N r−1 .

Channel Capacity: for a gaussian

channel[?] C −−−→r→∞

j20 r !

2J2N r−1 ln 2. where

we have used[?] J0 = signal amplitude, J= noise amplitude.

Shannon Heartely bound is satisfied for this encoding - interesting result !



REM

Random Energy Model

In the r →∞ limit[?]: R =r !

N r−1 .



j20 r !






REM

Random Energy Model

In the r →∞ limit[?]: R =r !

N r−1 .



j20 r !






Convolution Codes

Convolution Codes

A convolution code (CC) is an ECC encoding m bit string to n-bit string with R =m

n.

Consider: CC with R =1

2, encoding:

x1i = ui + ui−1,

x2i = ui + ui−1 + ui−2

, where ui - raw input and

xi - encoded symbol1.Corresponding spin glass: → Ground state - raw input given by spins Si & encoded

bits using link variables given by J(1)i ,i−2 = SiSi−2, J

(2)i ,i−1,i−2 = SiSi−1Si−2 and

hamiltonian[?]: H = −∑

i

(J(1)i ,i−2SkSk−2 + J

(2)i ,i−1,i−2SkSk−1Sk−2

).

Convolution Codes correspond to 1D spin glasses

1All in binaryPavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing


Convolution Codes

Convolution Codes


n.


2, encoding:

x1i = ui + ui−1,

x2i = ui + ui−1 + ui−2






i

(J(1)i ,i−2SkSk−2 + J

(2)i ,i−1,i−2SkSk−1Sk−2

).




Convolution Codes

Convolution Codes


n.


2, encoding:

x1i = ui + ui−1,

x2i = ui + ui−1 + ui−2






i

(J(1)i ,i−2SkSk−2 + J

(2)i ,i−1,i−2SkSk−1Sk−2

).




Convolution Codes

Convolution Codes


n.


2, encoding:

x1i = ui + ui−1,

x2i = ui + ui−1 + ui−2






i

(J(1)i ,i−2SkSk−2 + J

(2)i ,i−1,i−2SkSk−1Sk−2

).




some (open) questions

• Measuring overlap for convolution codes - error free decoding

• r ≥ 3 models - finite range spin-spin interactions - numerical results

• finite size effects



References I

R.B. Ash, Information theory, Dover books on mathematics, Dover Publications,1990.

Tommaso Castellani and Andrea Cavagna, Spin-glass theory for pedestrians,Journal of Statistical Mechanics: Theory and Experiment 2005 (2005), no. 05,P05012.

S F Edwards and P W Anderson, Theory of spin glasses, Journal of Physics F:Metal Physics 5 (1975), no. 5, 965.

Francesco Zamponi, Mean field theory of spin glasses, arxiv cond-mat:1008.4844v1, ( 28 Aug, 2010), no. 5, 965.

M. Mezard, G. Parisi, and M.A. Virasoro, Spin glass theory and beyond, WorldScientific lecture notes in physics, World Scientific, 1987.



References II

Ralf .R. Muller, The replica method, Lecture Notes for TT8107: Random MatrixTheory for Wireless Communications.

H. Nishimori, Statistical physics of spin glasses and information processing: anintroduction, International series of monographs on physics, Oxford UniversityPress, 2001.

Giorgio Parisi, On spin glass theory, Physica Scripta 1987 (1987), no. T19A, 27.

Nicolas Sourlas, Statistical mechanics and capacity-approaching error-correctingcodes, Physica A: Statistical Mechanics and its Applications 302 (2001), no. 1-4,14 – 21.



My sincere thanks to

Prof. V.V. Sreedhar for helping me through the long calculations.

Nana Siddharth for taking time to help me with mathematica.

... and my friends for thier help and support.


Documents

Spin Glasses and Information Processingpavithra/pavithranHomepage/Notes_Pavithran_S_Iyer... · Spin Glasses and Information Processing Pavithran S Iyer Guide: Prof. V.V Sreedhar Chennai