09.07.2015 1

Content

• Motivation

• Principles

Hopfield model

Magnetic analogy

Patterns in the Hopfield model

Pattern retrieval

Memory capacity

• Improvement

- Low activity patterns

- Network with spiking neurons

09.07.2015 2

http://imgkid.com/3d-neuron-model.shtml

Motivation

09.07.2015 3http://neuronaldynamics.epfl.ch/online/

Motivation

How can a neuronal network store and retrieve patterns?

09.07.2015 4

Hopfield model

• Network of N binary neurons with the state of neuron 𝑖𝑆𝑖 𝑡 = ±1

• Each neuron is connected to each other

• Influence of states 𝑗 on neuron 𝑖 is given by the

input potential

ℎ𝑖 𝑡 =

𝑗

𝑤𝑖𝑗𝑆𝑗(𝑡)

• 𝑤𝑖𝑗 is the weight neuron 𝑖 “feels” the state 𝑆𝑗 of the

other neurons 𝑗

• 𝑆𝑖(𝑡) changes by time and is dependent on ℎ𝑖 𝑡Prob 𝑆𝑖 𝑡 + ∆𝑡 = +1 ℎ𝑖 𝑡 = 𝑔[ℎ𝑖 𝑡 ]

09.07.2015 5

http://imgkid.com/3d-neuron-model.shtmlhttp://brainmagazine.co.uk/%E2%80%98terry-pratchett-choosing-to-die’-series-part-2-motor-neurone-disease/

Hopfield model

Prob 𝑆𝑖 𝑡 + ∆𝑡 = +1 ℎ𝑖 𝑡 = 𝑔(ℎ𝑖 𝑡 )

• 𝑔 is a gain function, e.g.𝑔 ℎ = 0.5 [ 1 + tanh 𝛽ℎ ]

• 𝛽 can be a measure for fluctuation

Finite 𝛽 => stochastic dynamics:

0 < 𝑔 ℎ < 1

𝛽 → ∞ => deterministic dynamics:

𝑔 ℎ = 1 ℎ > 0𝑔 ℎ = 0 ℎ < 0

update rule: 𝑆𝑖 𝑡 + ∆𝑡 = sgn[ℎ𝑖 𝑡 ]

09.07.2015 6

Magnetic analogy

Ising model

• Atoms with 1d spins 𝑆𝑖 = ±1

• Spins interact with interaction ℎ𝑖 = 𝑖,𝑗 𝑤0𝑆𝑗(𝑡)

𝑖, 𝑗 nearest neighbours

𝑤0 same weight for every atom

• Probability of spin flipProb 𝑆𝑖 𝑡 + ∆𝑡 = +1 ℎ𝑖 𝑡 = 𝑔[ℎ𝑖 𝑡 ]

• 𝑔 can be obtained by the Boltzmann-distribution

𝛽 = (𝑘𝐵𝑇)−1 spins are thermally fluctuating

• Non-fluctuating alignment at low temperatures

09.07.2015 7

http://neuronaldynamics.epfl.ch/online/Ch17.S2.html

Magnetic analogy

Ising model

• Tow types of atoms A and B

• Anti - ferromagnet if spins of A and B are oppositely aligned

𝑤𝑖𝑗 = +1 if 𝑖 and 𝑗 both belong to type A ore B

𝑤𝑖𝑗 = −1 if 𝑖 and 𝑗 belong to different types

• Same idea is used for the

Hopfield model

• Weights for a stored pattern with active and inactive neurons𝑤𝑖𝑗 = 𝑝𝑖𝑝𝑗

09.07.2015 8

http://neuronaldynamics.epfl.ch/online/Ch17.S2.html

Patterns in the Hopfield model

• A pattern 𝜇 is a desired configuration of neuron activity

𝑝𝑖𝜇= ±1; 1 ≤ 𝑖 ≤ 𝑁 𝑝𝑖

𝜇= 0

• A neuronal network represents a pattern 𝜇 if

𝑆𝑖 𝑡 = 𝑆𝑖 𝑡 + ∆𝑡 = 𝑝𝑖𝜇

∀𝑖

Patterns are fixed points under the dynamicsProb 𝑆𝑖 𝑡 + ∆𝑡 = +1 ℎ𝑖 𝑡 = 𝑔[ℎ𝑖 𝑡 ]

With the input potential ℎ𝑖 𝑡 = 𝑗𝑤𝑖𝑗𝑆𝑗(𝑡)

And the weights

𝑤𝑖𝑗 =1

𝑁

𝜇=1

𝑀

𝑝𝑖𝜇𝑝𝑗𝜇

09.07.2015 9

http://neuronaldynamics.epfl.ch/online/Ch17.S2.html

Pattern retrieval

• Mimic noisy image by initialize the network𝑆 𝑡0 = {𝑆𝑖 𝑡0 ; 1 ≤ 𝑖 ≤ 𝑁}

• Network evolves freely under the dynamicsProb 𝑆𝑖 𝑡 + ∆𝑡 = +1 ℎ𝑖 𝑡 = 𝑔[ℎ𝑖 𝑡 ]

• 𝑆 𝑡𝑜 should converge to most similar pattern 𝜇

• Similarity is measured by the overlap

𝑚𝜇 𝑡 =1

𝑁

𝑖

𝑝𝑖𝜇𝑆𝑖(𝑡)

𝑚𝜇 = 10−1

09.07.2015 10

, 𝑆𝑖 𝑡 = 𝑝𝑖𝜇

,no correlation

, 𝑆𝑖 𝑡 = −𝑝𝑖𝜇

http://neuronaldynamics.epfl.ch/online/

Pattern retrieval

• The overlap is a macroscopic state variable

𝑚𝜇 𝑡 =1

𝑁

𝑖

𝑝𝑖𝜇𝑆𝑖(𝑡)

• Input potential

𝑤𝑖𝑗 =1

𝑁 𝜇=1𝑀 𝑝𝑖

𝜇𝑝𝑗𝜇

• With this input potential the probabilistic update

Prob 𝑆𝑖 𝑡 + ∆𝑡 = +1|ℎ𝑖(𝑡) = 𝑔

𝜇=1

𝑀

𝑝𝑖𝜇𝑚𝜇(𝑡)

is completely determined by the macroscopic overlap

09.07.2015 11

ℎ𝑖 𝑡 =

𝑗

𝑤𝑖𝑗𝑆𝑗(𝑡) =1

𝑁

𝑗=1

𝑁

𝜇=1

𝑀

𝑝𝑖𝜇𝑝𝑗𝜇𝑆𝑗(𝑡) =

𝜇=1

𝑀

𝑝𝑖𝜇𝑚𝜇(𝑡)

http://neuronaldynamics.epfl.ch/online/

Memory capacity

• How many patterns can be stored?

• A stored pattern is a fixed point under the dynamics

• Therefore the initial state 𝑆𝑗 𝑡0 = 𝑝𝑗𝜈 should stay the same

• For 𝛽 → ∞ the dynamics is

𝑆𝑖 𝑡0 + ∆𝑡 = sgn ℎ𝑖(𝑡)

ℎ𝑖 𝑡 = 𝑗𝑤𝑖𝑗𝑆𝑗(𝑡) input potential

𝑤𝑖𝑗 =1

𝑁 𝜇=1𝑀 𝑝𝑖

𝜇𝑝𝑗𝜇

weight

𝑆𝑖 𝑡0 + ∆𝑡 = 𝑝𝑖𝜈sgn 1 − 𝑎𝑖𝜈

with 𝑎𝑖𝜈 =1

𝑁 𝑗 𝜇≠𝜈 𝑝𝑖

𝜇𝑝𝑖𝜈𝑝𝑗

𝜇𝑝𝑗𝜈

• When does the fixed point exist?

09.07.2015 12

Memory capacity

• 𝑆𝑖 𝑡0 + ∆𝑡 = 𝑝𝑖𝜈sgn 1 − 𝑎𝑖𝜈

• Fixed point exists for 1 > 𝑎𝑖𝜈 =1

𝑁 𝑗 𝜇≠𝜈 𝑝𝑖

𝜇𝑝𝑖𝜈𝑝𝑗

𝜇𝑝𝑗𝜈

• Probability of flipping the sign of 𝑆𝑖 𝑡0 + ∆𝑡

probability of finding 𝑎𝑖𝜈 > 1

• 𝑝𝑖𝜇𝑝𝑖𝜈𝑝𝑗

𝜇𝑝𝑗𝜈 = ±1 and 𝑝𝑖

𝜇are independent

=> 𝑎𝑖𝜈 is a random walk of 𝑁(𝑀 − 1) steps

• For large 𝑁

=> Gaussian distributed walking distance 𝑥

𝑃 =1

2𝜋𝜎 exp

−𝑥2

2𝜎2d𝑥

09.07.2015 13

http://www.4dsolutions.net/ocn/numeracy3.htmlhttp://zoonek2.free.fr/UNIX/48_R/07.html

Memory capacity

• Probability of finding 𝑎𝑖𝜈 > 1 is given by

𝑃𝑒𝑟𝑟𝑜𝑟 =1

2𝜋𝜎 1

∞

exp−𝑥2

2𝜎2d𝑥

• standard deviation 𝜎 ≈ 𝑀/𝑁 for 𝑀 ≫ 1

Probability of an erroneous state flip increases with 𝑀/𝑁

• Storage capacity 𝐶𝑠𝑡𝑜𝑟𝑒 =𝑀𝑚𝑎𝑥

𝑁=

𝑀𝑚𝑎𝑥𝑁

𝑁2

Number of bits 𝑀𝑚𝑎𝑥𝑁 per 𝑁2 fully connected neurons

• What error 𝑃𝑒𝑟𝑟𝑜𝑟 one is willing to accept?

09.07.2015 14

http://matheguru.com/stochastik/31-normalverteilung.html

http://matheguru.com/stochastik/31-normalverteilung.html

Memory capacity

• e.g.: 𝑃𝑒𝑟𝑟𝑜𝑟 = 0.001 => 𝐶𝑠𝑡𝑜𝑟𝑒 = 0.105

10 000 neurons store 1050 patterns

10 erroneous neurons per pattern

• Whole discussion of 𝑃𝑒𝑟𝑟𝑜𝑟 for each iteration step

• One erroneously flipped state can cause others to flip

• Theoretical physics predicts fixed points for ongoing iterations till

𝐶𝑠𝑡𝑜𝑟𝑒 = 0.138

09.07.2015 15

Low activity patterns

• Until now, the neuron activity in a pattern was 50 %

𝑝𝑖𝜇= 0

• Patterns with a lower level of activity shall be possible

𝑤𝑖𝑗 =1

2𝑎 1 − 𝑎 𝑁

𝜇=1

𝑀

𝜉𝑖𝜇− 𝑏 𝜉𝑗

𝜇− 𝑎

With 𝜉𝑖𝜇∈ 0,1 and 𝜉𝑖

𝜇= 𝑎

and a constant 0 ≤ 𝑏 ≤ 1

• This means 𝑎 ∙ 𝑁 active neurons and 1 − 𝑎 𝑁 inactive neurons

• Pattern retrieval works analogously

• How to implement spiking neurons?09.07.2015 16

http://neuronaldynamics.epfl.ch/online/https://en.wikipedia.org/wiki/Action_potential

Network with spiking neurons

More biological neuron description by the

membrane potential

𝑢𝑖 𝑡 =

𝑓

𝜂 𝑡 − 𝑡𝑖𝑓+ ℎ𝑖 𝑡 + 𝑢𝑟𝑒𝑠𝑡

𝜂 Spike after-potential

𝑡𝑖𝑓

past firing times of neuron 𝑖

ℎ𝑖 input potential

ℎ𝑖 𝑡 =

𝑗

𝑤𝑖𝑗 0

∞

휀 𝑠 𝑆𝑗 𝑡 − 𝑠 d𝑠

With the spike train 𝑆𝑗 𝑡 = 𝑓 𝛿(𝑡 − 𝑡𝑗𝑓)

09.07.2015 17

http://neuronaldynamics.epfl.ch/online/

Network with spiking neurons

• The new macroscopic overlap

𝑚𝜇 𝑡 =1

2𝑎 1 − 𝑎 𝑁

𝑗

𝜉𝑗𝜇− 𝑎 𝑆𝑗(𝑡)

describes the input potential

ℎ𝑖 𝑡 =

𝜇=1

𝑀

𝜉𝑖𝜇− 𝑏

0

∞

휀 𝑠 𝑚𝜇 𝑡 − 𝑠 d𝑠

• In the case just one overlap 𝑚𝜈 is significant the network splits up into two populations

Active neurons in pattern 𝜈 𝜉𝑖𝜈 = +1

Inactive neurons 𝜉𝑖𝜈 = 0

• Network can be analysed by population dynamics

09.07.2015 18

Summary

• Hopfield model consist of N binary neurons

𝑆𝑖 𝑡 = ±1 𝑤𝑖𝑗 =1

𝑁 𝜇=1𝑀 𝑝𝑖

𝜇𝑝𝑗𝜇

• A stored pattern is a fixed point under the dynamics

Prob 𝑆𝑖 𝑡 + ∆𝑡 = +1 ℎ𝑖 𝑡 = 𝑔[ℎ𝑖 𝑡 ]

• Dynamics is completely determined by the macroscopic overlap

𝑚𝜇 𝑡 =1

𝑁

𝑖

𝑝𝑖𝜇𝑆𝑖(𝑡)

09.07.2015 19

Reference

• Wulfram Gerstner ed al.: Neuronal Dynamics - From Single Neurons to Networks and Models of Cognition. Cambridge University Press 2014

09.07.2015 20

Thank you for your attention