Integrate and Fire Neural Network

Conceptual Chip DesignSecond draft March 27, 2006

Computational Approaches to Cortical FunctionsThe Banbury Center

Robert Shapiro: Cape Visions and Global 360

Al Davis: School of Computing, University of Utah

Objective

• Explore the possibility of using custom VLSI* chip assembly to aid in simulating large Integrate and Fire Neural Networks– between 105 and 106 neurons.– Current and conductance models

• Possibly exponential models

– Setup and analysis controlled by conventional digital work station.

Very Large Scale Integrated Circuit

Challenge

• Custom chips offer significant speed and capacity benefits but sacrifice flexibility– Can the user community determine now the

modeling requirements: current-based, conductance-based, exponential etc.

– Can we determine accuracy requirements.– Can we determine ???

• Expected lifetime ???

Design Decisions

• Maximize parallelism and circuit homogeneity– parallelism both intra- and inter-chip– Building blocks: two chips (first design)

• System needs to be extensible– # of chips determine the size of the system

• Ideal performance– roughly scale linearly with # of chips

• will be sub linear– higher stage ripple for final neural excitation contribution– longer wires in array will lead to longer axon excitation

delay• This is a digital system for doing IAF simulations, not an

analog chip for simulating neurons.– Speed improved by factor of 103 over conventional digital.

NPU

SPU

Dataflow

synapses contributingto a single neuron

output axon

One NPU and one SPU

NPU

SPU

NPU

SPU

SPU

SPU

100 NPUs and 100 x 100 array of SPUs

Start w

Start w

Compute gex,gin

Compute Vinf, V

Compute spike

Broadcast spike

Done withspike

Done with all spikes

Done with spikes

collector

Start trigger

RefractoryPeriod?

Computation for a single Neuron in NPU

Incrementtime

Time step size??

How many different time constants??

Start w

C W S

A

synapse Wout

w

C

NPU

Computation for a single Synapse in SPU

Transmission delay

From action potential spike to

synapse

Synaptic PlasticityHabituation,

Modulation and Learning

Synapse Column

Start w

Start w

Compute gex,gin

C W S

C W S

C W S

C W S

w

w

w

w

w

w

C W S

C W S

C W S

C W S

w

w

w

w

w

w

C W S

C W S

C W S

C W S

w

w

w

w

w

w

C W S

C W S

C W S

C W S

w

w

w

w

w

w

axon

start

C W S

C W S

C W S

C W S

w

p

w

w

w

w

w

dGV

axon

params

w

Major Missing Pieces

• How should inputs be handled– Membrane Potential of specific neurons set at specific

times???• How should outputs be handled

– Spike trains must be output: each spike in a time step characterized by neuron identifier

– What about spikes from/to other boards or devices?• What software is required on digital workstation

controlling the IAF engine?– Wiring and initialization– Input generation– Output analysis and display

Questions (so far)

• How important is it to make initialization fast?

• What is the range of weight values at the SPU’s– integer, fixed point, floating point??

• What is the range of firing thresholds?– is it potentially different for each neuron?

• And many many more

Candidate for Study

• Model of the Lateral Geniculate Nucleus and Primary Visual Cortex– Much is known about the wiring, allowing

study of neural network dynamics with random connections replaced by anatomical data.

What do you Think ?

• What would this device need to do for it to be useful in your work ?

Acknowledgements

• Larry Abbott– General guidance, direction and references

• Tim Vogels– Simulation specifics, intro to neural network

models, suggestions for this presentation

• Stefano Fusi– Discussions about time step size, synaptic

plasticity, propagation delays

Comparison with Blue Brain

• Objective– IAF simulator, not ‘brain’

• Approach– Design optimized to task, not a general

purpose digital computer

Blue Brain Quotes

• For now, Markram sees the BlueGene architecture as the best tool for modeling the brain. Blue Brain has some 8,000 processors, and by mapping one or two simulated brain neurons to each processor, the computer will become a silicon replica of 10,000 neurons. "Then we'll interconnect them with the rules [in software] that we've worked out about how the brain functions," says Markram.

Synapse Processing Unit • Actions

– initialization• set connected and weight registers

– axon spike• if it sees a spike and is connected then weight is placed in shift register else

act as a shunt• forwards left inputs to right inputs

– intra-SPU add phase• acts as shift register element or shunt• shifts up on shift enabled

• Contents– registers: inhibit weight, excitation weight, connected flag, shunt

• I/O’s– tbd – need to think about programming model and need to know the range of

weight values

Neuron Processing Unit• Actions

– initialization• set parameters

– Start cycle– intra-NPU add phase

• adds up shifted values and places in NPU-total register• shifts up on shift enabled

– inter-NPU communication• Signals completion of synapse summing and recognizes when all neurons are finished with spikes.

– Intra-npu update phase • membrane threshold is updated with new value in the NPU-total register and if the threshold exceeds

the firing value then set spike flag output axon• this involves 2 multiplies• When all neurons finished with spike info, broadcast new spikes• Update time

• Contents– registers: fire threshold, membrane potential, inhibit multiplicand and current sum, excite multiplicand and

current sum,– adder, comparator, multiplier, (table for exponential calculation in conductance model)

• TBD– number of I/O’s– use one multiply per chip and apply to neuron row sequentially vs. parallel multiply per TSU w/ neuron flag

set

Project Time Line

ToDo• Lots to figure out

– how many SU’s will fit within area, power and I/O constraints• circuitry is pretty trivial• balance of parallelism vs. faster adder tradeoff will be more tricky

– vertical forwarding mechanism• synapses will be sparsely connected• take advantage to minimize shift register length

– Figure out done • vertical shift registers will vary in length in each NPU• all NPU’s must be done to move to inter-NPU add phase

– inject done values on unconnected bottom lines– ripple neuron TSU done values

• intra-NPU done will be sped up if early completion can be figured out– maybe extra register on bottom SU to issue a done token for intra-NPU

add– ripple TSU values to emit NPU done signal?

gex

w

eexpFacEx

gex = (gex + w) x e

expFacEx = exp( -dt / tAMPAParam)

expFacIn = exp( -dt / tGABAParam)

Excitatory Synapse Effect

What about NMDA ??

gex

input

gain

VInf = VRest + gain *(gEx - gIn + input + Theta) gin

theta

Vinf

VRest

Membrane Potential Change

gTot = gLeak + gEx + gIn

VInf = ((gLeak* VRest + gEx * EAMPA+ gIn *EGABA + iMag*iExt) / gTot

V

expV

Vinf

V = VInf + (V - Vinf) * expV where expV= exp( -dt / tau)

New Membrane Potential

V = VInf + (V - VInf)*exp((-dt/tau)*gTot);

Since gTot is not a constant, this exponential must be computed: use table lookup to approximate

CClock pulse

Wweight

Ssynapse

wout

Aaxon

Wout = (S==1 and A==1) ? W : 0

CClock pulse

wout

Computation for a single Synapse in SPU

Shift and add

Sources

• Books– The Computational Brain, Churchland and Sejnowski,

1992– Essentials of Neural Science and Behavior, edited by

Kandel, Schwartz and Jessell, 1995– Pulsed Neural Networks, edited by Maass and

Bishop, 1999– Principles of Neural Science, edited by Kandel,

Schwartz and Jessell, 2000– Theoretical Neuroscience, Dayan and Abbott, 2001– Spiking Neuron Models, Gerstner and Kistler, 2002

Sources• Articles

– Pyramidal cell communication within local networks in layer 2/3 of rat neocortex; Holmgren, Harkany, Svennenfors and Zilberter; J Physiol (2003), 551.1, pp. 139–153

– Activity dynamics and propagation of synchronous spiking in locally connected random networks, Mehring, Hehl, Kubo, Diesmann, Aertsen, Biol. Cybern. 88, 395–408 2003.

– Mexican hats and pinwheels in visual cortex; Kang, Shelley, and Sompolinsky; 2848–2853 PNAS March 4, 2003 vol. 100 no. 5

– An egalitarian network model for the emergence of simple and complex cells in visual cortex; Tao, Shelley, McLaughlin, and Shapley; 366–371 PNAS January 6, 2004 vol. 101 no. 1

– Distributed High-Connectivity Network Simulation, A. Morrison, C. Mehring, T. Geisel, A. Aertsen, and M. Diesmann, Neural Computation 17, 1776–1801, 2005.

– Adaptive Exponential Integrate-and-Fire Model as an Effective Description of Neuronal Activity, Romain Brette and Wulfram Gerstner, J Neurophysiol 94: 3637–3642, 2005.

– Neural Network Dynamics, Tim P. Vogels, Kanaka Rajan, and L.F. Abbott, 2005– Signal Propagation and Logic Gating in Networks of Integrate-And-Fire Neurons, Vogels and

Abbott, Journal of Neuroscience 2005.– Geometric and functional organization of cortical circuits; Shepherd, Stepanyants, Bureau.

Chklovskii and Svoboda; June 2005 Nature Neuroscience– Highly Nonrandom Features of Synaptic Connectivity in Local Cortical Circuits; Song,

Sjostrom, Reigl, Nelson and Chklovskii, PLoS Biology March 2005 | Volume 3 | Issue 3 | e68– Excitatory cortical neurons form fine-scale functional networks; Yoshimura, Dantzker &

Callaway; NATURE |VOL 433 | 24 FEBRUARY 2005