Neural Networks Architecture Baktash Babadi IPM, SCS Fall 2004

Neural Networks Neural Networks ArchitectureArchitecture

Baktash BabadiBaktash Babadi

IPM, SCSIPM, SCS

Fall 2004Fall 2004

The Neuron ModelThe Neuron Model

Architectures (1)Architectures (1) Feed Forward NetworksFeed Forward Networks

The neurons are arranged in The neurons are arranged in separate layersseparate layers

There is no connection between There is no connection between the neurons in the same layerthe neurons in the same layer

The neurons in one layer receive The neurons in one layer receive inputs from the previous layerinputs from the previous layer

The neurons in one layer delivers The neurons in one layer delivers its output to the next layerits output to the next layer

The connections are unidirectionalThe connections are unidirectional (Hierarchical)(Hierarchical)

Architectures (2)Architectures (2)

Recurrent NetworksRecurrent Networks Some connections are Some connections are

present from a layer to present from a layer to the previous layersthe previous layers

Architectures (3)Architectures (3)

Associative networksAssociative networks There is no hierarchical arrangementThere is no hierarchical arrangement The connections can be bidirectionalThe connections can be bidirectional

Why Feed Forward?Why Feed Forward?

Why Recurrent/Associative?Why Recurrent/Associative?

An Example of Associative An Example of Associative Networks: Hopfield NetworkNetworks: Hopfield Network

John Hopfield (1982)John Hopfield (1982) Associative Memory via artificial neural Associative Memory via artificial neural

networksnetworks Solution for optimization problemsSolution for optimization problems Statistical mechanicsStatistical mechanics

Neurons in Hopfield NetworkNeurons in Hopfield Network

The neurons are binary unitsThe neurons are binary units They are either active (1) or passiveThey are either active (1) or passive Alternatively + or –Alternatively + or –

The network contains The network contains NN neurons neurons

The state of the network is described as a The state of the network is described as a vector of 0s and 1s:vector of 0s and 1s:

)1,0,0,...,1,0,1,0(),...,,( 21 NuuuU

The architecture of Hopfield The architecture of Hopfield NetworkNetwork

The network is fully interconnectedThe network is fully interconnected All the neurons are connected to each otherAll the neurons are connected to each other The connections are bidirectional and symmetricThe connections are bidirectional and symmetric

The setting of weights depends on the The setting of weights depends on the applicationapplication

ijji WW ,,

Updating the Hopfield NetworkUpdating the Hopfield Network

The state of the network changes at each time The state of the network changes at each time step. There are four updating modes:step. There are four updating modes: Serial – Random: Serial – Random:

The state of a randomly chosen single neuron will be The state of a randomly chosen single neuron will be updated at each time stepupdated at each time step

Serial-Sequential :Serial-Sequential :The state of a single neuron will be updated at each time The state of a single neuron will be updated at each time step, in a fixed sequencestep, in a fixed sequence

Parallel-Synchronous:Parallel-Synchronous:All the neurons will be updated at each time step All the neurons will be updated at each time step synchronouslysynchronously

Parallel Asynchronous:Parallel Asynchronous:The neurons that are not in refractoriness will be updated at The neurons that are not in refractoriness will be updated at the same timethe same time

The updating Rule (1):The updating Rule (1):

Here we assume that updating is serial-RandomHere we assume that updating is serial-Random

Updating will be continued until a stable state is Updating will be continued until a stable state is reached.reached. Each neuron receives a weighted sum of the inputs Each neuron receives a weighted sum of the inputs

from other neurons:from other neurons:

If the input is positive the state of the neuron will If the input is positive the state of the neuron will be 1, otherwise 0:be 1, otherwise 0:

jh

0 if 0

0 if 1

j

j

j h

hu

ij

N

jii

ij wuh ,1

.

The updating rule (2)The updating rule (2)

Convergence of the Hopfield Convergence of the Hopfield Network (1)Network (1)

Does the network eventually reach a stable Does the network eventually reach a stable state (convergence)?state (convergence)?

To evaluate this a ‘energy’ value will be To evaluate this a ‘energy’ value will be associated to the network: associated to the network:

The system will be converged if the energy is The system will be converged if the energy is minimizedminimized

j

N

jii

jiij uuwE1

, 2

1


Why energy?Why energy? An analogy with spin-glass models of Ferro- An analogy with spin-glass models of Ferro-

magnetism (Ising model):magnetism (Ising model):

The system is stable if the energy is minimized The system is stable if the energy is minimized

01

00

000 0

00

0

1 1111

1 1 1 1

111

1 1

111

1 1 1

j

N

jii

jiij

jj

jjj

N

jii

iijj

j

jiji

ji

uuwE

eE

juhe

juwh

ju

dd

kw

1,

1,

,,

2,

2

1

system theofenergy potential overal The:

unit ofenergy potential The: 2

1

unit upon the exerted field local the:

unit ofspin the:

distance,


Why convergence?Why convergence?ij

N

jii

ij wuh ,1

.

0 if 0

0 if 1

j

j

j h

hu

j

N

jii

jiij uuwE1

, 2

1

j

N

jii

iijj uwu1

, 2

1 j

jjhu 2

1

0 changenot will then 1 and 0 if jjjjjj hhuuuh

0 change will then 0 and 0 if jjjjj huuuh

0 changenot will then 1 and 0 if jjjjj huuuh

0 change will then 1 and 0 if jjjjjj hhuuuh

changenot do values if minimum is 2

1-E

changenot does when maximum is caseeach in

jj

jj

jjj

uhu

uhu


The changes of E with updating:The changes of E with updating:

kkkoldknewkkoldkkj

jjknewkkj

jjoldnew huhuuhuhuhuhuEEE .2

1)(

2

1)

2

1

2

1()

2

1

2

1(

ij

N

jii

ij wuh ,1

.

0 if 0

0 if 1

j

j

j h

hu

j

N

jii

jiij uuwE1

, 2

1j

jjhu 2

1

0.2

1100 and 0 if

0.2

1100 and 0 if

0.2

1100 and 1 if

0.2

1010 and 1 if

kkknewkkoldk

kkknewkkoldk

kkknewkkoldk

kkknewkkoldk

huuuhu

huuuhu

huuuhu

huuuhu

In each case the energy will decrease or remains constant thus the system tends toStabilize.

The Energy Function:The Energy Function:

The energy function is similar to a The energy function is similar to a multidimensional (N) terrainmultidimensional (N) terrain

Global Minimum

Local MinimumLocal Minimum

Hopfield network as a model for Hopfield network as a model for associative memoryassociative memory

Associative memoryAssociative memory Associates different features with eacotherAssociates different features with eacother

Karen Karen greengreen

George George redred

Paul Paul blueblue

Recall with partial cuesRecall with partial cues

Neural Network Model of Neural Network Model of associative memoryassociative memory

Neurons are arranged like a grid:Neurons are arranged like a grid:

Setting the weights Setting the weights

Each pattern can be denoted by a vector of Each pattern can be denoted by a vector of -1s or 1s:-1s or 1s:

If the number of patterns is m then: If the number of patterns is m then:

Hebbian Learning:Hebbian Learning: The neurons that fire together , wire togetherThe neurons that fire together , wire together

),...,,()1,1,1,...,1,1,1,1( 321pN

pppp ssssS

m

p

jpp

iji ssw1

,

Limitations of Hofield associative Limitations of Hofield associative memorymemory

1) The evoked pattern is sometimes not 1) The evoked pattern is sometimes not necessarily the most similar pattern to the necessarily the most similar pattern to the inputinput

2) Some patterns will be recall more than 2) Some patterns will be recall more than othersothers

3) Spurious states: non-original patterns3) Spurious states: non-original patterns

Capacity: Capacity: 0.15 N0.15 N

Hopfield network and the brain (1):Hopfield network and the brain (1):

In the real neuron, synapses are distributed In the real neuron, synapses are distributed along the dendritic tree and their distance along the dendritic tree and their distance change the synaptic weightchange the synaptic weight

In hopfield network there is no dendritic In hopfield network there is no dendritic geometrygeometry

If they are distributed uniformly, the geometry is If they are distributed uniformly, the geometry is not importantnot important

In the brain the Dale principle holds and In the brain the Dale principle holds and the connections are not symmetricthe connections are not symmetric

The hopfield network with assymetric The hopfield network with assymetric weights and dale principle, work properlyweights and dale principle, work properly


The brain is insensitive to noise and local The brain is insensitive to noise and local lesionslesions

Hopfield network can tolerate noise in the Hopfield network can tolerate noise in the input and partial loss of synapsesinput and partial loss of synapses


In brain the neurons are not binary In brain the neurons are not binary devices, they generate continuous values devices, they generate continuous values of firing ratesof firing rates

Hopfield network with sigmoid transfer Hopfield network with sigmoid transfer function is even more powerful than the function is even more powerful than the binary versionbinary version


In the brain most of the neurons are silent In the brain most of the neurons are silent or firing at low rates but in hopfield or firing at low rates but in hopfield network many of the neurons are activenetwork many of the neurons are active

In sparse hopfield network the capacity is In sparse hopfield network the capacity is even moreeven more


In hopfield network updating is serial In hopfield network updating is serial which is far from biological realitywhich is far from biological reality

In parallel updating hopfield network the In parallel updating hopfield network the associative memories can be recalled as associative memories can be recalled as wellwell


When the number of learned patterns in When the number of learned patterns in hopfield network will be overloaded, the hopfield network will be overloaded, the performance of the network will fall performance of the network will fall abruptly for all the stored patterns abruptly for all the stored patterns

But in real brain an overload of memories But in real brain an overload of memories affect only some memories and the rest of affect only some memories and the rest of them will be intactthem will be intactCatastrophic inferenceCatastrophic inference


In hopfield network the usefull information In hopfield network the usefull information appears only when the system is in the appears only when the system is in the stable statestable state

The Brain do not fall in stable states and The Brain do not fall in stable states and remains dynamicremains dynamic


The connectivity in the brain is much less The connectivity in the brain is much less than hopfield networkthan hopfield network

The diluted hopfield network works wellThe diluted hopfield network works well


Documents

Neural Networks Architecture Baktash Babadi IPM, SCS Fall 2004