VLSI Implementation of Artificial Neural Network

Dr. S. L. Pinjare

Human Brain

The human brain is the center of the human nervous system.

The human brain has been estimated to contain 80 or 90 billion (~86 109) neurons. These neurons pass signals to each other via as many as 1000 trillion (1015, 1 quadrillion) synaptic connections.

The Neuron

04/10/23 3

Basic Components of Biological Neurons

1. Neurons encode their outputs as a series of brief electrical pulses

2. The neuron’s cell body (soma) processes the incoming activations and converts them into output activations.

3. Dendrites are fibres which originate from the cell body and provide the receptive zones that receive activation from other neurons.

4. Axons are fibres acting as transmission lines that send activation to other neurons.

5. The junctions that allow signal transmission between the axons and dendrites are called synapses. 6. The process of transmission is by diffusion of chemicals called neurotransmitters across the synaptic cleft.

Artificial Neuron

• A vastly simplified model of real neurons

• 1. A set of synapses (i.e. connections) brings in activations from other neurons.

• 2. A processing unit sums the inputs, and then applies a non-linear activation function

• (i.e. squashing/transfer/threshold function).

• 3. An output line transmits the result to other neurons.

Artificial Neural Network

• Artificial neural systems can be considered as simplified mathematical models of brain-like systems and they function as parallel distributed computing networks.

• However, in contrast to conventional computers, which are programmed to perform specific task, most neural networks must be taught, or trained.

• They can learn new associations, new functional dependencies and new patterns.

04/10/23 6

Why are Artificial Neural Networks worth studying?

1. They are extremely powerful computational devices 2. Massive parallelism makes them very efficient.3. They can learn and generalize from training data – so

there is no need for enormous feats of programming.4. They are particularly fault tolerant – this is equivalent to

the “graceful degradation” found in biological systems.5. They are very noise tolerant – so they can cope with

situations where normal symbolic systems would have difficulty.

6. In principle, they can do anything a symbolic/logic system can do, and more. (In practice, getting them to do it can be rather difficult…)

04/10/23 7

What are Artificial Neural Networks used for?

• As with the field of AI in general, there are two basic goals for neural network research:

– Brain modelling : The scientific goal of building models of how real brains work. This can potentially help us understand the nature of human intelligence, formulate

– better teaching strategies, or better remedial actions for brain damaged patients.– Artificial System Building : The engineering goal of building efficient systems

for– real world applications. This may make machines more powerful, relieve humans– of tedious tasks, and may even improve upon human performance.– These should not be thought of as competing goals. We often use exactly the

same– networks and techniques for both. Frequently progress is made when the two

approaches– are allowed to feed into each other. There are fundamental differences though,

e.g. the– need for biological plausibility in brain modelling, and the need for computational– efficiency in artificial system building.

04/10/23 8

Artificial Neural Network

• There are two important aspects of the network’s operation to consider:– Learning The network must learn decision surfaces

from a set of training patterns so that these training patterns are classified correctly.

– Generalization After training, the network must also be able to generalize, i.e.correctly classify test patterns it has never seen before.

• Usually we want our neural networks to learn well, and also to generalize well.

04/10/23 9

ANN Architectures/ Structures/ Topologies

• Mathematically, ANNs can be represented as weighted directed graphs.

• For our purposes, we can simply think in terms of activation flowing between processing units via one-way connections.

04/10/23 10

ANN architectures

• Three common ANN architectures are:– Single-Layer Feed-forward NNs:

• One input layer and one output layer of processing units. No feed-back connections.

– Multi-Layer Feed-forward NNs:• One input layer, one output layer, and one or more hidden layers of

processing units. No feed-back connections. The hidden layers sit in between the input and output layers, and are thus hidden from the outside world.

– Recurrent NNs:• Any network with at least one feed-back connection. It may, or may

not, have hidden units.

04/10/23 11

Single LayerFeed-forward

Multi-LayerFeed-forward

RecurrentNetwork

04/10/23 12

Mathematical description

Sum Squared Error Function

Gradient descent weight update equation

Basic neuron equation

Sigmoid activation function

04/10/23 13

Learning by Error Minimisation

• The Perceptron Learning Rule is an algorithm for adjusting the network weights wij to minimise the difference between the actual outputs outj and the desired outputs targj.

• Sum Squared Error function:

04/10/23 14

• Suppose we have a function f(x) and we want to change the value of x to minimise f(x).

• What we need to do depends on the gradient of f(x). • There are three cases to consider:

– If df(x)/dx > 0 then f(x) increases as x increases so we should decrease x

– If df(x)/dx < 0 then f(x) decreases as x increases so we should increase x

– If df(x)/dx = 0 then f(x) is at a maximum or minimum so we should not change x

04/10/23 15

Gradient descent minimisation

• In summary, we can decrease f(x) by changing x by the amount:

• where n is a small positive constant specifying how much we change x by, and the derivative df/dx tells us which direction to go in. If we repeatedly use this equation, f(x) will (assuming h is sufficiently small) keep descending towards its minimum,

04/10/23 16

Neural Network Learning

• The aim of learning is to minimise this error by adjusting the weights wij.

• Typically we make a series of small adjustments to the weights wij ® wij + Dwij until the error E(wij) is ‘small enough’.

• A systematic procedure for doing this requires the knowledge of how the error E(wij) varies as we change the weights wij, i.e. the gradient of E with respect to wij.

04/10/23 17

Artificial Neural NetworkHistorical Perspective

• The study of brain-style computation has its roots over 50 years ago in the work of McCulloch and Pitts (1943) and slightly later in Hebb’s famous Organization of Behavior (1949).

• However, the 1980s showed a rebirth in interest in neural computing:

04/10/23 18

• 1982 Hopfield provided the mathematical foundation for understanding the dynamics of an important class of networks.

• 1984 Kohonen developed unsupervised learning networks for feature mapping into regular arrays of neurons.

• 1986 Rumelhart and McClelland introduced the backpropagation learning algorithm for complex, multilayer networks.

04/10/23 19

• Beginning in 1986-87, many neural networks research programs were initiated.

• The list of applications that can be solved by neural networks has expanded from small test-size examples to large practical tasks.

• Very-large-scale integrated neural network chips have been fabricated.

• In the long term, we could expect that artificial neural systems will be used in applications involving vision, speech, decision making, and reasoning, butalso as signal processors such as filters, detectors, and quality control systems.

04/10/23 20

Artificial neural systems

• Artificial neural systems, or neural networks, are physical cellular systems which can acquire, store, and utilize experiental knowledge.

• The knowledge is in the form of stable states or mappings embedded in networks that can be recalled in response to the presentation of cues.

04/10/23 21

04/10/23 22

• The basic processing elements of neural networks are called artificial neurons, or simply neurons or nodes.

• Each processing unit is characterized by an activity level (representing the state of polarization of a neuron), an output value (representing the firing rate of the neuron), a set of input connections, (representing synapses on the cell and its dendrite), a bias value (representing an internal resting level of the neuron), and a set of output connections (representing a neuron’s axonal projections).

• Each of these aspects of the unit are represented mathematically by real numbers.

• Thus, each connection has an associated weight (synaptic strength) which determines the effect of the incoming input on the activation level of the unit.

• The weights may be positive (excitatory) or negative (inhibitory).

04/10/23 23

• The general procedure is to have the network learn the appropriate weights from a representative set of training data.

• In all but the simplest cases, however, direct computation of the weights is intractable.

• Instead, we usually start off with random initial weights and adjust them in small steps until the required outputs are produced.

04/10/23 24

Hardware Implementation

• Most artificial neural network models have been implemented in software, but the size and complexity of many problems has quickly exceeded the power of conventional computer hardware.

• It is the goal of neural network engineers to transfer the progress made into new hardware systems.

• These are intended to accelerate future developments of algorithms and architectures, and to make possible the use of dedicated neural networks in industrial applications.

04/10/23 25

24 April 2011 26