Upload
janice-dixon
View
226
Download
3
Embed Size (px)
Citation preview
Intro to Neural Intro to Neural NetworksNetworks
UNIT-IIUNIT-II
Neural networks is a branch of "Artificial Neural networks is a branch of "Artificial Intelligence“. Intelligence“.
““Artificial Neural Network” Artificial Neural Network” is a system is a system loosely modeled based on the human loosely modeled based on the human brain. brain.
The field goes by many names, such as The field goes by many names, such as connectionism, parallel distributed connectionism, parallel distributed processing, neuro-computing, natural processing, neuro-computing, natural intelligent systems, machine learning intelligent systems, machine learning algorithms, and artificial neural networks. algorithms, and artificial neural networks.
INTRODUCTIONINTRODUCTIONWhat is a neural network (NN)?What is a neural network (NN)?
DefinitionDefinition• Neural networks are simplified Neural networks are simplified models of the biological nervous models of the biological nervous system and therefore they have system and therefore they have drawn their motivation from the drawn their motivation from the kindkind of computing performed by a of computing performed by a human brain.human brain.• Neural networks are a powerful technique to Neural networks are a powerful technique to solve many real world problems. solve many real world problems. • They have the ability to learn from They have the ability to learn from experience experience in order to improve their performance in order to improve their performance and to adapt themselves to changes in the and to adapt themselves to changes in the environment. environment.
• In addition to that they are In addition to that they are able to deal with able to deal with incomplete incomplete information or noisy datainformation or noisy data and can be very effective especially and can be very effective especially in situations where it is not possible in situations where it is not possible to define the rules or steps that lead to define the rules or steps that lead to the solution of a problem.to the solution of a problem.
Continuation………..Continuation………..
ApplicationsApplicationsNeural Network Applications can be grouped in following Neural Network Applications can be grouped in following
categories:categories:
Clustering:Clustering:A clustering algorithm explores the similarity A clustering algorithm explores the similarity
between patterns and places similar patterns in a cluster. between patterns and places similar patterns in a cluster. Best known applications include data compression and Best known applications include data compression and data mining.data mining.
Classification/Pattern recognition:Classification/Pattern recognition:The task of pattern recognition is to assign an The task of pattern recognition is to assign an
input pattern (like handwritten symbol) to one of many input pattern (like handwritten symbol) to one of many classes. This category includes algorithmic classes. This category includes algorithmic implementations such as associative memory.implementations such as associative memory.
Continue…Continue…..Function approximation:Function approximation:
The tasks of function approximation is to find The tasks of function approximation is to find an estimate of the unknown function f() subject to an estimate of the unknown function f() subject to noise. Various engineering and scientific disciplines noise. Various engineering and scientific disciplines require function approximation. require function approximation. Prediction/Dynamic Systems:Prediction/Dynamic Systems:
The task is to forecast some future values of a The task is to forecast some future values of a time-sequenced data. Prediction has a significant time-sequenced data. Prediction has a significant impact on decision support systems.impact on decision support systems.
Prediction differs from Function Prediction differs from Function approximation by considering time factor.approximation by considering time factor.
Here the system is dynamic and may produce Here the system is dynamic and may produce different results for the same input data based on different results for the same input data based on system state (time).system state (time).
OverviewOverviewBiological inspiration
Artificial neurons and Artificial neural networks
Learning processes
Learning with artificial neural networks
The Brain
The Brain as an Information Processing System
The human brain contains about 1010 basic units called neurons or nerve cells.
Each neuron in turn, is connected to about 104 other neurons
On average, each neuron is connected to other neurons through about 10,000 synapses.
Neurons and Synapses• The basic computational unit in the nervous system is the nerve cell, or neuron. • A neuron is a small cell that receives electron chemical signals from its various sources and in turn responds by transmitting electrical impulses to other neurons. • A neuron has:
Dendrites (inputs) Cell body Axon (output)
•
Continue……Continue……A neuron receives input from other neurons A neuron receives input from other neurons
(typically many thousands). (typically many thousands). Inputs sum (approximately). Inputs sum (approximately). Once input exceeds a critical level, the Once input exceeds a critical level, the neuron discharges a spike - an electrical neuron discharges a spike - an electrical pulse that travels from the body, down the pulse that travels from the body, down the axon, to the next neuron(s) (or other axon, to the next neuron(s) (or other receptors). receptors). This spiking event is also called This spiking event is also called depolarization, and is followed by a depolarization, and is followed by a refractory period, during which the neuron is refractory period, during which the neuron is unableunable to fire. to fire.
CONTINUE…….The axon endings (Output Zone) almost touch the
dendrites or cell body of the next neuron. Transmission of an electrical signal from one neuron to
the next is effected by neurotransmittors, chemicals which are released from the first neuron and which bind to receptors in the second.
This link is called a synapse. The extent to which the signal from one neuron is
passed on to the next depends on many factors, e.g. the amount of neurotransmittor available, the number and arrangement of receptors, amount of neurotransmittor reabsorbed, etc.
Difference between biological Difference between biological neuron and artificial neuronneuron and artificial neuron
Characteristics
Artificial neural network
Biological neural network
Speed Faster in processing information
Slower in processing information
Processing In a sequential mode(One after another)
Parallel operations
Size and complexity
Small size and less complex
Big size and more complex
Fault tolerance Not fault tolerant(information corrupted in the memory cannot be retrieved)
Tolerant
Control mechanism
Control unit monitors all the activities
There is no central control for processing information in the brain
STRUCTURE OF NEURONSTRUCTURE OF NEURON
Artificial neuronsArtificial neurons
Neurons work by processing information. They receive and provide information in form of spikes.
The McCulloch-Pitts model
Inputs
Outputw2
w1
w3
wn
wn-1
. . .
x1
x2
x3
…
xn-1
xn
y)0(;
1
fyxwon
iii
Artificial neural networksArtificial neural networks
Inputs
Output
An artificial neural network is composed of many artificial neurons that are linked together according to a specific network architecture. The objective of the neural network is to transform the inputs into meaningful outputs.
A vague description is as followsA vague description is as follows: :
• An ANN is a network of many very simple An ANN is a network of many very simple processors ("units"), each possibly having a processors ("units"), each possibly having a (small amount of) local memory. (small amount of) local memory.
• The units are connected by unidirectional The units are connected by unidirectional communication channels ("connections"), communication channels ("connections"), which carry numeric (as opposed to which carry numeric (as opposed to symbolic) data. symbolic) data.
• The units operate only on their local data The units operate only on their local data and on the inputs they receive via the and on the inputs they receive via the connections. connections.
CONTINUE…..CONTINUE….. The design motivation is what The design motivation is what
distinguishes neural networks from other distinguishes neural networks from other mathematical techniquesmathematical techniques
A neural network is a processing device, A neural network is a processing device, either an algorithm, or actual hardware, either an algorithm, or actual hardware, whose design was motivated by the whose design was motivated by the design and functioning of human brains design and functioning of human brains and components thereof. and components thereof.
Most neural networks have some sort of Most neural networks have some sort of "training" rule whereby the weights of "training" rule whereby the weights of connections are adjusted on the basis of connections are adjusted on the basis of presented patterns. presented patterns.
CONTINUE…..CONTINUE…..
In other words, neural networks "learn" from In other words, neural networks "learn" from examples, just like children learn to recognize examples, just like children learn to recognize dogs from examples of dogs, and exhibit some dogs from examples of dogs, and exhibit some structural capability for generalization. structural capability for generalization.
Neural networks normally have great potential Neural networks normally have great potential for parallelism, since the computations of the for parallelism, since the computations of the components are independent of each other. components are independent of each other.
Biological neural networks are much more Biological neural networks are much more complicated in their elementary structures than complicated in their elementary structures than the mathematical models we use for ANNs.the mathematical models we use for ANNs.
They typically consist of many simple They typically consist of many simple processing units, which are wired together in a processing units, which are wired together in a complex communication network.complex communication network.
Basic ConceptsBasic Concepts
Neural Network
Input 0 Input 1 Input n...
Output 0 Output 1 Output m...
A Neural Network generally A Neural Network generally maps a set of inputs to a set of maps a set of inputs to a set of outputsoutputs
Number of inputs/outputs is Number of inputs/outputs is variablevariable
The Network itself is composed The Network itself is composed of an arbitrary number of nodes of an arbitrary number of nodes with an arbitrary topologywith an arbitrary topology
Activation functionActivation function Activation function is used to calculate the output response Activation function is used to calculate the output response
of a neuron. of a neuron. A few linear and non linear activation functions areA few linear and non linear activation functions are 1. Identify function1. Identify function 2. sigmoidal functions2. sigmoidal functions 3. Binary sigmoidal functions3. Binary sigmoidal functions 4. Bipolar sigmoidal function4. Bipolar sigmoidal function 5. Signum function5. Signum function 6.Binary step function6.Binary step function 7. Bipoloar binary function 7. Bipoloar binary function 8. Uni poloar binary function 8. Uni poloar binary function
Neural network Neural network ArchitecturesArchitectures
1.Single layer feedforward network1.Single layer feedforward network
- only input and output layer- only input and output layer
2. Multi layer Feedforward network2. Multi layer Feedforward network
- Input,output and hidden layer.- Input,output and hidden layer.
3. Recurrent networks3. Recurrent networks
- Contain feedback link - Contain feedback link
Different Forms of Learning:Different Forms of Learning:– Learning agent receives feedback with Learning agent receives feedback with
respect to its actions (e.g. using a teacher)respect to its actions (e.g. using a teacher) Supervised LearningSupervised Learning: feedback is : feedback is
received with respect to all possible received with respect to all possible actions of the agentactions of the agent
Reinforcement LearningReinforcement Learning: feedback is : feedback is only received with respect to the taken only received with respect to the taken action of the agentaction of the agent
– Unsupervised Learning: Unsupervised Learning: Learning when Learning when there is no hint at all about the correct there is no hint at all about the correct actionaction
Learning methodsLearning methods
Classification of learningClassification of learning FigFig
Neural networks learning algorithm
HebbianStochasticError correction
GradientDescent
BackPropagation
Least MeanSquare
Unsupervised learningSupervised learning(Error based)
Competitative
Reinforced learning(output based)
Supervised LearningSupervised Learning Every input pattern that is used to train the n/w Every input pattern that is used to train the n/w
is associated with an output pattern,which is is associated with an output pattern,which is the target or the desired pattern.the target or the desired pattern.
A A teacherteacher is assumed to be present during the is assumed to be present during the learning process, when a comparison is made learning process, when a comparison is made between the networks computed output and between the networks computed output and the correct expected , to deternmine the error.the correct expected , to deternmine the error.
The error can then be used to change n/w The error can then be used to change n/w parameters , which result is an improvement in parameters , which result is an improvement in performanceperformance
Block diagram of supervised Block diagram of supervised learninglearning
XXAdaptive Network
Distance Generator
O
Learning signal
d
Unsupervised LearningUnsupervised Learning The target output is not presented to the n/wThe target output is not presented to the n/w No teacherNo teacher to present the desired patterns. to present the desired patterns. Systems learns of its own by discovering a d Systems learns of its own by discovering a d
adapting to structural features in the input adapting to structural features in the input patterns.patterns.
OO XX
Adaptive NetworkW
Reinforced LearningReinforced Learning A teacher though available, does not A teacher though available, does not
present the expected answer but only present the expected answer but only indicates is the computed output is indicates is the computed output is correct or incorrect.correct or incorrect.
The information provided helps the n/w The information provided helps the n/w in its learning process.in its learning process.
A reward is given for a correct answer A reward is given for a correct answer computed and a penalty for a wrong computed and a penalty for a wrong answer. But , reinforced learning is not answer. But , reinforced learning is not one of the popular forms of learning.one of the popular forms of learning.
Supervised and unsupervised are most Supervised and unsupervised are most popular formspopular forms
Various RulesVarious Rules Those learning methods have found expression Those learning methods have found expression
through various learning rules,through various learning rules, Some of the rules are,Some of the rules are, 1. Hebbian learning rule1. Hebbian learning rule 2. Perceptron learning rule2. Perceptron learning rule 3. Delta learning rule3. Delta learning rule 4. Widrow-Hoff learning rule4. Widrow-Hoff learning rule 5. Correlation learning rule5. Correlation learning rule 6. Winner-Take-All Learning rule6. Winner-Take-All Learning rule 7. Outstar Learning rule7. Outstar Learning rule
Hebbian Learning ruleHebbian Learning rule Proposed by Hebb(1949)Proposed by Hebb(1949) Based on correlative weight adjustmentBased on correlative weight adjustment Oldest learning mechanismOldest learning mechanism The input output patterns pairs (Xi,Yi ) are The input output patterns pairs (Xi,Yi ) are
associated by the weight matrix w, associated by the weight matrix w, unknown as the correlation matrix.unknown as the correlation matrix.
It is computed as,It is computed as,
W W i+1i+1 = W = Wii + C Oi xi + C Oi xi
Perceptron Learning rulePerceptron Learning rule For the perceptron learning rule, the For the perceptron learning rule, the
learning signal is the difference learning signal is the difference between the desired and actual between the desired and actual neuron's response neuron's response
Learning is supervisedLearning is supervised r=d r=d i i - o - o i i
The rule isThe rule is , ,
W W i+1i+1 = W = Wii + C (d + C (dii - O - Oii )xi )xi
Perceptron Learning rulePerceptron Learning rule
TLUTLU XX11 W Wi1 i1
XX22 W Wi2 net ii2 net i
WWi3 i3 OOii
XX33
C (di-oi)X i C (di-oi)X i
XX44 W Wi4i4
XX ddii-O-Oii d dii
CC
+ 1
0 net -1
Delta learning ruleDelta learning rule It is only valid for continuous It is only valid for continuous
activation function.activation function. Supervised learning.Supervised learning.
f’ (net i) is the derivative of the f’ (net i) is the derivative of the activation function f(net)activation function f(net)
W W i+1i+1 = W = Wii + C (d + C (dii - O - Oii ) f’(net i)xi ) f’(net i)xi
Delta Learning ruleDelta Learning rule
XX11 W Wi1 continuous perceptron i1 continuous perceptron
XX22 W Wi2 net ii2 net i
WWi3 i3 OOii
XX33
W i W i
XX44 W Wi4i4
X X ddii-O-Oii d dii
CC
PerceptronPerceptronThe perceptron is a computational model The perceptron is a computational model
of the retina of the eye and hence, is named of the retina of the eye and hence, is named ‘Perceptron’ .‘Perceptron’ .
The network comprises three units, The network comprises three units, 1. Sensory unit (S)1. Sensory unit (S)2. Association Unit (A)2. Association Unit (A)3. Response Unit (R)3. Response Unit (R)
The S unit comprising 400 photodetectors The S unit comprising 400 photodetectors receives input images and provides a 0/1 receives input images and provides a 0/1 electric signal as output.electric signal as output.
If the output signals exceed a threshold, then If the output signals exceed a threshold, then the photodetector outputs 1 else 0.the photodetector outputs 1 else 0.
Rosenblatt’s original Perceptron modelRosenblatt’s original Perceptron model
Sensory unit(s) Association ResponseSensory unit(s) Association Response
unit(A) unit(R)unit(A) unit(R)
O/PO/P
Fixed W adjustable WFixed W adjustable W
Photo detectorsPhoto detectors
Continue….Continue…. Photo detector are randomly connected Photo detector are randomly connected
to the association unit Ato the association unit A The unit A comprises feature The unit A comprises feature demons or demons or
predicatespredicates TheThe predicates predicates examine the output of the examine the output of the
S unit for specific features of the image.S unit for specific features of the image. The R unit comprises pattern recognizers The R unit comprises pattern recognizers
or perceptrons, which receives the results or perceptrons, which receives the results of the predicate , also in binary form.of the predicate , also in binary form.
While the weights of the S and A units are While the weights of the S and A units are fixed, those of R are adjustable.fixed, those of R are adjustable.
Output calculationOutput calculation The output of the R unit could be such The output of the R unit could be such
that if the weighted sum of its inputs is that if the weighted sum of its inputs is less the or equal to 0, the output is 0 less the or equal to 0, the output is 0 otherwise it is the weighted sum itself.otherwise it is the weighted sum itself.
The training algorithm of the perceptron is The training algorithm of the perceptron is a supervised learning algorithma supervised learning algorithm
Yj = f(net j) = 1, if net j > 0 = 0, otherwiseWhere, net j = ;
1
n
iiixw
ADALINEADALINE Adaptive Linear network.Adaptive Linear network. Benard widrow of Stanford UniversityBenard widrow of Stanford University Makes use if Supervised learningMakes use if Supervised learning A simple ADALINE networks is belowA simple ADALINE networks is below W0W0
W1 Y OutputW1 Y Output
w2w2
WnWn
Thresholding functionThresholding function
Σx0
x1
x2
xn
Continue…Continue… There is only one output neuron and the There is only one output neuron and the
output values are bipolar(-1 or +1)output values are bipolar(-1 or +1) The learning algorithm is Least mean The learning algorithm is Least mean
square or Delta rule.square or Delta rule.
ADALINE n/w s has had the most ADALINE n/w s has had the most successful applications because it is successful applications because it is virtually in all high speed modems and virtually in all high speed modems and telephone switching systems to cancel telephone switching systems to cancel the echo in long distance the echo in long distance communication circuits.communication circuits.
W W i+1i+1 = W = Wii + C (d + C (dii - O - Oii ) f’(net i)xi ) f’(net i)xi
MADALINE NetworkMADALINE Network A MADALINE(Many ADALINE) network is A MADALINE(Many ADALINE) network is
created by combining a number of adalinescreated by combining a number of adalines The n/w of ADALINES can span many layers.The n/w of ADALINES can span many layers. A simple MADALINE N/W is shown belowA simple MADALINE N/W is shown below Inputs outputsInputs outputs
:Adaline nw:Adaline nw
A
A
A
A
A A
A
A
Hop field networkHop field network Described by J.J.HopfieldDescribed by J.J.Hopfield Topology is simpleTopology is simple It consists of ‘n’ neurons which are all It consists of ‘n’ neurons which are all
networked with each other.networked with each other. A Hopfield network is able to recognize A Hopfield network is able to recognize
unclear pictures correctly.unclear pictures correctly. The discrete Hopfield net is a fully The discrete Hopfield net is a fully
interconnected neural net with each unit interconnected neural net with each unit connected to every other unit.connected to every other unit.
The net has symmetric weights with no self The net has symmetric weights with no self connections that is all the diagonal elements connections that is all the diagonal elements of the weight matrix of a Hopfield net are of the weight matrix of a Hopfield net are zero.zero.
continuecontinue Wij = Wji and Wii = 0Wij = Wji and Wii = 0
Weight matrix is,Weight matrix is,
[0 W12 W13 ……. W1n[0 W12 W13 ……. W1n
W = W21 0 W23 ……. W2n]W = W21 0 W23 ……. W2n]
W31 W32 0 …… W3nW31 W32 0 …… W3n
. . . …… .. . . …… .
. . . ….. .. . . ….. .
Wn1 Wn2 Wn3 ….. 0 ]Wn1 Wn2 Wn3 ….. 0 ]
Some times called connectivity matrixSome times called connectivity matrix
Single-layer feedback neural Single-layer feedback neural networknetwork
-1 -1
i1 T1i1 T1
VV11 wn1wn1 VV11
i2i2
VV22 w2nw2n T2T2 VV2 2
Wn2Wn2
VVnn -1 -1 VVnn
1 2
n