View
306
Download
1
Category
Preview:
Citation preview
Neural Networks
Dr. Randa Elanwar
Lecture 3
Lecture Content
• Basic models of ANN
– Activation functions
– Interconnections (different NN structures)
– Important notations
2Neural Networks Dr. Randa Elanwar
Basic models of ANN
3Neural Networks Dr. Randa Elanwar
Basic Models of ANN
Activation function
Interconnections Learning rules
Activation function
• Bipolar binary and unipolar binary are called as hard limiting activation functions used in discrete neuron model
• Unipolar continuous and bipolar continuous are called soft limiting activation functions are called sigmoidal characteristics.
Neural Networks Dr. Randa Elanwar 4
Activation functions
Neural Networks Dr. Randa Elanwar 5
Bipolar continuous (sigmoidal)
Bipolar binary functions (sign function)
Activation functions
Neural Networks Dr. Randa Elanwar 6
Unipolar continuous (sigmoidal)
Unipolar Binary (Step function)
Activation functions
• An output of 1 represents firing of a neuron down the axon.
7Neural Networks Dr. Randa Elanwar
f(in) f(in) f(in)
Basic models of ANN
8Neural Networks Dr. Randa Elanwar
Basic Models of ANN
Activation function
Interconnections Learning rules
Classification based on interconnections
9Neural Networks Dr. Randa Elanwar
Interconnections
Feed forward
Single layer
Multilayer
Feed Back Recurrent
Single layer
Multilayer
The Perceptron
• First studied in the late 1950s (Rosenblatt).
• Definition: an arrangement of one input layer (more than 1 unit/node) of McCulloch-Pitts neurons feeding forward to one output layer of McCulloch-Pitts neurons is known as a Perceptron.
• Any number of McCulloch-Pitts neurons can be connected together in any way we like. Thus, it is also known as Layered Feed-Forward Networks.
• We can use McCulloch-Pitts neurons to implement the basic logic gates. All we need to do is find the appropriate connection weights and neuron thresholds to produce the right outputs for each set of inputs.
10Neural Networks Dr. Randa Elanwar
11
The Perceptron
Neural Networks Dr. Randa Elanwar
Single layer Feedforward Network
12Neural Networks Dr. Randa Elanwar
Feedforward Network
• Its output and input vectors are respectively
• Weight wij connects the i’th neuron with j’thinput. Activation rule of ith neuron is
where
13Neural Networks Dr. Randa Elanwar
Multilayer feed forward network
Can be used to solve complicated problems
14Neural Networks Dr. Randa Elanwar
Feedback network
15Neural Networks Dr. Randa Elanwar
When outputs are directed back as inputs to same or preceding layer nodes it results in the formation of feedback networks
Lateral feedback
16Neural Networks Dr. Randa Elanwar
If the feedback of the output of the processing elements is directed back as input to the processing elements in the same layer then it is called lateral feedback
Recurrent networks
17Neural Networks Dr. Randa Elanwar
• Types:
• Single node with own feedback
• Competitive nets
• Single-layer recurrent networks
• Multilayer recurrent networks
Feedback networks with closed loop are called Recurrent Networks. The response at the k+1’th instant depends on the entire history of the network starting at k=0.
A Brief History
• 1943 McCulloch and Pitts proposed the McCulloch-Pitts neuron model
• 1949 Hebb published his book The Organization of Behavior, in which the Hebbian learning
rule was proposed.
• 1958 Rosenblatt introduced the simple single layer networks now called Perceptrons.
• 1969 Minsky and Papert’s book Perceptrons demonstrated the limitation of single layer
perceptrons, and almost the whole field went into hibernation.
• 1982 Hopfield published a series of papers on Hopfield networks.
• 1982 Kohonen developed the Self-Organizing Maps that now bear his name.
• 1986 The Back-Propagation learning algorithm for Multi-Layer Perceptrons was re-
discovered and the whole field took off again.
• 1990s The sub-field of Radial Basis Function Networks was developed.
• 2000s The power of Neural Networks Ensembles & Support Vector Machines is apparent.19Neural Networks Dr. Randa Elanwar
Linearly Separable Functions
• Consider a perceptron:
• Its output is
– 1, if W1X1 + W2X2 >
– 0, otherwise
• In terms of feature space
– hence, it can only classify examples if a line can separate the positive examples from the negative examples
20Neural Networks Dr. Randa Elanwar
Learning Linearly Separable Functions
• What can these functions learn ?
• Bad news:
- There are not many linearly separable functions.
• Good news:
- There is a perceptron algorithm that will learn
any linearly separable function, given enough
training examples.
21Neural Networks Dr. Randa Elanwar
22
Important notations
• One neuron can’t do much on its own. Usually we will have many neurons labeled by indices k, i, j and activation flows between them via links with strengths wki, wij:
Neural Networks Dr. Randa Elanwar
Recommended