View
1
Download
0
Category
Preview:
Citation preview
Perceptron Learning
CSC 302 1.5 Neural Networks
Perceptron Learning
Rule
1
Objectives
� Determining the weight matrix and bias for
perceptron networks with many inputs.
� Explaining what a learning rule is.
� Developing the perceptron learning rule.� Developing the perceptron learning rule.
� Discussing the advantages and limitations of
the single layer perceptron.
2
Development
� Introduced a neuron model by Warren
McCulloch & Walter Pitts [1943].
� Main features� Weighted sum of input signals is compared to a
threshold to determine the output.threshold to determine the output.
� 0 if weighted_sum < 0
� 1 is weighted_sum >= 0
� Able to compute any logical arithmetic function.
� No training method was available.
3
Development �
� Perceptron was developed by Frank Rosenblatt [1950].
� Neurons were similar to those of McCulloch & Pitts.
� Key feature – introduced a learning rule.� Key feature – introduced a learning rule.
� Proved that learning rule is always converged to correct weights if weights exist for the problem.
� Simple and automatic.
� No restriction on initial weights - random
4
Learning Rules
� Procedure for modifying the weights and biases of a
network to perform a specific task.
� Supervised Learning - Network is provided with a set of
examples of proper network behaviour (inputs/targets)
� Reinforcement Learning - Network is only provided with a � Reinforcement Learning - Network is only provided with a
grade, or score, which indicates network performance.
� Unsupervised Learning - Only network inputs are
available to the learning algorithm. Network learns to
categorize (cluster) the inputs.
5
Perceptron Architecture
6
Perceptron Architecture ?
7
� Output of the ith neuron
Perceptron Architecture ?
Therefore, if the inner product of the ith row of the weight
matrix with the input vector is greater than or equal to –bi
the output will be 1, otherwise the output will be 0.
8
Each neuron in the network divides the input space into
two regions.
Single-Neuron Perceptron
9
� Decision boundary
n = 1wTp+b = w1,1p1 + w1,2p2 + b = 0
Decision Boundary
� Decision boundary
1wTp+b = 0 or 1w
Tp = -b
� All points on the decision boundary have the
same inner product with the weight vector.
� Decision boundary is orthogonal to weight vector.
10
� Decision boundary is orthogonal to weight vector.
�1wTp1= 1w
Tp2 = -b for any two points in the decision
boundary.
�1wT(p1 – p2) = 0
�Weight vector is orthogonal to (p1 – p2).
Direction of the Weight Vector
� Any vector in the shaded region
will have an inner product
greater than –b and
� Vectors in the un-shaded region
will have inner product less than
11
will have inner product less than
–b.
� Therefore the weight vector 1w
will always point toward the
region where the neuron output
is 1.
Graphical Method
� Design of a perceptron to implement the AND
gate.
12
Input space – each input vector labeled
according to the target.
Dark circle – output is1
Light circle – output is 0
Graphical Method ?
� First select a decision boundary
that separates dark circles and
light circles.
� Next choose a weight vector that
is orthogonal to the decision
13
is orthogonal to the decision
boundary.
� The weight vector can be any
length.
� Infinite no of possibilities.
� One choice is
Graphical Method ?
� Finally, we need to find the bias, b.
� Pick a point on the decision boundary (say
[1.5 0]T)
14
� Testing
Multiple-Neuron Perceptron
� Each neuron will have its own decision boundary.
� iwTp + bi = 0
� A single neuron can classify input vectors into two
categories.
� A multi-neuron perceptron can classify input
15
� A multi-neuron perceptron can classify input
vectors into 2S categories.
Perceptron Learning Rule
� Supervised training
� Provided a set of examples of proper network
behaviour
where p – input to the network and
16
where pq – input to the network and
tq – corresponding output
� As each input is supplied to the network, the
network output is compared to the target.
� The learning rule then adjusts the weights and
biases of the network in order to move the
network output closer to the target.
Test Problem
=
−
=
=
−=
=
= 0,
1
00,
2
11,
2
13321 ttt ppp 21
� Input/target pairs
Decision
17
� Removed the bias for the simplicity.
� Decision boundary must pass the origin.
Decision boundaries
WeightVectors
Starting Point
−
=8.0
0.1w1
� Random initial weight
� Present p1 to the network:
T 1
18
a hardlim wT
1 p1( ) hardlim 1.0 0.8–1
2
= =
a hardlim 0.6–( ) 0= =
Incorrect Classification.
Tentative Learning Rule
�We need to alter the
weight vector so that it
points more toward p1,
19
points more toward p1,
so that in the future it
has a better chance of
classifying p1.
.
Tentative Learning Rule ?
�One approach would be
to set 1w equal to p1.
� This rule cannot find a
solution always.
20
If we apply the rule 1w = p
every time one of these
vectors misclassified, and
network weights will simply
oscillate back and forth.
If we apply the rule 1w = p
every time one of these
vectors misclassified, and
network weights will simply
oscillate back and forth.
Tentative Learning Rule ?
� Another possibility would
be to add p1 to 1w.
� This rule can be stated
as
21
Second Input Vector
22
Third Input Vector
23
Unified Learning Rule
24
Unified Learning Rule ?
25
Multiple-Neuron Perceptron
26
Apple/Banana Example
27
Second Iteration
28
Check
29
Perceptron Rule Capability
30
Perceptron Limitations
31
Recommended