SOFT COMPUTING

Subject Code : 18MIT24C

Prepared by Dr. N.Thenmozhi

UNIT-I: Fundamentals of Neural Networks: Basic concepts of Neural Networks –Human

Brain – Model of an Artificial Neuron – Neural Network Architectures – Characteristics of

Neural Networks – Learning methods - Easy Neural Network Architectures – Some

Application domains.

UNIT-II: Back propagation Networks: Architecture of a Back-Propagation Network – Back

propagation Learning- Effect of Tuning parameters of the Back Propagation Neural Network

– Selection of various parameters in BPN.

UNIT-III: Adaptive Resonance Theory: Introduction: Cluster Structure, Vector

Quantization, Classical ART Networks, Simplified ART Architecture. ART1: Architecture of

ART1–Special features of ART1 Models-ART1 Algorithms. ART2: Architecture of ART2–

ART2 Algorithms.

UNIT-IV: Fuzzy Set Theory: Fuzzy versus crisp, Crisp sets: Operation on Crisp sets-

Properties of Crisp Sets-Partition and Covering. Fuzzy sets: Membership Function – Basic

fuzzy set Operations-properties of fuzzy sets. Crisp relations: Cartesian product-Other Crisp

Relations-Operations on Relations. Fuzzy relations: Fuzzy Cartesian product- Operations on

Fuzzy Relations.

UNIT-V: Fuzzy Systems: Crisp logic: Laws of Propositional Logic-Inference in

propositional Logic. Predicate logic: Interpretations of Predicate Logic Formula – Inference

in Predicate Logic. Fuzzy logic: Fuzzy Quantifiers – Fuzzy Inference, Fuzzy rule based

system – Defuzzification.

TEXT BOOK 1. S.Rajasekaran & G.A.Vijayalakshmi Pai, “Neural Networks, Fuzzy logic, and Genetic

Algorithms

Synthesis and Applications, PHI, 2005.

REFERENCE BOOKS

1. James A. Freeman, David M.Skapura, “Neural Networks-Algorithms, Applications, and

Programming Techniques”, Pearson Education.

2. Fredric M. Ham, Ivica Kostanic, “Principles of Neuro computing for science of

Engineering”, TMCH.

UNIT-I: Fundamentals of Neural Networks: Basic concepts of Neural Networks –Human

Brain – Model of an Artificial Neuron – Neural Network Architectures – Characteristics of

Neural Networks – Learning methods - Easy Neural Network Architectures – Some

Application domains.

1. Fundamentals of Neural Networks

What is soft computing? An approach to computing which parallels the remarkable ability of the human mind

to reason and learn in an environment of uncertainty and imprecision.

It is characterized by the use of inexact solutions to computationally hard tasks such

as the solution of nonparametric complex problems for which an exact solution can‟t be

derived in polynomial of time.

Why soft computing approach? Mathematical model & analysis can be done for relatively simple systems. More

complex systems arising in biology, medicine and management systems remain intractable to

conventional mathematical and analytical methods. Soft computing deals with imprecision,

uncertainty, partial truth and approximation to achieve tractability, robustness and low

solution cost. It extends its application to various disciplines of Engg. and science. Typically

human can:

1. Take decisions

2. Inference from previous situations experienced

3. Expertise in an area

4. Adapt to changing environment

5. Learn to do better

6. Social behaviour of collective intelligence

Intelligent control strategies have emerged from the above mentioned characteristics

of human/ animals. The first two characteristics have given rise to Fuzzy logic;2nd , 3rd and

4th have led to Neural Networks; 4th , 5th and 6th have been used in evolutionary algorithms.

Characteristics of Neuro-Fuzzy & Soft Computing: 1. Human Expertise

2. Biologically inspired computing models

3. New Optimization Techniques

4. Numerical Computation

5. New Application domains

6. Model-free learning

7. Intensive computation

8. Fault tolerance

9. Goal driven characteristics

10. Real world applications

Intelligent Control Strategies (Components of Soft Computing): The popular soft computing

components in designing intelligent control theory are:

1. Fuzzy Logic

2. Neural Networks

3. Evolutionary Algorithms

Fuzzy logic: Most of the time, people are fascinated about fuzzy logic controller. At some point of time in

Japan, the scientists designed fuzzy logic controller even for household appliances like a

room heater or a washing machine. Its popularity is such that it has been applied to various

engineering products.

Neural networks: Neural networks are basically inspired by various way of observing the biological

organism. Most of the time, it is motivated from human way of learning. It is a learning

theory. This is an artificial network that learns from example and because it is distributed in

nature, fault tolerant, parallel processing of data and distributed structure.

The basic elements of artificial Neural Network are: input nodes, weights, activation

function and output node. Inputs are associated with synaptic weights. They are all summed

and passed through an activation function giving output y. In a way, output is summation of

the signal multiplied with synaptic weight over many input channels.

Evolutionary algorithms: These are mostly derivative free optimization algorithms that perform random search

in a systematic manner to optimize the solution to a hard problem. In this course Genetic

Algorithm being the first such algorithm developed in 1970‟s will be discussed in detail. The

other algorithms are swarm based that mimic behaviour of organisms, or any systematic

process.

1.1. Basic concepts of Neural Networks

Neurons are not only enormously complex but also vary considerably in the details of

their structure and function. We will therefore describe typical properties enjoyed by a

majority of neurons and make the usual working assumption of connectionism that these

provide for the bulk of their computational ability. Readers interested in finding out more

may consult one of the many texts in neurophysiology; Thompson (1993) provides a good

introductory text, while more comprehensive accounts are given by Kandel et al. (1991) and

Kuffler et al. (1984).

All the methods discussed so far makes a strong assumption about the space around;

that is, when we use whether a neural network or fuzzy logic or/and any method that may

have been adopted in intelligent control framework, they all make always very strong

assumptions and normally they cannot work in a generalized condition. The question is that

can they hypothesize a theory? When I design all these controllers, I always take the data; the

engineer takes the data. He always builds these models that are updated. They update their

own weights based on the feedback from the plant. But the structure of the controller, the

model by which we assume the physical plant, all these are done by the engineer and also the

structure of the intelligent controller is also decided by the engineer. We do not have a

machine that can hypothesize everything; the model it should select, the controller it should

select, looking at simply data. As it encounters a specific kind of data from a plant can it

come up with specific controller architecture and can it come up with specific type of system

model? That is the question we are asking now.

You will see that in the entire course we will be discussing various tools. They will

only be dealing with these two things; behaviour. These tools are actually developed by

mimicking the human behavior, but not the human way of working. An intelligent machine is

one which learns, thinks and behaves in line with the thought process. That we would like but

we are very far from it. At least, at the moment, we are very far from this target of achieving

real intelligence.

We perceive the environment in a very unique way, in a coherent manner. This is

called unity of perception and intelligence has also something to do with this unity of

perception, awareness and certain things are not very clear to us until now. So an intelligent

machine is one which learns, thinks & behaves in line with thought process.

Neural networks are analogous to adaptive control concepts that we have in control

theory and one of the most important aspects of intelligent control is to learn the control

parameters, to learn the system model. Some of the learning methodologies we will be

learning here is the error-back propagation algorithm, real-time learning algorithm for

recurrent network, Kohonen‟s self organizing feature map & Hopfield network.

Features of Artificial Neural Network (ANN) models:

1. Parallel Distributed information processing

2. High degree of connectivity between basic units

3. Connections are modifiable based on experience

4. Learning is a continuous unsupervised process

5. Learns based on local information

6. Performance degrades with less units

Definition: the ability to learn, memorize and still generalize, prompted research in

algorithmic modeling of biological neural systems.

Do you think that computer smarter than human brain?

“While successes have been achieved in modeling biological neural systems, there are

still no solutions to the complex problem of modeling intuition, consciousness and emotion –

which form integral parts of human intelligence” Alan Turing, 1950

---Human brain has the ability to perform tasks such as pattern recognition, perception and

motor control much faster than any computer---

1.2. Human Brain

What is a neuron? A neuron is the basic processing unit in a neural network sitting on our

brain. It consists of

1. Nucleus-

2. Axon- Output node

3. Dendrites-Input node

4. Synaptic junction

The dynamics of this synaptic junction is complex. We can see the signal inputs from

the action of a neuron and through synaptic junction an output is actuated which is carried

over through dendrites to another neuron. Here, these are the neurotransmitters. We learned

from our experience that these synaptic junctions are either reinforced or in the sense they

behave in such a way that the output of synaptic junction may excite a neuron or inhibit the

neuron. This reinforcement of the synaptic weight is a concept that has been taken to artificial

neural model.

The objective is to create artificial machine and this artificial neural networks are

motivated by certain features that are observed in human brain, like as we said earlier,

parallel distributed information processing.

Biological Neuron

A nerve cell neuron is a special biological cell that processes information. According

to an estimation, there are huge number of neurons, approximately 1011

with numerous

interconnections, approximately 1015

.

Figure 1 : Structure of a neuron

Working of a Biological Neuron As shown in the above diagram, a typical neuron consists of the following four parts with

the help of which we can explain its working −

• Soma: Nucleus of neuron (the cell body) - process the input

• Dendrites: long irregularly shaped filaments attached to the soma – input channels

• Axon: another type link attached to the soma – output channels

• Output of the axon: voltage pulse (spike) that lasts for a ms

• Firing of neuron – membrane potential • Axon terminates in a specialized contact

called the synaptic junction – the electrochemical contact between neurons

• The size of synapses are believed to be linked with learning

• Larger area: excitatory—smaller area: inhibitory

ANN versus BNN Before taking a look at the differences between Artificial Neural Network ANN and

Biological Neural Network BNN, let us take a look at the similarities based on the

terminology between these two.

Biological Neural Network Artificial Neural Network

Soma Node

Dendrites Input

Synapse Weights or Interconnections

Axon Output

The following table shows the comparison between ANN and BNN based on some

criteria mentioned.

Criteria BNN ANN

Processing Massively parallel, slow

but superior than ANN

Massively parallel, fast but inferior than BNN

Size 1011

neurons and

1015

interconnections

102 to

104 nodes mainlydependsonthetypeofapplicatio

nandnetworkdesignermainlydependsonthetypeo

fapplicationandnetworkdesigner

Learning They can tolerate

ambiguity

Very precise, structured and formatted data is

required to tolerate ambiguity

Fault

tolerance

Performance degrades with

even partial damage

It is capable of robust performance, hence has

the potential to be fault tolerant

Storage

capacity

Stores the information in

the synapse

Stores the information in continuous memory

locations

1.3. Model of Artificial Neural Network

The following diagram represents the general model of ANN followed by its processing.

Figure 2 : Simple model of an artificial neuron

Our basic computational element(model neuron) is often called a node or unit.it

receives input from some other units, or perhaps from a external source. Each input has an

associated weight w, which can be modified so as to model synaptic learning. The unit

computes some function f of the weighted sum of its inputs.

Its output, in turn, can serve as input to other units. The Bias is

Activation functions : Also called the squashing function as it limits the amplitude of

the output of the neuron.

1.4 Artificial Neural Network architectures

• Neural Networks are known to be universal function approximators

• Various architectures are available to approximate any nonlinear function

• Different architectures allow for generation of functions of different complexity and power

A network topology is the arrangement of a network along with its nodes and

connecting lines. According to the topology, ANN can be classified as the following kinds −

Feedforward Network It is a non-recurrent network having processing units/nodes in layers and all the nodes

in a layer are connected with the nodes of the previous layers. The connection has different

weights upon them. There is no feedback loop means the signal can only flow in one

direction, from input to output. It may be divided into the following two types −

• Single layer feedforward network − The concept is of feedforward ANN having

only one weighted layer. In other words, we can say the input layer is fully

connected to the output layer.

• Multilayer feedforward network − The concept is of feedforward ANN having

more than one weighted layer. As this network has one or more layers between the

input and the output layer, it is called hidden layers.

Feedback Network

As the name suggests, a feedback network has feedback paths, which means the

signal can flow in both directions using loops. This makes it a non-linear dynamic system,

which changes continuously until it reaches a state of equilibrium. It may be divided into the

following types −

• Recurrent networks − They are feedback networks with closed loops. Following are

the two types of recurrent networks.

• Fully recurrent network − It is the simplest neural network architecture because all

nodes are connected to all other nodes and each node works as both input and output.

• Jordan network − It is a closed loop network in which the output will go to the input

again as feedback as shown in the following diagram.

1.5 Characteristics of Artificial Neural Network Any Artificial Neural Network, irrespective of the style and logic of implementation,

has a few basic characteristics. These are mentioned below.

• An Artificial Neural Network consists of large number of “neuron” like processing

elements.

• All these processing elements have a large number of weighted connections between

them.

• The connections between the elements provide a distributed representation of data.

• A Learning Process is implemented to acquire knowledge.

1.6 Learning Methods

• Artificial neural networks work through the optimized weight values.

• The method by which the optimized weight values are attained is called learning.

• In the learning process -> try to teach the network how to produce the output when

the corresponding input is presented

• When learning is complete: the trained neural network, with the updated optimal

weights, should be able to produce the output within desired accuracy corresponding

to an input pattern.

The Learning methods are classified into

• Supervised learning

• Unsupervised learning

• Reinforced learning

Supervised learning

Unsupervised learning The objective of unsupervised learning is to discover patterns or features in the input

data with no help from a teacher, basically performing a clustering of input space.

The system learns about the pattern from the data itself without a priori knowledge.

This is similar to our learning experience in adulthood “For example, often in our working

environment we are thrown into a project or situation which we know very little about

However, we try to familiarize with the situation as quickly as possible using our

previous experiences, education, willingness and similar other factors”

Hebbian Learning Hebb’s rule: It helps the neural network or neuron assemblies to remember specific

patterns much like the memory. From that stored knowledge, similar sort of incomplete or

spatial patterns could be recognized. This is even faster than the delta rule or the

backpropagation algorithm because there is no repetitive presentation and training of input–

output pairs.

Reinforced learning

Hebbian Learning

Hebbian Learning Rule, also known as Hebb Learning Rule, was proposed by

Donald O Hebb. Hebbian rule works by updating the weights between neurons in the neural

network for each training sample. Hebbian Learning Rule Algorithm : Set all weights to

zero, wi = 0 for i=1 to n, and bias to zero.

Competitive Learning

Stochastic Learning Stochastic refers to a variable process where the outcome involves some randomness

and has some uncertainty. It is a mathematical term and is closely related to “randomness”

and “probabilistic” and can be contrasted to the idea of “deterministic.”

In this method, weights are adjusted in a probabilistic fashion. An example is evident

in simulated annealing – the learning mechanism employed by Boltzmann and Cauchy

machines, which are a kind of NN system.

Figure 2 : Classification of learning algorithms

1.7. Early Neural Network Architectures

1.8 Some Application Domain

• Pattern Recognition and Image processing

• Optimization/constraint satisfaction

• Forecasting and risk assessment

• Control Systems