By Nermeen Shaltout

Problem DefinitionMotivationResearch ObjectiveApproachRelated WorkReferences

A neural network is a mimic of a biological neurone.It fires an output when a certain threshold is reached. The goal of training neural networks is to adjust the weights of the neural network, such that the neural network produces a specific output for a certain input.Usually simple algorithms were used to adjust the weights during training. E.g. Gradient Descent

Neural networks have been used in computer science and other fields to act as classifiers and predictors.The more complex the neural network the harder it is to train it using conventional methods such as gradient decent, like in Forward Neural Networks.For example, the whole algorithm in Recurrent neural networks has to be unrolled which leads to time complexities.Recurrent neural networks are complex neural networks with feedback connections which are impossible to solve using basic methods such as gradient decent; it has both forward and backward connections.

With the increasing amount of complex neural network being used to simulate problems in the medical field, faster and more efficient ways in training neural networks have to used.For example, to simulate a neural network that acts like a biological neural networks a recurrent neural network is needed.Although lots of studies have been conducted to use evolutionary techniques such as genetic algorithms, and particle swarm optimization to train complex neural networks. The techniques are still inefficient and need further improvement to yield both fast and optimal results.

To train complex neural networks such as recurrent neural networks with less errors, and in a shorter time.In order to achieve this objective, Breeding Swarms are used which combine a number of evolutionary algorithm techniques to train the neural network; Particle Swarm Optimization and Genetic Algorithms.The main goal is to make Breeding Swarms more efficient in training complex neural networks.

Briefing on PSO, and Training a Neural Network with PSO (Old method) Briefing on GA and Training a Neural Network with GA (Old method)Training a Neural Network with Breeding Swarms Comparison of Results of using Breeding Swarms with GA, and PSOImprovement on Breeding Swarms Technique

Borrows it’s concepts from genes; Survival of the fittestUses a fitness function to measure which genes or solutions can pass onto the next generation. E.gshortest path for travelling salesman problem.Discards the a portion of the unfit solutions.May mutate or cross over other solutions.Disadvantage: Has no memory. Does not always produce the best solution. Doesn’t converge swiftly.Advantage: Less chance of falling in local minima.

It is a form of swarm intelligenceOne of its applications is training neural networksConsist of multiple agents working in parallel.The algorithm works by updating the positions/performance of the agents, while storing the best performance of each agent and as well as the best global performance of all the agents.All of particles have a fitness values which are evaluated by the fitness function to be optimized, and have velocities which direct the flying of the particles. The particles fly through the problem space by following the current optimum particles. Algorithm is stopped after a certain number of iterations, or after the error reaches a certain threshold.

In particular, each particle in PSO denotes a set of weights of the network.The dimension of each particle is same as the number of weights of a certain network.Training a network using PSO means moving the particle among the weight space to minimize the MSE.Each particle in the PSO can also be used to represent the structure of the neural network, i.e the number of hidden layers, the number of neurones in each layer, and the connections between nodes, to train the network accordingly.This is a better strategy than GA but training the weights weights alone is time consuming as many architectures have to be tried to see which one is the most optimal.

An inner PSO is used to train the weights of the network.An outer PSO is used to come up with the best structure for the network. Even though this approach takes into consideration both the weight and the structure of the algorithm it is still not optimal compared to GA and evolutionary algorithms

The Classification Error Percentage (CEP) of PSO-PSO is higher than using one PSO, but it is still lower than some evolutionary and genetic algorithm techniques. It is also very time consuming due to using two PSOs.

Breeding swarms combines PSO and genetic algorithms (GA)They are better than PSO, and genetic algorithms as they combine the advantages of both. PSO tends to converge quickly at a local minimum, so the GA is used to reset some of the memories of the next generation so that new solutions can be found.The Breeding Swarm is used to update the weights of a complex network such as a recurrent neural network.

A recurrent neural network (RNN) is a class of neural network where connections between units form a directed cycle. This creates an internal state of the network which allows it to exhibit dynamic temporal behaviour.Recurrent neural networks must be approached differently from feedforward neural networks, both when analyzing their behaviour and training them. While a feed forward network propagates data linearly from input to output, recurrent networks (RN) also propagate data from later processing stages to earlier stages.

A breeding coefficient is used to decide which portion of the population is to be discarded, like GA.The non-discarded of particles (containing RNN weights) are updated, as in normal PSO.Tournament selection is then used to select the best k particles or solutions as parents for crossover. In this case 2 parents. A special crossover operator is used for breeding swarms; Velocity Propelled Averaged Crossover(VPAC), that takes into account PSO parameters.Gaussian mutation is used to alter the weights with a probability of 0.1 after crossover.

The goal is to create children with a position in between the two parents, but that are accelerated away from the parents.This is to avoid premature convergence in the search space.c1(xi) and c1(xi) : positions of child 1 and 2 in dimension i.p1(xi) and p2(xi) : positions of parents 1 and 2 in dimension i.p1(vi) and p2(vi) : velocities of parents 1 and 2 in dimension i.Φ: is a uniform random variable in the range [0.0:1.0]. The child particles retain their parent’s velocity vector, c1(~v) = p1(~v),c2(~v) = p2(~v). The previous best vector is set to the new position vector, restarting the child’s memory, c1(~p) = p1(~x),c2(~p) = p2(~x).

An additional advantage of the Breeding Swarm algorithm may be the ability to evolve network topologies and weight simultaneously, a trait not available in the standard PSO implementation.Through the crossover operator, a particle may be allowed to add new nodes or remove nodes.Breeding swarms are have been proven to be more optimal than PSO and GA alone, so updating the architecture simultaneously with the weights should make both faster and more optimal. The performance will be measured by seeing which training method generates the lowest Classification Error Percentage(CEP) used in PSO-PSO method.

Xiaodong Li, Andries P. Engelbrecht , GECCO 2007 Tutorial, “Particle Swarm Optimization”.Marcio Carvalho and Teresa B. Ludermir, “Particle Swarm Optimization of Neural Network Architectures and Weights”, 2007<http://doi.ieeecomputersociety.org/10.1109/HIS.2007.45>Mathew Settles, Paul Nathan, Terence Soul, “Breeding Swarms: A New Approach to Recurrent Neural Network Training”Bo Liu, Ling Wang, Yihui Jin, and Dexian Huang, “Designing Neural Networks Using PSO-Based MemeticAlgorithm””,http://0-www.springerlink.com.lib.aucegypt.edu<http://www.swarmintelligence.org/tutorials.php>