Statistical methods in ANN

7/21/2019 Statistical methods in ANN

http://slidepdf.com/reader/full/statistical-methods-in-ann 1/42

MODULE-IV

STATISTICAL METHODS IN ANN



Module 4

Statistical Methods: Boltzmann's Training - Cauchy

training - Artificial specific heat methods - applications

to general non-linear optimization problems



Statistical Methods are used for

Training ANN

roducing out!ut fro" trained net#or$

Training Methods

Deter"inistic Methods

Statistical Training Methods



Deter"inistic Training Method

%ollo#s a ste! &' ste! !rocedure(

)eights are changed &ased on their current

*alues of #eight(

It also &ased on the desired out!ut and the

actual out!ut(

E(g(+-erce!tron Training Algorith"(

,ac$ ro!agation Algorith" etc



Statistical Training Methods

Ma$e !seudo rando" change in the #eights

.etains onl' those change #hich results in

i"!ro*e"ents(



GENERAL PROCEDURE

( FOR STTISTICAL TRAINING METHOD) A!!l' a set of in!ut and co"!ute the resulting

out!ut

Co"!are the result #ith target/ find the error(

The o&0ecti*e of the training is to "ini"i1e the error(

Select a #eight in rando" and ad0ust it &' a s"all

rando" a"ount(



If the ad0ust"ent i"!ro*es our o&0ecti*e retain

the change

Other#ise return the #eight to the !re*ious

*alue

.e!eat the !rocedures until the net#or$ is

trained to the desired le*el



The local minima problem

The o&0ecti*e function "ini"i1ation !ro&le" can get

tra!!ed in !oor solution(

)eight

AB

O & 0 e c t i * e % u n c t i o n



If the o&0ecti*e function is at A and if the rando"

#eight changes are s"all then the #eight

ad0ust"ent #ill &e re0ected(

The su!erior #eight setting at !oint , #ill ne*er

found and the s'ste" #ill &e tra!!ed in local

"ini"a instead of glo&al "ini"a at !oint ,(

If the rando" #eight changes are large &oth !oint

A and , are *isited fre2uentl'/ &ut so #ill e*er'

other !oint(The #eight #ill change so drasticall' that it #ill

ne*er settle at desired !oint(



Solution & Eplanation

Statistical "ethod o*erco"e local "ini"a !ro&le" &'

a #eight ad0ust"ent strateg'(

E3a"!le+

Let the fig( re!resents a &all on a surface in a

&o3(

If the &o3 is sha$en *iolentl' /then the &all #ill

"o*e ra!idl' fro" one side to the other side(

The !ro&a&ilit' to occu!' an' !oint on the

surface is e2ual for all !oints(



If the *iolence of sha$ing is graduall' reduced the &all

#ill stic$ to &oth !oint A and ,(

If the sha$ing is again reduced it #ill settle to !oint ,(



The ANN are trained in the sa"e #a' as through

rando" #eight ad0ust"ent(

At first large rando" ad0ust"ent are "ade(

The #eight change that i"!ro*es the o&0ecti*e

function is retained(

The a*erage ste! si1e is hen graduall' reduced to

reach glo&al "ini"u"(



Annealin! " Bolt#mann La$ %

Annealing+-If a "etal is raised to a te"!erature

a&o*e "elting !oint /the ato"s are in *iolent rando"

"otion( The ato"s al#a's tend to reach a "ini"u"

energ' state( As the "etal is graduall' cooled theato"s enters a "ini"u" !ossi&le energ' state

corres!onds to each te"!erature(

)/(

ep)( kT e

e P

−

α

4e56!ro&a&ilit' that the s'ste" is in a state #ith

energ' e(/$ ,olt1"ann7s constant( T 8te"!erature(



Simulate Annealin! "Bolt#mann Train!%

Define a *aria&le T that re!resents an artificial

te"!erature( 4Start #ith T at large *alue5(

A!!l' a set of in!ut to the net#or$/ and calculate

the out!uts and o&0ecti*e function(

Ma$e a rando" change #eight and recalculate the

net#or$ out!ut(

Calculate ne# o&0ecti*e function(



If the o&0ecti*e function is reduced/ retain the

#eight change(

If the #eight change results in an increase in

#eight change /calculate the !ro&a&ilit' of acce!ting

the #eight change(

)/(ep)( kT cc P

−

α

4c56!ro&a&ilit' of a change of c in the o&0ecti*e

function/$ ,olt1"ann7s constant( T 8te"!erature(



Select a rando" nu"&er r fro" a unifor"

distri&ution &et#een 1ero and one(

If !4c5 is greater than r/ retain the change other#ise

return the #eight to !re*ious *alue(

This allo#s the s'ste" to ta$e a ste! in a

direction that #orsen the o&0ecti*efunction/hence esca!es fro" local

"ini"u"(

.e!eat the #eight change !rocess o*er each of the

#eights in the net#or$/ graduall' reducing the

te"!erature T untill an acce!ta&l' lo# *alue for

o&0ecti*e function is o&tained(



'o$ to elect $ei!ht)arti*icial Temperature *or trainin!

The si1e of the rando" #eight change is selected &'

*arious "ethods(

Eg+-

4#56ro&a&ilit' of a #eight change of si1e #(

)/ep()( !!T w P −=ω

T6artificial te"!erature

To achie*e glo&al "ini"u" at the earliest thecooling rate is usuall' e3!ressed as follo#s

))"(log()( #

t

T t T

+

=



9The "ain disad*antage of ,olt1"ann7s training is *er'

lo# cooling rate and hence long co"!utations(

9,olt1"ann7s "achine usuall' ta$es i"!ractical ti"e

for training(



Cauch+ Trainin!

Cauch' training "ethod is "ore ra!id than

,olt1"ann training(

Cauch' training su&stitutes cauch'7s distri&ution for

,olt1"ann:s distri&ution(

Caush'7s distri&ution has longer ;tails</ hence "ore

!ro&a&ilit' for larger ste! si1e(

The te"!erature reduction rate is changed to

in*erse linear( 4%or ,olt1"ann training it #as in*erse

logarith"ic(5















































Documents

Statistical methods in ANN