Training Artificial Neural Networks for Fuzzy Logic

8/3/2019 Training Artificial Neural Networks for Fuzzy Logic

1/15

Comp lex Sy st ems 6 (1992) 443-457

Training A rtificial Neura l Net work s for Fuz zy Logic

Ab hay B ulsari *

K emisk-tekniska f akulteten , Abo Akade mi ,

SF 20500 Turku jAbo, Fin land

Abstract. P roblems r equiring infe rencing with Bo olean log ic havebeen im plemented in p erceptron s or feedforward n etworks, and someatte m pts have been mad e t o imp lement fuzzy logic based inferencin g insimilar n etworks. In th i s paper , we pr esent produ ctive networks , whichare a rt ificial n eural ne tworks , meant for fu zzy logic based infer encing.

Th e nod es in t hese netwo rk s collect a n offset product of th e input s,fur th er offse t by a bias. A m eaning ca n be ass ign ed t o ea ch n od e insuch a n etwork , s ince t he offsets must be e ither - 1 , 0, or l .

E arli er , it was s hown t hat fuzzy logic inferencing co uld b e performed in produ cti ve networks by m anually s et t ing t he offsets . Thi spr ocedur e , how ever , en countered c riticism, since t here is a fee lin g t hatneura l networks s hould i nvolve tr aining. We describ e an algo rit hm fortr aining pro duct ive n etworks from a s et of tra ining in stances. Un likefeedfo rwar d neur a l n etwo rks with sigmoida l ne urons , th ese netwo rk sca n be t rained wit h a sma ll numb er of t raining in stan ces .

T he t hree main logica l op erations t ha t form t he ba sis of inf erencing- NO T , OR , and A ND - c a n be impl em ented ea sily in pro du ctivenetw ork s. T he networks der ive th eir nam e from t he way t he offsetprod uct of input s for ms t he act ivatio n of a nod e.

1. In t rodu c t i on

Prob lem s requi r ing inferen c in g w ith Bo o lean logi c h ave b een impl em ent edin p er ce p t r on s or fe edforwa rd n etworks [1], a nd som e at t em pts h ave b ee nmad e to impl em ent fu zzy logic b ase d inferencin g in si m ilar n etwo rk s [2] .How ever , feedf or w ard n eur al networks with s igmoi da l act ivat ion fun ct ionsca n not ac c urate ly eva luate fu zzy logic ex pr ess ions u s in g th e T-n orm (seesec t ion 2.1) . Th erefore , a n eur a l netwo rk a rc h itect ur e was prop osed [3] inwh ich th e elem entar y fu zzy logic op er ation s cou ld b e p erformed acc ur a te ly.(For a g oo d ov erv iew of fuzz y logic, see [4, 5].) A n eur a l n etwork ar ch it ectur ewas d es ir ed for w h ich t he te d ious tas k of t ra in ing could b e av oided, a nd

*On leave from L appe enrant a University of Technology. Electronic m ail addr ess:abul sari@bo . f i


2/15

444 A bh ay Bul sari

in which eac h no de carr ied a speci fic m eanin g . Produ cti ve networks , ast he n d efined , offered m any a dva ntages (as describ ed in [3]). I t was s how nt hat fuzzy logic infer en cing c ould be performed in p rodu cti ve networks bym an ua lly set ting t he offsets . Th i s pro cedure , however , encou nt ered crit icism ,

since th ere is a feelin g t hat ne u ra l networ ks should involved t ra ining.W ith m inor mo dificat ion - nam ely, th e addi t ion of t he di sconn ect in goffset - it is now poss ibl e to b egi n with a networ k of a s ize as large or largert han requir ed , and t ra in i t wit h a few t raini ng inst ances . Becaus e of t hena t ur e of prod uct ive n etworks , a sma ll numb er of t raining inst ances sufficesto t ra in a netwo rk wit h m any more para m eters . Th e paramet er s mu st t a ket he v alues - 1 , 0, or l .

Th ese mo dified pr od u ctive network s ret a in m ost of t he useful fea t ur esof t he previou s design. I t is st ill po ssible to m anu ally se t th e offsets for

well-d efined pr oblems o f fuzzy logic . E ach n ode st ill ca rr ies a m ea ning int he sa me m an n er as befor e. Th e network s are v ery simil ar t o feedforwa rdneural net works in st ruct ure. H owever , ex tr a conn ections ar e permi ssible,whi ch was not th e cas e pr eviou sly . A pri ce has be en pa id for t his in creasedflexibility , in the form of a mor e compli ca t ed ca lcu lation of t he net input toa nod e.

2 . The basic f uzzy logica l ope rations

T here is inc reasing int er est in t he use of fu zzy logic a nd fu zzy se ts , for var iousapplica t ions. Fu zzy logic m ak es it p oss ibl e to h ave s had es of grey betwee n t het rut h values of (false) and 1 (t rue ) . St a t emen ts s uch as "the temperat ur eis h igh " need no t have cr isp t ru t h values , and t his flexibility has p ermitt edt he dev elopment o f a wid e ran ge of app licati ons, from consume r pr odu ct s toth e co ntro l of heavy m achin ery. Fu zzy ex pe rt systems ar e expert syste msth at use fuzzy logic bas ed infere ncing.

Almo st a ll logic al op erat ion s ca n b e repr esente d as c ombin at ions of N OT(rv) , OR (V) , and AN D (1\) op eratio ns . I f th e t ru t h va lue of A is repr esented

as t (A ), t he n we shall ass um e t ha t

t( rvA) = 1 - t (A )t(A V B ) = t( A) + t (B ) - t (A ) t(B)t(A 1\ B ) = t (A ) t( B )

Th e OR equation shown above c an be m odified to a mor e suita ble f ormas follow s:

A V B = rv(rvA 1\ rvB )t(A V B) = 1 - (1 - t(A) ) ( l - t( B ))

whi ch is equiv alent to t he equat ion s hown above, but is in a mor e usefulform . S imilarly , for thr ee o perand s , on e ca n write

t (A 1\ B 1\ C ) = t(A )t( B) t(C )t (A V B V C) = 1 - (1 - t( A))( l - t(B ))( l - t (C ))


3/15

Train ing Ar tificial Neural Networks for Fu zzy Logic 44 5

Since Boolea n logic is a s pec ial case o f fuzzy logic (in which t rut h va lues a reeithe r 0 o r 1) , p rodu ct ive networks ca n b e used for Boo lean logic as we ll.

2.1 U nfit ness of th e sigm oid

We have yet to ex pla in w hy fuzzy logic ca nnot be impl eme nted in feedfo rwardnetworks wit h sigmo ida l activat ion func t ion s. Su ch n etworks have a s moot ht ransition f rom t he "yes" to t he "no" state , a nd are sa id to have a g racefu ldegrad at ion.

A /\ B ca n b e p erform ed in a feedfor wa rd neur a l network by cr (t(A) +t (B) - 1.5 ) , whe re a is the s igmo id fun ct ion from 0 to 1. Th i s pro cedure hastw o m a jo r li m it ati on s. T he t rut h value of A /\ B is a fu ncti on o f t he sum oft heir in div idu al t rut h va lues, which is far from t he result of equat ions givenabove. T his t rut h va lue is almo st zero until th eir sum ap proach es 1.5, a ndafte r th at it is a lmost one . T he width of th is tr ansition can b e ad just ed , butt he cha ra cte r of t he fu nct ion rema ins t he sa me . I f t (A ) = 1 and t( B ) = 0,t( A /\ B ) is not exact ly zero .

Ano t her ob ject ion to t he use of th e s igmo id is more se rious . Us ingcr (t (A ) + t(B ) - 1.5) to ca lcu late t(A /\ B ) yields t he foll owing resul t :

t((A /\ B ) /\ C) # t (A /\ (B /\ C ))

3 . P rodu cti ve n et work s

A p roduct ive net work , as d efined here , no long er imp oses a limi t on t he n umber of con nect ions , as do feedfor ward ne tworks. Th e extraneo us connect ionsdo no t m at t er s ince t heir weight s ca n b e se t t o zero. E ach no de pl ays am ean in gfu l ro le in th ese networ ks . I t collects an offset pr od uc t of inpu ts ,furt her o ffset b y a bias. For exa m ple , the act ivat ion of th e no de s hown infigur e 1 ca n b e writ t en as

when the offset s a re 0 or 1. I f an offset is - 1 , it effectively d isconnects t helink . In ge ne ra l,

a = Wo -IpW-Xj )[1 +~Wj ( l- Wj )11T hu s, if a n offset is - 1, it on ly mul t iplies t he produ ct by 1. Th e ou t put oft he nod e is th e a bsolu te va lu e of t he ac t ivat ion , a.

y = la l

T he nond ifferentia bili ty o f t he ac t iva t ion func tion is not a pro blem . ev

ert heless, if d esi re d , t he ac tivat ion funct ion ca n be mad e cont inuo us by


4/15

446

y

D "iJ nodeo f f s e t ~

x3

Fi gu r e 1: A no d e in a producti ve neur al network.

y

Abh ay Bul set i

node

Fi gure 2: Inv er ti n g a n inpu t in a produc tive network

re placing it with a produ ct of t he argument a nd t he - 1 to 1 sigmoid, wi th alarge gain, (3:

y = a ( - 1 + 2 -1 -+ - e x -~---;-(-- (3----=-a--:-)

Th e offset , W j, shifts t he v alue of th e input by some amount , usu ally 0 or1. Th e input remains un affected wh en th e offset i s 0, a nd a logical i nverse(negation) is t aken w hen t he offset i s 1. In addition t o the offset input s, th ereis a bi as , Wo , which furth er offsets th e produ ct o f th e offset inpu t s.

Th e productiv e network has se veral nod es with one or m o r e input s (seefigures 6 and 8). Th e input s should b e p ositive numb ers j ; 1. Each o f th enod es has a bi as , alte rna t ively c alled th e offset of th e nod e. A bi as of 0 or- 1 h as th e same e ffect - a nod e offset of zero i s the sa me as n ot I having anod e offset . Th e output is a p ositive numb er :::; 1. Produ ctiv e networks a reso n amed b ecause o f t he mul tiplication of input s at each node.

4. Implementat ion of the basic fuzzy logical operations

To show th at on e can represent a ny co mplicated fu zzy logi c op erat ion i nprodu ctive networks , it suffices t o s how t hat t he th ree b asic op erations canb e implemented in thi s framework.

Th e s implest op eration , NOT , requires a n offset o nly , which ca n b e provided b y th e inpu t link (as s hown in fi gur e 2 ) . Altern ati vely , t his off set ca nb e provided b y th e bi as instead of t he link, with th e sa me r esult.


5/15

Tr ainin g Artificial N eur al Netwo rk s for Fu zzy Logi c

x3

Figur e 3: Appl ying AND to t hree inpu t s in a produ ctiv e network

y

Figure 4 : App lying OR to t hree inpu ts in a produ ct ive network

447

AN D is a lso im plemented in a f acil e m ann er in t his fram ework. It needson ly th e p rod uct of t he trut h values of its a rg um ent s; hence , ne ith er t he link snor th e bi as h ave offsets (as show n in figur e 3, for thr ee inputs ).

On t he ot her h an d , OR needs o ffs et s on all t he in put links , as w ell as ont he bi as ( see figure 4 ). Th e offsets f or OR and A ND indic ate th at th ey aretw o extrem es of a n op erat ion , which would h ave off sets betwee n 0 and 1. Inot her wor ds, o ne c an p erfo rm a 0.75 AN D a nd a 0.25 OR of two op erands Aa nd B by set ting Wo = 0.25 , W I = 0.25 , a nd W 2 = 0.2 5.

F igur e 5 shows h ow ~ A V B can b e imp lemented . Thi s is equi valen t toA impli es B (A * B ).

I f th e fun ct ions clar it y (A) a nd fu zzines s (A) are defin ed as

clari ty(A ) = 1 - fuzzine ss(A)fuzzine ss(A) = 4 x t(A 1\ ~ A ) ,

t hey can th en b e calcu la t ed in t his fram ewor k.

Figure 5: ~ XI V X2


6/15

448

A B

Figur e 6: Network configuration for A XOR B

A I \ B

in puts

Abh ay Bul sari

5 . I llus t ra t ions of th e manual set t ing of offsets

Appl yin g AN D an d OR op erat ions over sev era l inpu ts ca n b e p er formed b y asingl e node. I f som e of th e inputs must b e inver t ed , t his can b e acc omp lishe dby cha ng ing t he offset of t he pa rt icul ar link. T hu s , no t on ly can a s ing le no dep erform A I\E I\ C and A v E V C , but a lso A I\ E I \ ~ C (which wou ld r equir eWo = 0, W I = 0, Wz = 0, and W 3 = 1.) However, oper at ions t hat requir ebr ack ets for express ion- for exa mpl e, (A V E) 1\ C - r equire more t han o nenod e.

An exclusi ve O R appl ied two v ar iables - A XOR E - ca n also be wr itt enas (A V E ) 1\ ~ ( A1\ E ). E ach of t he brac ket ed e xp re ssions r equir es one n od e,with on e m ore nod e requir ed to p erform AN D b etween th em ( see fig ur e 6). (AXO R E ) ca n also b e wr itt en as (A v E ) I \ ( ~ AV ~E ) , or ( AI \ ~E ) V ( ~ AI \ E ) ,eac h of w hich could r esult in differ ent config urat ions.

A program , SNE , was deve lop ed to eva luate th e outp uts of a prod uctiv enetwork ; an outp ut for t his XO R prob lem is shown in App endix A . Feedfor ward neur al network st udies oft en b egin w ith t his p rob lem , fitting fourp oints w it h ni ne weights by backpr opagation . (Produ ctive networks a lso requir e nine parameters , but fit t he ent ire ra nge o f t ruth values between 0and 1.) F igur e 7 shows t he XOR va lues for f uzzy argum ents. t (A ) increas esfrom left t o right , t(E) in creases from t op to botto m , and t he valu es in t hefigur e a re te n ti mes th e ro unded value of t(A XOR E ).

In [1], a s m all se t of rules w as prese nted , designed t o govern t he selectionof t he type and m ode of o peration of a c hemi ca l reactor c ar r ying out a s ingl ehomog ene ous reac ti on . As de fined , th ere are regions in t he sta te sp ace (forex amp le, betw een rO/ r l = 1.25 and 1.6 ) wher e no ne of t he ru les m ay appl y.In fact , thes e were meant t o be fu zzy region s , to b e filled in a s ubsequentwork s uch as th is one. T he set of rul es has th erefore b een s lightly mo dified ,and is given b elow. Th ere are two c hoices f or t he ty pe of rea ctor , st irred -ta nk


7/15

Tr aining Art ificial Neural Networ ks for Fu zzy Logi c

0 0 1 1 1 1 2 2 2 2 3 3 3 3 4 4 4 4 5 5 5 5 6 6 6 6 7 7 7 7 8 8 8 8 9 9 99 ** *00 1 11 1 2 2 2 2 3 3 3 3 4 4 4 4 5 5 5 5 66 6 6 6 7 7 7 7 8 9 8 8 9 9 9 9 . *

1 1 1 1 1 2 2 2 2 3 3 3 3 4 4 4 4 4 5 5 5 5 6 6 6 6 6 7 7 7 7 8 a 8 8 8 9 9 99 *1 11 1 2 2 2 2 33 3 3 3 4 4 4 4 s s s s s e e e e e7 7 7 7 7 s a e a a 9 9 9 9

11 1 2 2 2 2 3 3 3 3 3 4 4 4 4 4 s s s s s e e e e e 7 7 7 7 7 s e a a e S 9 9 91 1 2 2 2 2 3 3 3 33 4 4 4 4 4 s s s s s s e e e e e 7 77 7 77 a a a a a S9 92 2 2 2 2 3 3 3 33 3 4 4 4 4 4 5 5 5 5 5 5 6 6 6 6 6 7 7 7 7 7 7 7 8 8 a a a 8 922 2 2 3 3 3 3 3 3 4 4 4 4 4 s s s s s s e e S 6 6 S 6 77 7 7 7 7 7 e e e a a e2 2 2 3 3 3 3 3 3 4 4 4 4 4 4 s s s s s s e e e e e e S 7 7 7 7 7 7 7 7 e a a a a22 3 3 3 3 3 3 4 4 4 4 4 4 S s s s s s s e e e e e e e e 77 7 7 7 7 7 7 7 a a e3 3 3 3 3 3 3 4 4 4 4 4 4 S S s s s s s s e e e e e e e S S 7 7 7 7 7 7 7 7 7 7 83 3 33 3 4 4 4 4 44 4 S S S 56 S S S S 6e S e S6 6 G e 6 7 77 7 77 7 7 7 73 3 3 3 4 4 44 4 4 4 S S 5 s s s s s s e s e e e S S 6 G G e 6 7 7 7 7 7 7 7 7 73 3 4 4 4 44 4 4 4SSS S S S S S S S 6 a s e e e e e e G 6 S 6 6 77 7 77 7 74 4 4 4 4 4 4 4 4 5 5 5 5 5 5 5 5 5 5 5 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 7

4 4 4 4 4 4 4 5 5 5 5 5S G G S S S 5 6 6 6 6 6 6 e 6 e s e 6 6 6 6 6 e e e e s e

4 4 4 4 4 S 5 5 5 5 5 5 5 5 5 5 5 5 5 6 6 6 6 e S S SS 6 6 6 6 S 6 6 6 6 6 6 6 S

4 4 4 5 S 5 S S S S S S 5 5 S S S S 6 6 6 S 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 655 5 5 S 5 S 5 S 5 5 5 5 5 S S 5 e 6 6 6 6 6 e 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6

5 5 5 S S S S S S S S S S S S 6 6 6 6 6 6 6 6 6 6 6 6 6 e e e e e 6 5 S 5 5 5 5 5

5 S S S S S S 5 S S 5 S 6 6 6 e e e e e e e e e e e e e e S 5 S 5 S S S S S S SS55 5 5 5 s s e 6 e e e e 6 6 s e e e e e e e e e e S S S S5 5 5 S S S S S S S Ss e e s e e e e e 6 6 6 e 6 e s s e 6 e e 6 e 6 S S S S S S5 5 5 S S S S S SS S6 6 6 6 6 6 6 6 6 6 6 6 6 e S S 6 6 6 6 6 6 6 S S 5 5 5 S 5 5 5 5 5 5 5 S 5 4 4 4S6 6 6 6 s e 6 6 e S 6 6 6 e e e e 6 6 s e 5 5 5 S 5 S 5 S 5 S 5 5 S S 4 4 4 4 4

6 6 6 e e s e e e e e S 6 e e e e e e 6 6 6 S S S S S 5 5 5 S SS S 4 4 4 4 4 4 4

7 e e e e S 6 e e e e s e 6 e s e 6 6 6 6 S 5 5 S S S 5 5 S 5S 4 4 4 4 4 4 4 4 47 7 7 7 7 7 7 s e e e e e e s e e e e e e 5 S S 5 S S 5 S S S 4 4 4 4 4 4 4 4 3 3

7 7 7 7 7 7 7 7 7 6 6 6 6 6 6 6 6 6 e 6 6 5 5 5 5 5 5 5 5 5 4 4 4 4 4 4 4 3 3 3 37 7 7 7 7 7 7 7 7 7 6 6 e e e e 6 e 6 6 S S S 5 5 S 5 SS 4 4 4 4 4 4 4 3 3 3 3 3

8 7 7 7 7 7 7 77 7 7 6 6 6 6 e e 6 6 6 5 5 5 5 5 5 5 5 4 4 4 4 4 4 3 3 3 3 3 3 3

a a S 77 7 7 7 7 7 7 7 e e e s e s s e 5 S 5 5 5 S S 4 4 4 4 4 4 3 3 3 3 3 3 2 2a a a a a 7 7 7 7 7 7 7 7 6 e e e s s e 5 5 S S 5 S 4 4 4 4 4 4 3 3 3 3 3 3 2 2 2a a a a a a 7 7 7 7 7 7 7 6 6 e e e e e S S S S S S 4 4 4 4 4 3 3 3 3 3 3 2 2 2 29 a a a a e a 7 7 7 7 7 7 7 6 e e e 6 5 5 5 5 5 5 4 4 4 4 4 3 3 3 3 3 3 2 2 2 2 2

9 9 a a a a a S 7 7 7 7 7 7 e e e 6 6 5 S S 5 5 5 4 4 4 4 4 3 3 3 3 3 2 2 2 2 1 1

9 g 9 S e e e S S 7 7 7 7 7 e e e e 6 5 5 5 S 5 4 4 4 4 4 3 3 3 3 3 2 2 2 2 11 19 g 9 9 S e 8 8 8 7 7 7 7 7 6 6 6 6 6 5 5 5 5 5 4 4 4 4 3 3 3 3 3 2 2 2 21 1 11

* g 9 9 9 a 8 a a a 7 7 7 7 6 e e 6 6 5 5 5 5 4 4 4 4 4 3 3 3 3 2 2 2 2 1 1 1 1 1** 9 9 9 9 8 a a 8 7 7 7 7 6 6 S 6 6 5 5 5 5 4 4 4 4 3 3 3 3 2 2 2 2 1 1 1 1 0 0** * 9 9 9 g e 8 8 8 7 7 7 7 6 e 6 6 5 5 5 5 4 4 4 4 3 3 3 3 2 2 2 2 1 1 1 1 0 0

Figur e 7: A XOR B

449

or t ubul ar ; and two cho ices for t he mo de of ope ration , cont inuous or bat ch.In t he followin g rul es these choi ces are assume d to b e mu tu ally e xclusi ve;t hat is , t he sum of t heir trut h valu es is 1.

1. I f th e reac t ion i s highly exot herm ic or hi ghl y e nd ot her mi c - say 15kCalj gm mol ( else we call it a thermal) - s elect a stirr ed-t ank re actor.

2. I f t he r eacti on m ass is highly v iscous (say 50 cent ipoise or mo re) , se lecta s t irred -t ank reacto r.

3. I f t he rea ctor ty pe is t ubu la r, t he mo de of op eratio n is cont inuous .

4. I f rO frl < 1.6 , prefer a continuous st irred tan k reacto r.

5 . I f rO frl > 1.6 , th e reacti on m ass is not v ery visco us, and t he reactio nis quit e at hermal, pr efer a tubular reactor.

6. I f rOfrl > 1.6 , t he react ion m ass is not v ery visco us, bu t t he react ionis no t at he rrnal , pr efe r a s tirred - tan k reac tor opera ted in b at ch mode.

7. I f rO frl > 1.6 , t he react ion m ass is quit e visco us , and t he r eact ion isat herrn al, pr efer a s t irred- ta nk rea ctor o per at ed in ba tch mo de.

8 . I f th e production r at e is very high com pared to t he r ate o f reactionr l (say 12 m 3 or m ore) , and rO fr l > 5, pr efer a s t irr ed -ta nk r eac tor

operated in batch mode.


8/15

450 Ab hay Bul sari

9. I f t he prod uct ion rate is very high com pa red to t he rat e of react ionr l (say 12 m3 or mor e) , an d r O/r l < 5, pr efer a st irre d-ta nk re acto roperat ed in cont inuous mod e.

10. I f r O/ r l ~ 1.6 , t he rea ction m ass is qui t e viscous, a nd t he re act ion isnot a t hermal , prefer a s t irred -ta nk r eactor op erated in batch mo de.

ro is t he r ate o f react ion und er inl et conditio ns , and rl is t he r at e of react ionund er exit co nd itio ns. I f t hei r ratio is large , a plu g-flow (t ubu la r ) reactorrequir es s ign ifican tly l ess v olu me t ha n a st irre d-t ank r eacto r op erat e d continu ou sly. A st irr ed- ta nk r eactor oper ate d in ba t ch mode is s im ilar t o aplug-flo w reactor w hen t he length coordi na te o f t he t ubu la r rea ctor resembles ti m e in a bat ch r ea ctor. Th e a im of [1] was t o invest igat e t he feas ib ilit y ofimpl ementin g a fuzzy se lecti on ex p ert sys tem in a produ ctive neura l n etwork ;

henc e, t he heuri st ics enum erat e d ab ove a re typical. T he y a re not n ecessa ri lyt he b est set o f ru les for s elect ing r eactors for single ho mogen eous r eact ion s ;neit her are t hey c om ple te .

F igure 8 s hows t he i mpl em entatio n of t hese heuristics in a pro du cti ve network. I t is mu ch s imp ler t han a feedfo rward n eur al networ k ; it req uir es notr ain in g, and does not have to o m an y connect ions (weight s) . T he offset s a reeit her 0 or 1, un like t he weights (wh ich ca n have a ny value b etween - 0 0 and(0). Of cours e , it a lso ca lculates t he fuzzy t ru t h values for t he select ion ofty p e a nd m od e of o per at ion of a chemical reactor. I t is a litt le mo re reliabl e

since t he function of eac h of t he n od es in t he network is known an d u nd erst ood , and t here is no q uest ion of t he sufficiency o f t he numb er of t ra in inginst ances . I t m ay not b e po ss ib le t o rep resen t eve ry expr ess ion in tw o laye rs .Havin g several l ayer s , however , is no t prob lem atic for pro ductiv e network s,

Jt hou gh it does ca use di fficu lty in tra inin g feedforwa rd neura l networks.

It may be reca lled t hat t he inpu ts to produ ct ive networks a re t rut h val uesb etween 0 and 1. Th e five inpu ts to t he n etwork s hown in figur e 8 a re

A t(rO / rl < 1.6)B t (p, < 50)C t ( I ~ Hrx n l < 15)D t (r O r l < 5)E t( F / rl < 12)

Th ese t r uth values can be ca lculate d (using a ram p or a s igmoid) based oncr it er ia for t he wid th of fuzz iness. ( For exa mp le, for A , t( r O r l < 1.2) = 1and t(rO / r l > 2.0) = 0, wit h a linear i nt erpol at ion in b etwee n.) App end ix Bsho ws result s of th is sys t em w ith clear in pu ts ( t r ut h va lues of 0 or 1) . Forconfirm at ion , t hese r esul t s were fed in t o an indu ctive learning progra m . T hispro gram was able to e licit t he h euris tics for s elec ting a st irred -t an k for co nt inuou s oper at ion - A V ev B V evC V E) and (A V (B 1\ C ) V (D 1\ E ). T his,in effect, was th e s ame as (A V (evA 1\ B 1\ C) V (D 1\ E )) , imp lement ed int he network di rectly from t he heuris t ics. I t was a lso confi rme d th at t he imp lausib le se lect ion of a t ubu lar reacto r op erat ed in batc h m od e never too kpl ace.


9/15

Tr aining A r tificial Neural Net works f or Fu zzy Logic 45 1

A B c D B

Figure 8: Fuzzy -select ion expert syste m in a produ ctiv e netwo rk

6 . Limi t a t ion s of pr odu ctiv e n et work s lackin g th e disconn e ctingoff se t

I f t he input s a nd o utpu ts of a fuz zy logical oper at ion a re given , and if onewants a produ ct ive n etwork to learn t he corre lation , it is a lmost impo ssib lewit hout t he d isco nnectin g offset . A pr od uct ive n etwork witho ut t he d isconnect ing offset (- 1) cannot b e eas ily tr a ined. Th ere i s no way to swit ch offan ou t pu t f rom a nod e th at is connect ed . On e mu st d ecid e t he conn ectiv ityb efor ehand - w hich can b e, at best , good g uesswork.

Th e pr odu ctive network is in tend ed prim ar ily for represen ting fu zzy logical op erations , a nd ca n of cours e do Bo olean logic. Th at , however , is it slimi t. It has hardly a ny ot her app licat ion.

7 . Illu s t r a t ions of t ra ining produ cti ve n etwork s

T he tr ainin g of n eura l network s is in tended to redu ce t he sum of th e squaresof erro rs (SSQ) to a minim um , wh ere t he erro rs ar e th e diff erenc es b etweenthe desire d and t he ac t ua l neura l network out put s. In feedforwa rd neura l

netwo rk s, it is su fficient to m inimiz e th e SSQ ; in pro du ctive networks , however , it sho uld b e reduc ed t o ze ro wh en th e tr ainin g inst a nces a re kn own tob e accurate .

Th ere a re a fini te n umber of po ss ibili t ies for t he weight vecto r. Thi s isp resu ma bly an NP -com pl ete probl em. For a netwo rk with N weights , t her eare 3N p ossibilities. I t is t herefore cl ear t hat sea rching t he ent ire sp a ce is notfeas ib le when N excee ds 7 o r 8. To so lve t his pro blem w e present a si mp l ealgorithm in t he n ext s ubsection. Tr aining is p erform ed by mod ify ing t heoffsets in a discrete m an n er (vary ing a mo ng - 1 , 0 , and 1). Th e esse nce of

this algor ith m is sim ilar t o th e Hook e a nd J eeves method [6].


10/15

452 A bha y Bulsari

Sinc e t here i s a disco nn ect ing offset (-1) , it is p ermiss ible to have m oreno des t ha n n ecessary. T here is no o ver -p aramet rization i n t hese n etworks ;but a n over-eval uat ion of fuzzy ex pressio ns is po ssible wh en t he re are ext ranod es, a nd t h is ca n resul t in wro ng ou tp uts. Th e exac t numb er of nodes

ca n be dete rmin ed if a comp lete set of t ra in ing inst ances (a ll '2fVin cases) isavaila ble; bu t t his r equires t he a ppro ach of t he pr eviou s work , which h as b eenst rong ly criti cized. Despi t e th e fact th at man y res pec ta ble p eop le in th e fieldof ne ura l n etwork s think th at tr ainin g is n ecessa ry, or t hat "a network i s aneura l n etwor k on ly if i t ca n b e t rained, " we m ain t ain t hat t his is not t herigh t way to co nst ruct p rodu ctive netwo rk s.

7.1 T h e training algorithm

Hook e a nd J eeves [ 6] pro p ose d a zero -o rder opt im izat ion m et hod , w hich looksfor t he d irecti on in whi ch th e obj ect ive funct ion im pr oves by vary ing o nep aram ete r at a t ime. D eriva tives ar e no t re qu ire d. W hen no im provement isp ossible , t he ste p size is redu ced. In t he prob lem und er consider at ion here ,each p aram et er (offs et ) must t ake on e of only thr ee va lues . Henc e, wh ena param et er is b eing co nsid ered f or it s effect on th e obj ect ive fun ction, th eth ree cas es a re co m pa red , a nd t he on e wit h th e lowest SSQ is chos en.

P arame t ers t o b e conside red for t he ir effect o n t he SSQ can b e chose nse que ntia lly or r andoml y. Wi th a se que nt ial cons ide rat ion of par ameters , t he

sea rch pro cess o fte n fal ls in to cycles t hat are di fficul t to id entify, sto pp ingon ly at t he limi t of th e m aximum num ber of it er atio ns. W ith a rand omconsi de rat ion of p arameters , t he algor ithm ta kes on a stochas tic nat ur e a ndis very slow, bu t does no t fa ll in t o such cycles. N ever th eless , t he sequenti al consid erat ion i s found t o b e pr efera bl e for t he k ind s of prob lems to b econsid ered in sec t io ns 7.2 and 7 .3.

T ypica l output s of a progr am (SNT) imp lementing t his algorithm arepr esent ed i n App end ices C and D . Th e a lgorit hm i s fast a nd r eli abl e. I trun s into local mini ma at t im es (lik e an y op t imi za t ion algorit hm would on a

problem w ith se veral mi nim a ) , but since i t is relat ive ly fast , it is easy to t rydifferen t ini t ial guesses . Beca use of t he network a rch itect ure , it is po ssib let o in terpret int ermed iat e resul ts , eve n, pr esum ably, at every it erat ion. Th ealgor ithm worked q ui te we ll with t he prob lems c onsidered in t his p aper.

7. 2 T h e X OR prob lem

Th e XOR probl em es sentia lly cons ists of th e t ra in ing o f a pro ducti ve networkto p erform fuzzy XOR op erat ion on tw o in put s . Wi th eight t raining in st ancesit was p ossibl e to train (2 , 2,1 ) and (2 , 3 ,1) networks e as ily, while (2 ,1 ) and(2,1 , 1) n etwork s could n ot learn t he op erat ion. Ap p endix C p resents t heresul t s of t ra ining a (2 ,3 , 1) networ k.

I t is easy to inter pr et t he r esult s of tra ini ng t hese n etwo rk s. T he (2, 1)ne t wor k lea rn ed A A E fro m t he eig ht t ra ining in stances. Th e SSQ w as1.038. Th e (2, 1, 1) n etwo rk learn ed ~ A A E , resul t ing in t he sa me S SQ ,1.038. Th ese results a re dep end en t on t he ini tia l guesses (to t he extent t hat


11/15

Training Ar tificial N eu ral Networ ks for Fu zzy Logi c 453

a diff erent logica l ex pression m ay result from a diff erent ini ti al g uess), but

(2, 1) a nd (2, 1, 1) wou ld n ot have a n SSQ of less t ha n 1.0 38. Th e (2 ,2 , 1)network learn ed it p erfectl y, as (A V B ) 1\ ("-'A V ,,-, B) . Th e (2,3, 1) networkonce ran into a local min imum , wit h SSQ = 1.038 , exp re ss ing (,,-,B) 1\ ("-'A 1\"-'B ) 1\ ("-'B ). As shown in App endix C , th is network ca n also learn (A VB ) 1\ ("-'A V ,,-, B) , ignor ing one of th e no d es in t he hidd en laye r , which was"-'(A 1\ B ).

Wi th four t ra ining in sta nce s (all clear inp u ts a nd ou tp ut s) , the sa menetworks wer e agai n trained by th e algor ithm descr ib ed a bove. Unfortunate ly, t here is no w ay to pr eve nt th e networ k from learning (A V B) as(A 1\ ,,-, B) V ("-'A 1\ B ) V (A 1\ B) V B. A is equ iva lent to as A V A or A 1\ Ain Boo lean logic , bu t not in fu zzy logic , given t he way we have ca lcu late dcon junc t ions a nd di sjunct ion s. T herefore , if on e sta rts wit h cl ear t ra ininginstan ces (n o fuzz y inpu ts or o utp ut s) , t he network m ay learn on e of th eex pressio ns t hat is equ iva lent in B oolean logic . Fort u nat ely, however , t herewas a te ndency to leave hidd en nod es un u sed . T he (2 ,2 , 1) network learn edt he XOR fun ction as ,,-,( A V ,,-, B) V (A 1\ ,,-, B) . Th e (2, 5, 1) n etwork a lsolea rn ed it co rrectly as ("-'A 1\ B ) V "-'("-'A V B ) witho u t adding ext ra neousfeatur es from th e hidd en laya r (leaving 3 nod es unu sed) .

7 .3 Th e c he m ica l r eac tor s e lect io n pr obl em

Th e chemical r eac to r se lect ion h eur istics listed in sect ion 5 ca n a lso b e ta ughtto p rodu ct ive networks from th e 32 cl ear tr aining inst an ces (see App endix B).The (5 ,5 ,2 ) , (5 ,6 ,2) , ( 5, 8, 2) , a nd (5 , 10 ,2) networks wer e a ble to l earn th ecorr ect e xp re ssio ns, w ith an SSQ of 0. 00 . Th e nu mb er of it erat ions r equiredwas b etween 200 and 3 000 , but eac h it erati on took ver y littl e ti m e. Th ety pical run t imes on a mi cro Vax II were b etween 1 and 10 m inut es.

Th e (5,5 , 2) network u sed on ly four hidd en nod es , t he min imum req uir ed .T he (5 ,8 ,2) and (5 ,10,2) networks use d seven and eight nod es, respect ively.T hey typica lly h ad one or two nod es simply du plic at ing the inpu t (or it snegation). Th e res ult s obt a ined with th e (5 ,8 ,2) netwo rk are s hown in Appendi x D .

7 .4 Limitations of the t raining appro ach

In Boo lean l ogi c, (A V B) can a lso be wr itt en a s (A 1\ ,,-,B) V ("-'A 1\ B)V (A 1\B ) v B . A is equivale nt to as A V A or A 1\ A in Boo lean l ogi c, bu t not infu zzy log ic, g iven th e way we have ca lcu late d conjun ctions a nd di sjunctions.Ther efore, if one s tarts wi th clear t rai ning in st ances ( no fu zzy inpu ts o routput s) , th e network may lea rn on e of t he expressions t hat is equiva lent inBoo lean logic.

It is not ea sy to obta in tr a ining inst ances with fu zzy ou tput s . I t is alwayspo ss ibl e t o set th e offsets m anua lly, on ce th e logical ex pression i s kn own.Tr aining is ap pa rently an NP- comp let e pr ob lem. Networks ca nnot be t rained

se quentia lly li ke , as in b ackpropagation.


12/15


13/15

Tra ining Art ificial Neural Networ ks for Fu zzy Logic

layer 2 0 .6602l ayer 1 0 .8125 0 . 1875l a yer 0 0 .2500 0 . 7500

0.3333 0 .3333 0 .4938

l ayer 2 0 .4938l ayer 1 0 .5556 0 .1111

l ayer 0 0 .3333 0.33 33

Appendix B. Resul t s of reactor selection with clear inputs

Number of input and output nodes : 5 2Number of hidden layers : 1Number of nodes in e ach hid de n layer 4Number of offse ts : 20Pr in t option 0/1/2 : 0Offsets taken from f i l e SEL. I N

0 0 0 0 0 1 00 0 0 0 1 1 0

0 0 0 1 0 1 0

0 0 0 1 1 1 1

0 0 1 0 0 1 0

0 0 1 0 1 1 0

0 0 1 1 0 1 0

0 0 1 1 1 1 1

0 1 0 0 0 1 0

0 1 0 0 1 1 00 1 0 1 0 1 0

0 1 0 1 1 1 1

0 1 1 0 0 0 1

0 1 1 0 1 1 1

0 1 1 1 0 0 1

0 1 1 1 1 1 1

1 0 0 0 0 1 1

1 0 0 0 1 1 1

1 0 0 1 0 1 1

1 0 0 1 1 1 1

1 0 1 0 0 1 1

1 0 1 0 1 1 1

1 0 1 1 0 1 1

1 0 1 1 1 1 1

1 1 0 0 0 1 1

1 1 0 0 1 1 1

1 1 0 1 0 1 1

455


14/15

4 56

1

1

1

1

1

1

1

1

1

1

o1

1

1

1

1

oo1

1

1

o1

o1

1

1

1

1

1

1

1

1

1

1

A bha y Bul sari

Appe ndix C. A typica l ou tpu t from S N T

Th i s i s the outp ut from SNT .

Num be r of i nput and outp u t nodes 2 1Num ber of hidden l a yer s : 1Number of nodes in ea ch h id de n l a yer 3Number of off se t s : 13P ri n t opt ion 0/1/2 1The off se t s take n f r om WTS . I N

-1 - 1 -1 -1 -1 - 1 - 1 -1 - 1 - 1 -1 - 1 -1Num be r of i t e r a t io ns : 500

Number of p a t t e r ns in t he in put f i l e 8The off se t s W > 1 to 13

1 0 0 1 1 1 0 0 0 0 -1 0 1

The res idua ls F >> > 1 to 8 :O. OOOOOE+OOO.OOOOOE+OOO.OOOOOE+OOO.OOOOO E+OOO. OOOOO E+OO0 .43750 E-040 .43750 E-04O.OOOOOE+OO

SSQ: 0 .382 81E-08

Appen dix D. A n ou tpu t from SN T for the ch emical reacto rs e lection heuristics

Th i s i s t he outp ut from SNT .

Number of inp u t an d output nodesNum be r of hidden l ayer s : 1Number o f nodes in e ac h h id de n l ayerNumber of off se t s : 66Pr in t option 0/ 1 / 2 : 0Seed f or rand om number genera t i on

8

10201

5 2


15/15

Tr aining Art ificial Ne ural Networks fo r Fu zzy Logic 457

Number of i t e ra t ions : 1000

Numbe r of p a t t e rns in the inpu t f i l e 3 2The of f s e ts W > 1 t o 66

1 1 0 0 -1 -1 0 1 0 -1 -1 0 1 0 0 -11 1 0 -1 -1 -1 0 0 -1 0 0 1 -1 1 - 1 11 -1 - 1 0 0 1 -1 -1 - 1 -1 -1 0 1 -1 -1 01 1 1 0 1 -1 -1 0 -1 1 0 -1 0 1 0 00 0

SSQ O.OOOOOE+ OO

Ref erenc es

[1] A . B . Bulsari and H . Saxe n, "Imple men ta t ion o f Chemica l Reacto r Selec tionExpert System in a Feedfor war d Neura l Networ k ," Proc eedings of th e Au s-tralian Conference on Neura l Netw orks , Sy dney , Au str ali a , (1991) 227 - 229 .

[2] S .-C. Chan an d F .-H . Nah , "Fuzzy N eural Log ic Netwo rk and It s LearningAlgorithms, " P roceedings o f th e 24t h Annual H awaii Int ernational Conf erenceon Syst em Scienc es : Neura l Ne tworks and Related Emerging T echnologies ,Kailu a-K ona , Hawaii , 1 (1991) 476-485.

[3] A . Bul sari a nd H . Saxen , "Fuzzy Logic In fere ncing U sing a Spec iall y DesignedNeura l Netwo rk Ar chi t ectur e," P roce edings of th e Inte rnat iona l Symp osiumon Art ificial In telligen ce App lications and Neura l Networks, Zurich , Swit zerland , (1991) 57- 60.

[4] L. A . Zadeh , "A Th eory of A pp r oximat e R eason ing ," pages 3 67-4 07 in Fu zzySets an d App lica tions , edited by R. R. Yager e t al. (New York , W iley , 1987 ).

[5] L . A. Z ade h , "The Ro le of Fu zzy Logic in th e Man ag em ent of U ncer t ainty inExpert Systems, " Fu zzy Sets an d Sy stems, 11 (1983) 199-227.

[6] R. F letcher , P ra ctic al Met hods of Optim ization . Volume 1, Unconstrain ed Op -t im i zation (Chicheste r, W iley, 1980).

Documents

Training Artificial Neural Networks for Fuzzy Logic