Stability and Attractivity in Associative Memory Networks - Cottrell - En

8/8/2019 Stability and Attractivity in Associative Memory Networks - Cottrell - En

1/11

Biol. Cybern. 58, 129-139 (1988) B i o l o g i c a lC y b e r n e t i c s9 1988

Stabi l i ty and Attractivi ty in Associative Memory NetworksM . C o t t r e l lUA 743 CN RS Statistique Appliqure, Laboratoire de M athrm atique B~t. 425, Universit6 Par~s XI, F-91405 Orsay C edex, France

A b s t r a c t . W e f o c u s o n s t a b l e a n d a t t r a c ti v e s t a t e s i n an e t w o r k h a v i n g t w o - s t a t e n e u r o n - l i k e e l e m e n t s . W ec a l c u l a t e t h e c o n n e c t i o n m a t r i x w h i c h g u a r a n t e e s t h es t a b i li t y a n d t h e s t r o n g e s t a t t r a c t iv i t y o f p m e m o r i z e dp a t t e r n s . W e p r e s e n t a n a n a l y t i c a l e v a l u a t i o n o f t h epa t t e rns ' a t t rac t iv i ty . These resu l t s a re i l lus t ra t ed bys o m e c o m p u t e r s im u l a ti o n s.

1 IntroductionT h e a b i l i t y t o r e c a l l m e m o r i z e d p a t t e r n s i s a v e r yi m p o r t a n t h u m a n f e a t u re . M a n y m o d e l s o f n e u r a ln e t w o r k s i n c l u d e it , a n d t h e c a p a c i t y o f m e m o r y i su s u a l l y s p a t i a l l y d i s t r i b u t e d t h r o u g h o u t t h e n e t w o r k .I t i s exac t ly conta ined in the "e f f ic i enc ies " of synap t i cj u n c t i o n s .T h e s t u d y o f D i s t ri b u t e d A s s o c ia t iv e M e m o r yNetworks was in i t i a t ed in the f i f t i e s . We re fe r e .g . toRosenbla t t (1958) ; Ca ianie l lo (1961) ; Kohonen (1970,1972, 1976) ; Nakano (1972) ; Kohonen e t a l . (1974,1977) e t c . . . .I n t h e s e m o d e l s , t h e s t a t e o f e a c h n e u r o n i sr e p r e s e n t e d b y o u t p u t s p i k e fr e q u e n c ie s . T h e m e m o r i z -a t i o n o f p a t t e r n s r e li e s u p o n c h a n g e s i n t h e s y n a p t i ce f fi ci enc ies acc ord in g to the prese nta t io n of pa t t e rns .T h e f u r t h e r p r e s e n t a t i o n o f a p e r t u r b e d p a t t e r n ( o r apa r t o f i t) l eads to i t s recol l ec t ion . Thi s i s the p rop er tyof select ive recal l .T h i s t h e o r y i s l a r g e ly d e v e l o p e d i n K o h o n e n ( 19 8 4) .T h i s k i n d o f m o d e l s c a n a l s o b e u s e d f o r c o n s t r u c t -i n g s o m e a s s o c i a t i v e m e m o r y w h i c h n e e d n o t b e ar e a li s ti c m o d e l o f n e u r a l n e t w o r k . T h e y c a n b e c o n -c e i v e d f o r m e m o r i z a t i o n a n d r e t ri e v a l o f p a t t e r n s i nanother context , e .g . e r ror cor rec t ions in t rans -m i s s i o n s . I n t h i s c a s e , t h e l i k e - n e u r o n a u t o m a t a m a yh a v e c o n t i n u o u s - o r d i s c re t e -v a l u e d st at es . T h e y m u s tj u s t w o r k a s r e q u ir e d . T h e s i m p l e s t n e t w o r k s w i t h t w o -s t a te t h r e s h o l d " n e u r o n s " w e r e s tu d i e d b y m a n y

auth ors : L i t t l e (1974) , Ho pf ie ld (1982), Pere t to (1984),A m i t e t a l . ( 1 9 8 5 a , b ) , W e i c h b u c h a n d F o g e l m a n -Soul i6 (1985), Pe rson naz e t a l . (1985), e t c . . . .T h e u s e o f c o n c e p t u a l t o o l s o f s t a ti s ti c a l m e c h a n i c s ,espec ia l ly sp in g las ses mo de l s , has a l lowe d ' a go oda d v a n c e i n u n d e r s t a n d i n g t h e i r b e h a v i o r , a n d l e d t oa s y m p t o t i c r es u lt s , w h e n t h e n u m b e r o f u n i t s g r o w s t oinfini ty (Amit e t a l . 1985a, b) .

I n a l l t h e s e p a p e r s , t h e n e t w o r k s a r e u s e d f o rrecogniz ing , i .e . re t r i ev ing a g iven se t of conf igu ra t ionsr e f e r r e d t o a s p a t t e r n s . H o w e v e r t h e s e p a t t e r n s ,w h e t h e r d e t e r m i n i st i c a ll y o r s t o c h a s t ic a l l y c h o s e n , a r en o t a l w a y s a t t r a c t o r s , a n d n o t e v e n s t a b l e s t a t e s , i n then e t w o r k d e f i n e d b y t h e c l a s si c a l c o n n e c t i o n s s u g g e s t e db y H e b b ( 1 94 9 ) a n d a d v o c a t e d b y C o o p e r e t a l . ( 19 7 8) .

O n t h e o t h e r h a n d , t h e s e c o n n e c t i o n s a r e s y m -met r i c , and th i s is an unp leasan t res t r i c t ion , even i f i ta l l o w s t o d e f i n e o n e H a m i l t o n i a n w h o s e l o c a l m i n i m ac o n t a i n t h e p a t t e r n s ( H o p f i e l d 19 8 2; P e r e t t o a n d N i e z1985).A l l c o n t r i b u t i o n s h a v e s t u d i e d e i t h e r a d e t e r m i n -i s t ic a lgor i thm, w i th t emp era tu re T = 0 (Hop f ie ld 1982;P e r s o n n a z e t a l . 1 9 8 5 ; W e i c h b u c h a n d F o g e l m a n -S o u l i 6 1 9 8 5 ) o r a s t o c h a s t ic o n e w i t h t e m p e r a t u r e T > 0(Ami t e t a l . 1985a , b ; Pere t to 1984) .I n t h i s p a p e r w e s h a l l s t u d y t h e d e t e r m i n i s t i c

a l g o r it h m . W e c o n s i d e r a n e t w o r k c o n s i s t i n g o f N t w o -s ta t e uni t s . We def ine a c o n f i g u r a t i o n o r n e t w o r k s t a t ea s a n e l e m e n t S = ( S 1 , $ 2 , . .. ,S N ) o f th e h y p e r c u b e{ - 1 , + 1 } N, w i t h S i = + 1 ( o r - 1 ) i f t h e i -t h u n i t isa c t i v e (o r inac t ive) . T h e c o n f i g u r a t i o n a t t i m e t i sd e n o t e d b y S ~.

T h e c o l l ec t iv e b e h a v i o r o f s u c h a n e t w o r k i se n t ir e l y s p e c if i ed b y t h e s t r e n g t h s o f th e c o n n e c t i o n sC o , b e t w e e n th e s o u r c e j a n d t h e r e c ei v e r i, a n d b y th et h r e s h o l d v a l u e s. T h e N N m a t r i x C = ( C 0 a c t s a s ad e c o d i n g m a c h i n e a n d w i ll b e c a l le d c o n n e c t i o n m a t r i x .E a c h u n i t r e c e i v es i n p u t s f r o m a l l t h e o t h e r sw e i g h t e d b y t h e s t r e n g t h s o f c o n n e c t io n s . L i k e i n


2/11

130n e u r a l n e t w o r k s , t h e s u m o f t h e w e i g h t e d i n p u t sr e p r es e n t s t h e " m e m b r a n e p o t e n t i a l o f th e n e u r o n " : i tb e c o m e s ( o r r e m a i n s ) a c t i v e i f t h i s p o t e n t i a l i s h i g h e rt h a n t h e t h r e s h o l d , a n d i t b e c o m e s ( o r r e m a i n s )i n a c t iv e i f t h i s p o t e n t i a l i s s m a l l e r t h a n t h e t h r e s h o l d .

T h e r e a r e p l e a rn e d , p r e v i o u s ly k n o w n c o n f ig u r -a t i o n s r e f e r r e d t o a s t h e p a t t e r n s , d e n o t e d b y S 1 . . . . S p.W e w i s h , s t a r t i n g f r o m a c o n f i g u r a t i o n S , w h i c hc o n t a i n s s o m e e r r o r s , t o r e t r i e v e o n e o f t h e S " ,m = 1 , . . . , p , w h i c h i s t o b e t h e n e a r e s t o n e .

T h u s i t i s n e c e s s a ry t o d e t e r m i n e h o w t o c h o o s e t h e( C 0 m a t r i x i n o r d e r t o g e t t h e p a t t e r n s a s a t t r a c t i v e a sp o s si b le . M o r e o v e r w e h a v e t o u s e a n i t e r a t io n m o d e ,e i t he r t he se r i a l one , o r t he pa ra l l e l one .

D y n a m i c D e s c r i p t i o nA t t i m e O , t h e i n i t i a l c o n f i g u r a t i o n i n S ~ i s s u e d f r o mo n e o f t h e p a t t e r n s , b u t c o n t a i n i n g s o m e e r r o r s( t r a n s m i s s i o n e r r o r s , m i s c e l l a n e o u s d i s t u r b a n c e s ) .

W e c a l l S t= ( S~ ) t h e c o n f i g u r a t i o n a t t i m e t .At t ime t , t he un i t i r e ce ives t he s i gna l

Ec s +lit ~ 2 ) t h e v a r i a b l e - ~ - - i s e q u a l t o + 1 i f t h e\un i t j i s a c t i ve , 0 i f no t . I f un i t i r e ceives a s i gna l g rea t e r

( r e s p . s m a l l e r ) t h a n t h e t h r e s h o l d , t h e n i t b e c o m e sac t i ve ( r e sp . i nac t i ve ) . We sha l l choose , a s usua l , t het h r e s h o l d O ~= 89 f o r t h i s c h o i c e i m p l i e s t h a t :

J

T h e n t h e t w o t y p e s o f a l g o r i t h m a r e :a ) S e q u e n t i a l I t e r a t i o n A l g o r i t h m . A t t i m e t , p i c k a tr a n d o m a u n i t i 9 { 1 , . . . , N } , w i t h u n i f o r m d i s t r i b u t i o n ,o r i n a n y c a s e , i n s u c h a w a y t h a t a l l t h e u n i t s c a n b ese l ec t ed . Theni f E G j S} > 0 , s ~+ 1 = + 1Ji f Z C i j S } < O , S~ + a = - 1

1

i f E C~jS } = O , S ~ + 1= ~ .JA t e a c h s t e p , o n l y o n e u n i t i s c h e c k e d .

b ) P a r a l l e l I t e r a t i o n A l g o r i t h m . At t im e t , c a l cu l a t e a l lt h e s u m s ~ C i ~ a n d s e t S ~ + 1= + 1 , - 1 , o r S ~ a s1Z C ~ j S } > O , < 0 , o r = 0 .i T h e s y s t e m e v o l v e s b y l i n i n g u p t h e s t a t e S , w i t h t h eloca l f i e ld de f ine d a s lZ Ci r.JF o r t h e t w o k i n d s o f d y n a m i c s , t w o n o t i o n s a r ei n t e r e s t i n g :

T h e s t a b i l it y . A c o n f i g u r a t i o n S O s s t a b l e i f f S = S O o re v e r y t .T h e a t t r a c t i v i t y . A c o n f i g u r a t i o n S o is a k - a t t r a c t o r( f o r 1 < k < N ) i f f s t a r t i n g f r o m a c o n f i g u r a t i o n S w h i c hp r e s e n t s k e r r o r s w i t h r e s p e c t t o S ~ t h e d y n a m i c l ea d sto S ~O b v i o u s l y t h e s t a b i l i ty a n d t h e a t t r a c t i v i t y o f ac o n f i g u r a t i o n S a r e d e f i n e d f o r e a c h k i n d o f i t e r a t i o nm o d e a n d f o r e a c h g i v en m a t r ix .

I n t h i s p a p e r w e a t t e m p t t o s o l v e t h e f o l l o w i n gp r o b l e m : G i v e n p a t t e r n s $ 1 , . . . , S p , b u i l d a m a t r i x C ( i .e .a n a l g o r i t h m ) i n s u c h a w a y t h a t t h e S x . . . , S p a r e s t a b l ea n d k - a t t r a c t o r s w i t h k a s g r e a t a s p o s s i b l e .

T h e p a p e r i s o r g a n i z e d a s f o l l o w s :In Sec t . 2 , we g ive ex ac t d e f in i t i ons o f s t ab i l i t y ,

a t t ra c t i v it y , d o m a i n o f a t t r a c t i o n o f a c o n f i g u r a t io n .S e c t io n 3 i s d e v o t e d t o t h e s t u d y o f o r t h o g o n a lp a t t e r n s .In Sec t . 4 , fo l l ow ing Pe r s on na z e t a l . (1985), we g ive

t h e g e n e r a l f o r m u l a t i o n o f t h e c o n n e c t i o n m a t r i x Cw h i c h p r o v i d e s s t a b i l i t y t o p a t t e r n s S 1 . . . . S p. T h e n i nS e c t. 5 , w e g e t t h e m a t h e m a t i c a l e x p r e s s i o n o f p a t t e r n sa t t r a c t i v i t y f o r a g i v e n m a t r i x C .

S e c t i o n 6 d e s c r ib e s a c o n s t r u c t i o n o f t h e m a t r i x C ,t h a t m a x i m i z e s a l l t h e p a t t e r n s a t t r a c t i v i t y .

I n S e c t. 7 , w e c o m e b a c k t o t h e c a s e w h e r e a l lp a t t e r n s a r e p a ir w i s e o r t h o g o n a l , a n d s t r e s s t h e i n te r -e s t o f t h a t s i t u a t i o n .

I n S e c t. 8 , w e s o l v e t h e s e t t l e d p r o b l e m , b y c o n -s i d e r i n g t h e s i t u a t i o n w h e r e a l l t h e p a t t e r n s h a v e t h es a m e d e g r e e o f a t tr a c t i v i t y .I n S e c t. 9 , w e i n t r o d u c e t h e u s u a l n o t i o n o f c o n -f i g u r a t i o n e n e r g y , w h i c h i s o n l y d e f i n e d w h e n t h em a t r i x C i s s y m m e t r i c , a n d w e g i v e t h e r e l a t i o n sb e t w e e n e n e r g y m i n i m a a n d a t t ra c t o r s .

S e c t i o n 1 0 i s d e v o t e d t o a d i s c u s s i o n .I n A p p e n d i x 1 , s o m e p r o p e r t i e s o f t h e H a m m i n g

d i s t a n c e a r e r e c a ll e d , a n d i n A p p e n d i c e s 2 a n d 3 ,n u m e r i c a l e x a m p l e s i l l u s t r a t e o u r r e s u l t s a n d s h o wt h a t i t i s i m p o s s i b l e t o g e t a b e t t e r e s t i m a t e o f t h ea t t r a c t i v i t y t h a n t h e g i v e n o n e .

2 S t a b i l i t y - A t t r a c t i v i t yL e t C b e a c o n n e c t i o n m a t r i x , a n d S = ( Si) a c o n f i g u r -a t io n . W e c o n s i d e r e ac h o f t h e t w o k i n d s o f al g o r i t h m s(sequen t i a l o r pa ra l l e l ones) .

W e d e n o t e b y 6 (S , S ') th e H a m m i n g d i s t a n c e o f tw oc o n f i g u r a t i o n s S a n d S ' , i .e . t h e n u m b e r o f c o m p o n e n t sw h e r e S a n d S ' d if f er , a n d b y d(S , S~) t h e E u c l i d i a nd i s t a n c e o f S a n d S ' , v i e w e d a s e l e m e n t s o f ] RN.

O n e h a s d ( S , S ' ) = 2 ~ ( see App end ix I fo rp r o p e r ti e s o f t h e H a m m i n g d i s t an c e ).


3/11

Definition 2.1 . A conf igura t io n S i s s t ab l e (wi th respec tto C) i f f s t a r t ing f rom S , the ne t wo rk s t a t e remain s S , i .e .if f

>__o

T h e s e t o f st a b l e c o n f i g u r a t i o n s i s d e n o t e d b y E 0.Definition 2.2. Le t k be an in teger . A conf igu ra t ion S isa k - a t t r a c t o r (wi th respec t to C) if f s t a r t ing f ro m S ' w i th6 ( S , S ' ) = k , t h e n e t w o r k s t a t e e v o lv e s i n o n e s t e pt o w a r d s S " w i t h 6 ( S , S " ) < k - 1 o r l e a v e s S ' i n v a r i a n t(only in the s equ ent i a l a lgor i thm) . T hi s i s equiv a lent tot h e c o n d i t i o n :

V i, V j l , . . . , j k (wi th J l . . . . . J k al l di fferent)S i ~ C i j S j - 2 2 C i j, Sj z > O. (2.2)\ j t = l /T h e s e t o f k - a t t r a c t o r s i s d e n o t e d b y E k .P r o o f o f t h e A b o v e E q u i v a l e n c e . L e t S = ( S , ) b e ac o n f i g u r a t i o n w h i c h i s a k - a t t r a c t o r a n d S ' s u c h t h a t6(S, S ' ) = k . Th us S ' = (S' i) wi thS~ = -= Si f o r i ~ { J l . . . . . J k }S ~ = S i o t h e r w i s e .W e s t a r t f r o m S ' . F o r i i n t e g e r p i c k e d a t r a n d o m(sequent i a l a lgor i thm) or for every i (pa ra l l e l a l -g o r it h m ) w e c o m p u t e

k= 2 . c ij s = 2 c , j s j - 2 2 c , j , s j , .J J / = 1

I f 0q > 0 , (resp . a i < 0) we se t the c om po ne nt i t o be + 1(resp. - 1).T h e c o n d i t i o n a f l i > 0 m e a n s t h a t w e li n e u p t h es p i n i w i t h t h e c o r r e s p o n d i n g v a l u e o f S , a n d t h e r e f o r ea t t h e n e x t t i m e f o r t h e s e q u e n t i a l a l g o r it h m , t h e n e wc o n f i g u r a t i o n w i l l b e S " w i t h 6 ( S , S " ) = k - 1 i f

i e { j l . . . . , Jk } o r S ' i f n o t . F o r t h e p a r a l le l a l g o r i t h m t h en e w c o n f i g u r a t io n w il l b e S . [ ]R e m a r k s 2 . 3 . 1 ) S a n d - S s a ti s fy t h e s a m einequa l i t i e s .

2) S i s s t ab le ( resp . k-a t t rac tor ) for the s equ ent i a la lgor i thm i f f i t i s for the pa ra l l e l a lgor i thm . Ind eed , forb o t h a l g o r i t h m s , t h e i n e q u a l i t i e s t o c h e c k f o r e v e r yi n t e g e r i p i c k e d a t r a n d o m a r e t h e s a m e . T h i s w i l la p p e a r a s a c o n s e q u e n c e o f t h e f o l l o w i n g p r o p o s i t i o nwh ich ensures th a t th e Def in i t ion s 2 .1 and 2 .2 a rec o h e r e n t . NProposition 2.4. F o r k < 2 ' o n e ha sEkCEk _l C . . . C E I C E o .P r o o f . A d d i n g u p t h e i n e q u a l i ti e s d e f i n in g E k, o v e r a l li n t eg e r s J l , - - . , A , w e o b t a i n t h a t E k C E o . T h e n a d d i n g

Fig. 1. Ham ming radius

131

u p o v e r a n y s u b s e t o f ( k - 1 ) i n te g e rs t a k e n a m o n g{ J l . . . . , J k} , w e g e t E k C E k - 1 , u n d e r t h e c o n d i t i o n t h a tN > 2k . T h i s l a st c o n d i t i o n i s o b v i o u s i f w e n o t e t h a t6 ( S , - S ) = N , a n d t h a t S e E k if f - S e E k : i n d e e d i f2 k > N , t he re w ou ld ex i s t a s t a t e S ' w i th 6(S ', S )= k= 6 ( S ' , - S ) , w h i c h i s i m p o s s i b l e s i n c e t h e s p h e r e sB ~ ( S , k ) , B ~ ( - S , k ) a r e d i s j o i n e d w h e n S e E k [ s p h e r eB~(S, k) i s th e s e t o f c o n f i g u r a t i o n s w h o s e d i s t a n c e o f Sis less than k] .

M o r e o v e r , n o t e t h a t t h e i n e q u a li t ie s d e f i n in g E k a rei n c o m p a t i b l e f o r k= > 2 " [ ]

L e t u s n o w t o d e f in e th e d o m a i n o f a t t r a c t i o n o f aconf igura t ion .Definition 2.5. The d o m a i n o f a t t r a c t i o n ( D A ) o f ac o n f i g u r a t i o n S i s t h e ( m a y b e e m p t y ) s e t o f co n f i g u r -a t ions S ' such tha t , s t a r t ing f rom S ', t h e a l g o r i th m l e a d sto S .H o w e v e r , i t i s c o n v e n i e n t t o c o n s i d e r o n l y c i r c u l a rd o m a i n s , i . e . s p h e r e s f o r t h e H a m m i n g d i s t a n c e .Definition 2.6. The H a m m i n g r a d i u s o f t h e D A o f ac o n f i g u r a t i o n S is t h e H a m m i n g r a d i u s o f t h e g r e a t e s tsphere inc luded in i t .

C o n s e q u e n t l y , i f S ~ Ek, t h e r a d i u s o f i ts d o m a i n o fa t t r a c t i o n i s > k .N o t e t h a t a H a m m i n g s p h e r e w i t h c e n t e r S a n d

r a d i u s k, B~(S, k) , i s the in te rsec t ion of the Euc l id ians p h e r e w i t h c e n t e r S a n d r a d i u s 2 V ~ , Bd(S, 2] /~ ) , andt h e h y p e r c u b e { - 1 , + 1 }N.R e m a r k 2 . 7. T h e m a x i m a l s i ze o f t h e D A o f t h e p a t t e r n si s n e c e s s a r i l y b o u n d e d b y t h e m u t u a l H a m m i n g d i s -t ances o f the pa t t e rn s (and the i r opp os i t e ) , s ince S ~ Eka n d S ' s E k, r e qu i re k + k ' < 6 ( S , S ' ) a n d k + k '< - s ' ) = N - S ' ) .

3 Case of Orthogonal PatternsT w o c o n f i g u r a t i o n s S a n d S ' a r e o r t h o g o n a l i f f t h e i rE u c l i d i a n p r o d u c t ( S , S ' ) = ~ , S i S 'i = O . ( T h u s o r t h o -

ig o n a l i t y i s d e f m e d a s u s u a l i n t h e E u c l i d i a n s p a c e ~ - J .)


4/11


5/11

1 PW e s e e i m m e d i a t e l y t h a t f o r C i = - ~ , l Dp ~ = _n a t u r a l c o r r e s p o n d s t oh o i c e w h i c h

1 pC i j = e , ,- - /' -Z - l sms ' j " i . e . t o Hebb ian connec t ions (3 .1 ) ,a p a r t f r o m c o n s t a n t ~ ) , t h e S " m a y n o t b e s ta b le , a sc a n b e o b s e r v e d b y n u m e r i c a l s i m u l a t i o n s . ( S e e a l s oPe rsonnaz e t a l . 1985 . )

H o w e v e r t h i s c h o i c e i s o f t e n c o n v e n i e n t , e s p e c i a ll yw h e n t h e S m a r e c h o s e n a t r a n d o m , f o r t h e n t h e y a rea l m o s t o r t h o g o n a l ( s ee S e c t . 3 ).

I n P e r s o n n a z e t a l . ( 19 85 ), o n e c a n f i n d t h e g e n e r a le x p r e s s i o n f o r t h e m a t r i x C e n s u r i n g s t a b i l i t y o fp a t t e r n s S ~ . . . . S p , u n d e r t h e c o n d i t i o n t h a t t h e y a r el i n e a r l y i n d e p e n d e n t .

T h e i r f o r m u l a i sC = ( A ~ S ~ .. . , A p S p) ( S ' S ) - ~ S ' + C , (4.2)w h e r e Z i s t h e ( N x p ) m a t r i x w i t h c o l u m n s S ~ . . . . S p ,A a . . . . , A p a r e a r b i t r a r y p o s i t i v e d i a g o n a l N - m a t r i c e sa n d C is a ( N x N ) - m a t r i x s u c h t h a t C 2 = 0 . I n d e e d , t h es y s t e m t o b e s o l v e d i s ( C i, s m s m ) > o ( i = I , . . . , N ;rn = 1 , . . . , p ) o r equ iv a l en t l y :( C i , S m ) = a m s m i f o r a r b i t r a r i e s a m > 0 . ( 4.3 )A m o r e c o n d e n s e d f o r m i s : C S m = A m S i n, fo rm = 1 , . . . , p , w h e r e A m i s a n a r b i t r a r y p o s i t i v e d i a g o n a lN - m a t r i x .

T h e g e n e r a l e x p r e s s i o n o f C i i sP PC~ = ((a]Sr . . . , ai Si) (S 'S ) -~ 27'))'+ Ci , (4.4)

wh e re a~ , . . . , al ' a r e a r b i t r a ry pos i t i ve sca l a r s a nd Ci i so r t h o g o n a l to S a , . . . , S p . W e n o t i c e t h a t C i i s t h e s u m o fa l i n e ar c o m b i n a t i o n o f S ~ , . . . , S p , a n d o f a n o r t h o g o n a lv e c t o r , a n d t h a t i t is d e f i n e d u p t o a p o s i t i v e m u l t i p l ic a -t i v e c o n s t a n t .

W e m u s t d e t e r m i n e h o w t o c h o o s e t h e N x pc o n s t a n t s a m , a n d t h e v e c t o r s C i t o o p t i m i z e t h ea t t r a c t i v i t y o f S a , . . . , S p . S o , w e s h a l l s t u d y t h e s iz e o ft h e d o m a i n o f a t t r a c ti o n o f t h e S " , a s a f u n c t i o n o fa r b i t r a r y c o e f fi c ie n t s o f m a t r i x C .

133Le t us den o te by T jl . .. . Jk t he s i gn mo di f i e r i n

p o s i t i o n J l , . . . , J k o f a c o n f i g u r a t i o n , i . e . t h e m a p p i n gd e f i n e d b yTa . . .. . ~(S )=S *w i t hS * = S i fo r i q ~ { j D . . . , j k }S * = - - S i fo r i ~ { j l , . . . , j k } .T h u s S i n s E k (S m i s a k -a t t r ac to r ) i f f > 0 (5.1)f o r e v e r y i e { 1 , . . . , N } a n d f o r e v e r y s u b s e t { j~ . . . , j , } o fd i s t i n c t i n t eg e r s o f { 1 . . . N } . [ N o t a t i o n s o f ( 4 .1 ) a n dDef in i t i on 2 .2 . ]

W e m a y i n t e r p r e t t h e s e i n e q u a l i ti e s in a g e o m e t r i cw a y : w e d e n o t e b y a m ', ( re s p. b ) t h e e n d p o i n t s o f t h eve cto rs Dm [resp. Tj, . . .. ~(om)].

T h e c o n d i t i o n (5 .1 ) m e a n s t h a t S m i s a k - a t t r a c t o r i f ff o r a ll i, t h e H a m m i n g s p h e r e w i t h c e n t e r a m a n d r a d i u sk is e n t i r e l y o n t h e s a m e s i d e o f H i = C { ( o r t h o g o n a lspace o f vec to r C i ) see F ig . 3 . I t me ans t ha td ( a .~ , b ) = 2 ] / / b ( D m , T j l .. .. . j ~ (D m ) ) ( b y A p p e n d i x 1 )

= 2 ~< d ( a m , H i ) =

M o r e p r e c i s e l y ,

( C i , D m )IIGll

I ( Ci , Tj l . . . . j~(Dm) - ( C, , D .m ) l= l( C i, O h ) - ( Q , O a m )l= I ( Q , O h - O a m )l

k=2 Y IGj, l


6/11

134

H e n c eP r o p o s i t i o n 5 .2 . F o r a c o n n e c t i o n m a t r i x C = ( C ~ j ) ,w h o s e r o w s a r e C 1 . . . . , C n , e a c h p a t t e r n S ", is k - a t t r a c t o ra t l e a s t u p t o a n y k s u c h t h a tk < 89 (C i ' D~n) - ? " , , (5.2)i m a x l C l j [Jw h e r e D .~ = S ~ S ~ .

So i f k and k ' a re tw o in t ege r s sa t i s fy ing (5 .2 ), w. r . t.S", ( for k) and S"," ( fo r k ' ) , S" , e Ek , S~" e Ek , , a n d k + k '< inf(6(S ", S" ' ) , N - 6(S" , , S" ,' )) .N o w w e p r o c e e d t o s i m p l i fy t h e e x p r e s s i o n o f T , , [ i n(5 .2 ) ] when C i s g iven by (4 .2 ) , ( ensunng pa t t e rnss t ab i l i t y ) .T h e c h o i c e o f t h e C i ' s , i n (4 .4 ) ha s t o ensure t ha td(a'r, Hi) or ?m a re a s b ig a s poss ib l e .

F i r s t , w e see t ha t i n (4 .4 ) ~ i = 0 i s t he be s t cho ice .I n d e e d i f C i = C i + C~ w i t h C i in t h e s u b s p a c es p a n n e d b y { $1 , . . . , S p} a n d ~ i n ~ w e h a v e< C i , O m > = < ~ i , O T >a n dI l C i l l 2 = 1 1 r I I ~ i l l 2 , s o d ( a ' ~ , H i ) = - -wi l l be g rea t e r fo r ~ = O.Us ing (4.3) ,( C i , D T ) = S ' f ( C i , S " , ) = S . 7 ( a 7 S 7 ) = a T .As to t he vec to r C i , we wr i t e , f rom (4 .4 ) ,

C~ = S(~ '2 : ) - t\ ~ s U

( C i , D T )t l c i l l

( 5 . 3 )

(5.4)

a n d

I l C i l l 2 = C ' , C , = ( a g s 2 . . . . . ~fsD(z 'z) - t 9\ ~f s U

=(~r . . . . . ~ f ) d i a g ( S r S f ) ( S ' S ) - tx d i a g ( S ~ . . . S D "

[ w h e r e d i ag ( S ~ . . . S I ') i s t h e m a t r i x w h o s e d i a g o n a le l e m e n t s a r e S ~ , . . . , S f a n d o t h e r s a r e 0 ]= c~' iWie (obvious nota t ions) .

W e r e m a r k t h a t i f w e d e n o t e b y D i = ( D ~ , . . . , D f )= ( S ~ S t , . . . , S f S P ) , i . e . t h e m a t r i x o f t h e p a t t e r n s n o r -

m a l i z e d t o + 1 i n t h e i - t h c o m p o n e n t , w e h a v eWi = (D , iDi) - l .S odE(a? ,H,) = ( e ~ ) 2 ( 5 . 5 )a n d?" , = 89 aT'i maxICo[J

(5.6)

T h e i n e q u a l i t y (5 .2 ) d o e s n o t g i v e e x a c t v a l u e s o f t h es iz e s k t , . . . , k p o f t h e d o m a i n s o f a t t r a c t i o n ( D A ) o fS t , . . . , S p , becau se i t i s on ly a su f f i c i en t cond i t i on 9H o w e v e r , t h e p a t t e r n S m s a t t r a c t i v e a t l e a s t u p t o? ,, , f o r r e = l , . . . , p .U s i n g g e o m e t r i c n o t a t i o n s l e t u s c o n s i d e r p a t t e r n sS t , S a Le t us a ssu me tha t 6 (S t , S 2 ) i s sma l l w i th r e sp ec tt o N . S i n c e f o r e v e r y i s u c h t h a t S ~ = S ~ , t h e d i s t a n c e

d(a.~, H~) i s " g r e a t " , w h e r e a s t h e s iz e s o f t h e D A o f S tan d S z a re sma l l , [ s ince l e ss t ha n 6 (S t , Sz ) ] t h i s seemscon t rad i c t i o n 9 B u t w e m u s t n o t i c e t h a t o n t h e c o n t r a r y ,f o r i s u c h t h a t t 2i = - S i , w e h a v e D ~ = - - S t o rD ~ = - S 2 a n d d(a .~, Hi ) s m a l l , w h i c h l e a d s t o a s m a l lv a l u e o f ? , ,- f o r m = 1 , 2 .N o w , P r o p o s i t i o n 5 .2 c a n b e c o m p l e t e d b y :

P r o p o s i t i o n 5 .7 . F o r a c o n n e c t i o n m a t r i x C=(C,~) ,w h o s e r o w s C t . . . . . C N a r e g i v e n b y (4.4) w i t h C , = O , e a c hp a t t e r n S ", i s a t t r a c t o r a t l e a s t u p to H a m m i n g d i s t a n c e?m = 89 ami max[C~jlJ

( 5 . 7 )

6 O p t i m a l M a t r i xL e t u s s t a r t f r o m a n i n i t i a l c o n f i g u r a t i o n S O o b t a i n e db y d i s t o r t i n g o n e o f t h e p a t t e r n s , e .g . S ~ ~ T h e r e w i l ln o t b e a n y i d e n t i f i c a t io n e r r o r i f S O b e l o n g s t o t h ed o m a i n o f a t t r a c t i o n ( D A ) o f S ",~

H e n c e t h e n e x t d e f i n it i on :D e f i n i t io n 6 .1 . A m a t r i x C f o r w h i c h t h e p a t t e r n sS t . . . . , S p a re a t t r ac to r s , i s o p t i m a l i f i t m a x i m i z e s t h em i n i m u m r a d i u s ( D e f i n i ti o n 2 .6 ) o f th e D A s o f t h epa tter ns 9S i n c e w e h a v e n o t t h e e x a c t v a l u e o f th e s e r a d i i , w et r y t o d e t e r m i n e a m a t r i x C , c a l l e d s e m i - o p t i m a l , a n dw h i c h m a x i m i z e s t h e m i n i m u m d i s t a n c e d(a. ' f ,H i )(Fig. 3) .T h u s w e l o o k f o r p o s i t i v e c o n s t a n t s a~ , m = 1 , . . . , p ,i = 1 . . . , N , w h i c h m a x i m i z e f o r e a c h i , infdE(a'~, H~)


7/11

~ 2

Q ~ P 2~Q 2/ -C e s e 1P 2s s o l u t i o n

(I21 Q 1 ( ~ 2 ~ Q / Q 2

C a s e C a s eT h epo in t c t~= )P~s s o lu t ion is so tu t ion Fig. 4 . Case p = 2

13 5

. ~ ( a . 7 ' ) 2= l m ~ [ b y ( 5.5 )] . L e t u s s k e t c h a c o n s t r u c t i o n o fm a i W i a it h e a ~ f o r f i x e d i .a ) W e l o o k f o r a v e c t o r a i = ( a ~ . . . . . a ~') i n ( E ~ +)v,

w ith a ' iWia i = 1 .b ) W e c u t o u t ( N + ) p i n t o q u a d r a n t s Q j d e f i n e d b ya i = m i n (a m ), f o r j = 1 . . . ,p .I n e a c h q u a d r a n t Q , w e w a n t t o m a x i m i z e ( ai ) z , i.e .

a 1. F o r i n s t a n c e , i f p = 2 , w e c o n s i d e r t h e e l l i p s e w h o s ee q u a t i o n i s a 'i W ~ ai = 1 ,c ) W e c o m p u t e t h e p p o i n t s P j = ( a ~ . . . . , a f) ,j = 1 , . . . , p , s o l u t i o n s o f a ' i W ~ a i = O and 0a'J"O a 7 '_ ( W i a i )m = 0 f o r m=t=j, a n d k e e p t h e p o i n t s P j( w , a , ) jb e l o n g i n g t o t h e c o r r e s p o n d i n g q u a d r a n t Q j . W e h a v em a x ( m i n ~ ) = m a x { ( a, )2 / pj = ( a ~ , .. ., a [ ' ,s Q j}

9 \ i i i /u n d e r t h e c o n d i t i o n t h a t t h e a b o v e s e t i s n o t e m p t y .d ) I f f o r a l l j , P j ~ Q j ( c a s e 3) , w e r e s t r i c t o u r s e l v e si n t h e q u a d r a n t s o f ( ~ + ) p - 1 , ( ~ + ) p - 2 , e t c . . . d e f i n edb y i n e q u a l i t i es s u c h a s :a i = a { = m in (a ~ )

j _ j ' _ .-a - a i - a ~ = r a i n ( a T ' ) . . . a n d s o o n .I f n o n e o f t h e P j s s u c c es s i ve l y f o u n d b e l o n g s t o t h ec o n v e n i e n t d o m a i n , w e g et t h e s o l u ti o na ~ = a ~ = . . . = a l ' ( s e e S e c t . 8 ) .

W e c a n s u m u p t h e r e s u lt s a s fo l lo w s :Propos i t ion 6 .2 . By i terat ing this approach for everyi = 1 . . . , N , we cons truct a ma tr ix semi-opt imal C, whichleaves as muc h volume a s poss ible , around each point am,for every i , wi thc , = z ( z ' z ) - i ( a~ s ~ . . . a l s o ) ' (6.2)f o r Z = ( S l , . . . , S p ) a n d (a~, . . . , a /p) positive yielding thea m ) 2m a x i m u m o f m i n ~ , w i th W i =( D 'i Di) - 1 a nd

m a i W i a iD , = ( S ~ S ' . . . . , S ~ S ' ) .

7 C a s e o f O r t h o g o n a l P a t t e r n sI n t h i s s e c t i o n w e a s s u m e t h a t t h e p a t t e r n s S 1 , . . . , S pa r e p a i r w i s e o r t h o g o n a l .

I n t h i s c a s e , Z ' Z = N l d p , a n d f o r e v e r y i, D'~D~= N l d p , s i nc e th e v e c t o r s D T a r e a l so p a i r w i s eo r t h o g o n a l .

S o t h e e l li p s es o f e q u a t i o n s a ~W ~ a i= 1 a r e s p h e r e sa n d t h e r e s e a r c h o f t h e s e m i - o p t i m a l a~ ", l e a d s t o t h es o l u t i o n a ~ . . . . . a ~, ( s ee t h e t w o - d i m e n s i o n e de x a m p l e , c a s e 3 , i n S e c t . 6 ) .

S i n c e t h e a~ " a r e d e f i n e d ( f o r e a c h i ) u p t o a p o s i t i v em u l t i p l i c a t i v e c o n s t a n t , w e m a y c h o o s e a ] ' = 1 f o r e v e r yi a n d e v e r y m .

S o f r o m ( 5 . 4 ) ,C i = Z ( l l d , ) ( S ~ . . . S~)' 1 Pa n d C u = ~ m Z _ I S ~ iS ~ .

(7.1)I n o t h e r w o r d s , i n t h e o r t h o g o n a l c a s e , t h e s e m i -

o p t i m a l s o l u t i o n c o r r e s p o n d s t o t h e e q u a l i t y o f t h e a m ,a n d t o t h e c l a s s ic a l H e b b i a n c o n n e c t i o n s . [ I n t h a t c a s eC is t h e p r o j ec t i o n m a t r i x i n t r o d u c e d b y K o h o n e n(1 9 7 0 ) . ]

T h u s , w e m a y c o m p u t e ?m a n d infd2(a~, Hi), u s i n g(5 .5 ) and (5 .6 )dZ(aT,,Hi) = (aT')2 _ N (7.2)a ' iW ~ a i p 'a n d

a m_Vm =ffin ..~ N . 1, m . a x l C u l - ~ - l ~ p .j m a x . Z = SrS~'B u t m a x ~$7'S~' =p i s o b t a i n e d f o r j = i , s i n c e a l lJ mt e r m s o f t h e s u m a r e t h e n e q u a l t o + 1.

H e n c e w e g e tN? m = ~ ( i n d e p e n d e n t o f m ) . ( 7. 3)


8/11

136S o w e f i n d a g a i n t h e r e s u l t o f S ec t. 3 , w h i c h w e n o we n o u n c e a s f o ll o w s :

P r o p o s i t i o n 7.4. In t he case o f pa i rw i se o r t hogona lpa t t erns , t he Hebb i an connec t i on ma t r i x (3.1) i s a t leastsemi -op t i ma l and ensures the a t t rac t i v i t y o f t he pa t t ernsNa t l eas t up t o t he Hammi ng d i s t ance ~p .

I n t h e n o t o r t h o g o n a l c a s e , w e r e m a r k t h a t i fN6(S", Sm') > ~- fo r so me (m, m ' ), t he n 6(S" , - S m) < N ;t h e r ef o r e th e v o l u m e " w i n n e d " b y t h e c o m p o n e n t i w il lb e " l o s t " b y c o m p o n e n t i ' s u c h t h a t D ~ = D ~ a n dDT '= - -D .~ '.T h e s iz es o f t h e d o m a i n s o f a t t r a c t io n d e p e n d o nt h e m i n i m a o v e r i ; w e s e e t h a t t h e f a v o u r a b l e c a s e is t h eo r t h o g o n a l o n e .H e n c e t h e p r a c t i c a l i n t e r e s t o f- p i c k i n g a t r a n d o m [ w i t h I P ( S ~ + 1 )= P ( $ 7 ' = - 1 ) = 8 9 a s i n s p in gl as se s, e n s u r i n g , o na v e r a g e , o r t h o g o n a l i t y .- a d e t e r m i n i s t i c e n c o d i n g o f t h e o b j e c t s t o b e r e -c o g n iz e d , b y m e a n s o f p a i rw i s e o r t h o g o n a l p a t t e r n s .

8 D o m a i n s o f A t t r a c t i o n w i t h E q u a l S i z e sT h e r e s e a r c h o f t h e o p t i m a l m a t r i x C s k e t c h e d i nSec t . 6 i s t ed ious . W e sha l l s impl i fy i t by t ak i ng a l lc o n s t a n t s 0r7 ' = + 1 ( P e r s o n n a z e t al .' s m e t h o d ) , w h i c h i se q u i v a l e n t t o e q u a l a t t r a c t i v i t y o f e a c h p a t t e r n .W e n e e d n o w a g e o m e t r i c i n t e r p r e t a t i o n a n d al o w e r b o u n d o f t h e c o m m o n s i z e o f t h e d o m a i n s o fa t t r a c t i o n ( D A ) .S i n ce ~ ' = ( C i , D.~) , by (5.3) , i n t h e ca se ~" = 1 fo re v e r y m = 1 . . . , p , w e h a v e t o d e t e r m i n e a h y p e r p l a n eH i, o r t h o g o n a l t o C i a t e q u a l d i s t a n c e o f a l l t h e p o i n t sam fo r m = 1 . . . , p ( see F igs . 2 a nd 3 ). The vec to r C i i so r t h o g o n a l t o t h e a f f i n e s p a c e c o n t a i n i n g a l l p o i n t sa] , . . . , a~.

We wr i t e , f rom (5 .4 ) ,

C i = 2 ( 2 ' 2 ) - ~ " i.e. C = S ( I ' I ) - I I ' , (8.1)\ s f /

w h i c h i s t h e o r t h o g o n a l p r o j e c t i o n m a t r i x o f l R N o n t h ev e c t o r s p a c e Y /~ , s p a n n e d b y $ 1 , . . . , S p ( i n t r o d u c e d b yK o h o n e n 1 97 0) . T h e m a t r i x C i s s y m m e t r i c , a n d i tsc o l u m n s ( e q u a l t o i t s r o w s ) a r e i m a g e s o f t h e v e c t o r se l , . . . , eN (canon ica l b a s i s o f R N) by th i s p ro j ec t ion .S o w e h a v e C i = p r o j ~ ( e i ) , D m ~ / / " , a n d < C i , D ~ )= < e i , D m ) = + 1 b y t h e d e f i n i t i o n o f t h e v e c t o r s D T ' ,w h o s e i - t h c o m p o n e n t i s e q u a l t o + 1 .

H e n c e , n o t i n g t h a t m a x I C i j l = max(C/ i )= m a x 11Cil[2 s ince C i s a p ro j ec t ion m a t r i x (C 2 = C) , we

ig e tP r o p o s i t i o n 8.2. F o r t he m a t r ix C = Z ( S ' S ) - 1 2 ' w i t h

= ( Sa , . . . , SP), each pa t t ern S m i s a t trac t or a t l eas t up t oH a m m i n g d i s t a n c e1 1 1 1

? = 2 maxi I l C i l l 2 - 2 m a x C u. (8.2)S e e n u m e r i c a l e x a m p l e s o f e v a l u a t i o n o f 7 i nA p p e n d i x 2 .

R e m a r k 8 . 3 . T h e m a t r i x C o b t a i n e d w h e n c ~7 '= 1 , f o re v e r y i , m , i s s y m m e t r i c . H o w e v e r f o r w h a t c o n c e r n s t h ea l g o r i t h m , i t i s e q u i v a l e n t t o t h e m a t r i x o b t a i n e d b ym u l t i p l y i n g e ac h r o w b y a p o s i ti v e a r b i t r a r y c o n s t a n t :t h is m a t r i x i s n o m o r e s y m m e t r i c .N e i t h e r i t i s i n t h e o p t i m a l c a s e o f S e c t. 6.

9 E n e r g yT h e p a t t e r n s $ 1 , . . . , S v s p a n t h e s u b s p a c e ~ , a n d t h em a t r i xC = S ( S ' S ) - 1S ' [see (8.1)]i s t h e s y m m e t r i c p r o j e c ti o n m a t r i x o n r w h o s ee l e m e n t s a r e i n i n t e r v a l [ - 1 , + 1 ] , w i t h C = C z = C 'C ,h e n c eCij = ( Ci , C j ) = H ( S * ).

iii) The l earned pa t t erns a re abso l u t e maxi ma o f Hand any abso l u t e maxi mum i s a s t ab l e s t a t e .Demons t ra t i on . i ) i s c l ea r . We p rove i i ) n o t i n g t h a tH (S )-- H (S*) = 4 y, S i ( ~. CI jS3~ for I = {Jl .. . ,Jk}.i 8 I \ j ( s l , ISince S i s k -a t t r ac to r , i t i s s t ab l e and fo r a l l i ,


9/11

\ J }(2.2), hence H ( S ) > H ( S * ) . F o r i i i ) , w e w r i t e H ( S ) = S ' C S= ( c s ) ' c S - - [I C S I [ [ I p r o j ( S ) ] l 2 f o r a l l c o n f i g u r a t i o n

s e { - 1 , + 1 } " .Since [ IS[ I - -g, H(S) 0 u s i n g t h ea n n e a l i n g m e t h o d , o n l y t h e a b s o l u t e m a x i m a o f H w i llb e r e a c h e d , i.e . t h e s p u r i o u s s t a b l e s t a t es b e l o n g i n g t o

10 Provis ional C onc lus ions1 ) T o c h o o s e t h e s i z es o f t h e d o m a i n s o f a t t r a c t i o n( D A ) i n s u c h a w a y t h e y a r e e q u a l f o r a l l p a t t e r n s

S ~ , . . . , S v e n a b l e s u s t o g i v e n o p r e f e r e n c e t o a n yp a t t e r n .A s s u m e t h a t w e s t a r t f r o m a n i n i ti a l c o n f i g u r a t i o nS O d i s t o r t e d f r o m S ~ ~ W e m o d e l i z e t h i s d i s t u r b a n c e :a n e r r o r o c c u rs i n e a c h c o m p o n e n t i = 1 . . . , N , i n d e -p e n d e n t l y w i t h a s m a l l p r o b a b i l i t y q. T h e n u m b e r o fe r r o r s 6 ( S ~ S "~ i s a B i n o m i a l d i s t r i b u t i o n ~ ( N , q ) .S t a r t i n g f r o m S o t h e a l g o r i t h m a c t s a n d t h ep r o b a b i l i t y t h a t i t g i v e s a g o o d a n s w e r (Sin~ isb o u n d e d f r o m b e l o w b y t h e p r o b a b i l i t y t h a t S o b e l o n g sto t he D A of S m~ wi th r ad ius k , ,o , i .e . by P(6 (S ~ Sm~~ k m o ) .O f c o u r s e , w e c a n a p p r o x i m a t e t h i s p r o b a b i l i ty , b ys u b s t i t u t i n g t h e N o r m a l D i s t r i b u t i o n ~ A / ( N q , N q ) t ot h e B i n o m i a l d i s t r i b u t i o n . ( N i s g r e a t a n d q s m a l l . )S o s e l e c ti n g a n a l g o r i t h m , i. e. a m a t r i x C s u c h t h a ta l l t he km a r e e q u a l , d o e s y i e ld th e s a m e l o w e r b o u n d sf o r a l l t h e p r o b a b i l i t i e s o f c o r r e c t i d e n t i f ic a t i o n , w h a t -e v e r is t h e c o n f i g u r a t i o n S " t o i d e n t if y .

2 ) N o w l e t u s a s s u m e t h a t t h e c h o i c e o f t h ec o n f i g u r a t i o n S m t o i d e n t i f y , i s m a d e w i t h a p r o b a b i l i t y(P ro ) ( ~ P m = 1 ] . T h e n t h e p r o b a b i l i t y o f w r o n g i d e n t if i -

/c a t i o n i s l e s s t h a n ~ = ~ p m P ( J ( S ~ ~ ar isesmf r o m S m) a n d i f a n i d e n t i f i c a ti o n e r r o r f o r S m costs g , , ,

137t h e m e a n e r r o r c o s t i s le s s t h a n G = ~ g m p m ~ ( ~ ( S O , S m )> k m / S ~ ar ises f ro m s 'n) . '~I n t h a t c a s e, t h e c h o i c e o f th e s e m i - o p t i m a l m a t r i xC, desc r ibed in Sec t . 6 , y i e lds a r ea sonab le l owerb o u n d e a n d G .

I t r e m a i n s t h a t t h e s e c a l c u l a t i o n s a r e a p p r o x i -m a t i o n s ( a p a r t f r o m t h e f a c t w e c a n n o t c a l c u l a t e t h eexac t s i ze km of t he D A of t h e p a t t e rn s S m , w. r . t, a g ive nm a t r i x C ) .3 ) I f6 (S ~ S m) > k , , t he re i s an e r ro r i fS ~ f a l ls i n t heD A o f a n o t h e r c o n f i g u r a t io n , b u t w e d o n o t k n o w w h a th a p p e n s i f S Od o e s n o t b e l o n g to a D A o f t he Sm . S o S om a y b e a t t r a c t e d b y o n e o f t h e s p u r i o u s c o n f i g u r a t io n sm a d e a t t r a c t i v e b y t h e m a t r i x C , f o r i n s t a n c e ac o n f i g u r a t i o n - S m , o r o t h e r l i n e a r c o m b i n a t i o n o f t h e

Sm . ( Se e A p p e n d i x f o r n u m e r i c a l e x a m p l e s . )4 ) T h e c o m p l e t e s t u d y o f t h e d e t e r m i n i s t i c a l -

g o r i t h m ( T e m p e r a t u r e T = 0 ) e n a b l e s u s t o s e e t h a to n l y t h e s p u r i o u s s t a b l e s t a t e b e l o n g i n g t o t h e s u b -s p a ce ~ s p a n n e d b y t h e p a t t er n s , r e m a i n w h e n t h et e m p e r a t u r e T > 0 u s i n g t h e a n n e a l i n g m e t h o d . T h i sre su l t conf i rm s the r e su l t s o f Am i t e t a l . (1985a, b ) ,a n d s h o w s i t i s t r u e f o r a l l N , a n d n o t o n l y w h e nN ~ + o o .5 ) T h e a l g o r i t h m e n s u r e s a p e r f e c t r e t r i e v a l o fpa t t e rns i f t he i n i t i a l s t a t e S O sa t i s fi e s 6 (S~[de f ined in (8.2) ] fo r s om e Sm. But , o f course , t hea l g o r i t h m e n s u r e s a v e r y g o o d r e t r ie v a l i f S O i s m o r ed i s t a n t f r o m t h e p a t t e r n s . L e t b e N t h e s e t o f i n i ti a l

s t a t e s w h i c h l e a d t o s o m e p a t t e r n . T h e s i m u l a t i o n ss h o w t h a t N o c c u p i e s a g r e a t p o r t i o n o f t h e h y p e r c u b e[ s e e n u m e r i c a l e v a l u a t i o n s i n P e r e t t o a n d N i e z ( 1 98 5) ,w h i c h a g r e e w i t h e x a m p l e s o f A p p e n d i x 3 ] . O f c o u r s e,t he s i ze o f N dec reases w he n p inc reases , fo r a f i xed N,l i k e t h e a t t r a c t i v i t y 7.

A ppendix 1: Hamming D is t anceFor S, S' ~{ -I , +1} N, we denote by 6(S,S') the number ofdistinct compo nents of S and S': it is the Hamm ing distance of Sand S'.The following properties are easy to check1 ) 6 is a distance in {- 1, +1} ~,N2 ) 6 ( S , - S ) = N ; i f ~ SiS',=O, .e. ifS and S' are orthogonal

i= 1 N(only with N pair), 6(S, S' ) = 2 "3) ~(s , - s ' ) = N - ~(S , S ' ).If d is the Euclidian distance in NN, ( . ) the Euclidianproduct, [[-1[ the E uclidian norm, cos an d sin the usu al trig-onom etric functions, we have . 2 (s , s ' )4 ) I ISIJ=[/N, f i ( S , S ' ) = 8 9 and

d ( S , s ' ) = 2


10/11

1 3 8A p p e n d i x 2 : E v a l u a ti o n o f G a m m aF o r d i f f e r e n t v a l u e s o f N a n d p , w e c o m p a r e 7 in v a r i o u s c a se s

a ) o r t h o g o n a l c a s e ? = N / 2 pb ) r a n d o m c h o ic e o f p a tt e r n sc ) r a n d o m c h o i c e o f ( S 2 . . . . S p ) a n d S 1 = ( 1 . . . . . 1 )d ) a n o n o r t b o g o n a l c a s e .

N = 2 0 a b c dp = 3 3 .3 1 .9 2 .4 1 .5p = 4 2 .5 1 .4 1 .9 1 .1p = 5 2 1 . 2 1 .5 0 . 7

N = 3 0 !p-=-3p = 4p = 5

a b c5 3.7 3 .8 23 .7 2 .6 2 .7 13 1.8 1.9 0.7

N = 4 0 a b c dp = 3 6 .7 5 .3 4 .9 3 .1p = 4 5 4 3 . 9 1 . 2p = 5 4 3 .1 3 .1 0 .8p = 6 3 .3 2 . 3 2 . 4 0 . 7

N = 5 0 a b c dp = 4 6 .3 5 .8 5 .1 1 .4p = 5 5 3 . 8 3 .9 0 . 9p = 6 4 . 2 3 . 0 3 .1 0 . 8

N o t e t h a t i f w e w a n t t o r e t r i e v e e x a c tl y a p a t t e r n t r a n s m i t t e dw i t h 1 0 % e r r o r s a t m o s t , w e m u s t c h o o s e p ~ N , a p p r o x i m a t e l yp ~ 0 . 1 0 N ( f o r r a n d o m c a se ). T h i s a g r e es w i t h n u m e r i c a l r e s u l ts o fH o p f i e l d ( 1 9 8 2 ) o r A m i t e t a l . ( 1 9 8 5 a , b ) f o r i n s t a n c e . ( T a k ea c c o u n t t h e y u s e a c o n n e c t i o n m a t r i x w h i c h d o e s n o t e n s u r e t h es t a b i l i t y o f a l l p a t t e r n s . )

A p p e n d i x 3W e i n d i c a t e n u m e r i c a l r e s u lt s o f v a r i o u s s i m u l a t i o n s .

W e d e t e r m i n e a l l t h e s t a b l e st a te s , t h e i r o r d e r o f a t t r a ct i v it y , t h e i r H a m m i n g d i s t a n c e f r o m p a t t e r n s , a n d t h e i r e n er g y .I n e a c h c a se , 1 0 0 0 t r ia l s a r e p e r f o r m e d w i t h i n i ti a l s ta t e a t r a n d o m . T h e s y s t e m e v o l v e s v e r y q u i c k l y ( a t m o s t 4 p a r a l l e l i t e r a ti o n

s t ep s ) t o o n e o f t h e s t a b l e s t at e s. W e i n d i c a t e t h e n u m b e r o f t ri a l s e n d i n g i n t o e a c h s t a b l e s t a te s . S e e t h a t t h e v a l u e o f ~ a b o v e c a l c u l a t e d i sn o t s u b e s t i m a t e d , a n d g i v es a g o o d e s t i m a t i o n o f t h e B A ' s i ze .E x a m p l e IN = 2 0 , p = 3 , ? = 2 .4 (c a se c o f Ap p e n d i x 2 )S * = 1 i 1 1 1 1 1 1 1

S 2= 1 1 1 - - 1 1 - - 1 - - 1 - - 1 - - 1$ 3 = - - 1 - - 1 1 1 - - 1 1 - - 1 1 1E x a m p l e 2N = 2 0 , p = 3 , 7 = 1 . 9S 1 , S 2 , S a a r e r a n d o m ( c a s e b )E x a m p l e 3N -- 20 , p = 4 , y = 1 .4S 1 , S 2 , S 3 , S 4 a re r a n d o m (c a se b )E x a m p l e 4N = 2 0 , p = 4 , 7 = 1 .1S 1 S 2 , S a S 4 a r e n o t o r t h o g o n a l (6(S 1 S 3) = 6(S 1 S 4) = 3)

1 1 1 1 I 1 1 1 i 1 I

1 1 - - I 1 - I 1 - - 1 - - 1 - I I - I

1 - - 1 - - I - I 1 - - I - I - I - 1 I I

E x a m p l e I N b o f t r i al s e n d i n g i n t o o n e o f t h e p a t t e r n s( o r o p p o s i t e ) = 6 7 %

S t a t e S 1 - S 1 S 2 - S 2 S 3 - S 3 $ 7 $ 8 $ 9 S , o $ 1 1 $ 1 2 S t a S , +At t . o r d e r 2 2 2 2 2 2 1 1 0 0 0 0 0 0

~(s' ) {~(s )

0 20 11 9 11 9 15 5 17 3 14 6 6 141 1 9 0 2 0 1 2 8 1 4 6 6 1 4 1 5 5 1 7 31 1 9 1 2 8 0 2 0 1 4 6 6 1 4 3 1 7 5 1 52 0 2 0 2 0 2 0 2 0 2 0 1 5 .4 1 5 .4 1 5 .4 1 5 .4 1 5 .4 1 5 .4 1 5 .4 1 5 .4

% n .b . t r ia l s en d i ng 11 .7 10.7 10 .2 11 .6 11 .1 12 .2 6 .7 5 .8 5 .4 4 .0 1 .9 3 .1 3 .3 2 .3


11/11

139

Example 2 Nb of trials ending into one of the patterns(or opposite) = 54%State S 1 - S 1 S2 - S 2 S 3 - S 3 $7 $8 $9 $1o $11 $12 $13 S~4Att. order 1 1 1 1 1 1 2 2 0 0 0 0 0 0

~ ( s ~ ){/ r

0 20 12 8 12 8 14 6 18 6 6 2 14 1412 8 0 20 12 8 14 6 6 18 6 14 2 1412 8 12 8 0 20 14 6 6 6 18 14 14 220 20 20 20 20 20 16 16 16 16 16 16 16 16

% n.b. trials ending 9.6 8.8 9.9 8.1 8.3 9.6 9,9 8.2 4.7 6.0 4.3 5.0 2.8 4.8

Example 3. In that case S 1, S2, S 3, S4 and thei r opposite are 1-attractors (compare with y = 1.4). And there are 40 stable states whoseenergy are 14.1, 15.1,15.2, 15.6 or 16. The Hamming distance between a patt ern and a spurious stable state can be 2, and that shows thatthe evaluation of ~, is exact.Among 1000 trials, the system ends into one of the pat tern in 52% of the cases

Example 4. There are 8 1-attractors, S 1, S 2, S a, S4, their opposites and 4 configurations more which have the same energy (H(S)= 20).There are 4 stable states, and 4 1-attractors, with H(S) = 16S t a t e S 1 - S 1 S 2 - S 2 S 3 - S 3 S 4 - S 4 S 9 S l o S l l S 1 2 S 1 3 $ 1 4 $ 1 5 S 1 6

Att. orde r 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 06(S~) 0 20 11 9 3 17 3 17 6 14 6 14 12 9 8 1111 9 0 20 14 6 10 10 13 3 17 7 5 2 15 183 17 14 6 0 20 6 14 3 11 9 17 9 12 11 83 17 10 10 6 14 0 20 3 13 7 17 15 12 5 8H(S) 20 20 20 20 20 20 20 20 16 16 16 16 16 16 16 16

The system ends into one of the patt ern is 52 % of the cases

ReferencesAmit DJ, Gutfreund H, Sompolinsky H (1985a) Phys RevA 32:1007Amit DJ, Guffreund H, Sompofinsky H (1985b) Phys Rev Lett55:1530Caianiel lo ER (1961) J Theor Biol 1:204Cooper LN, Liberman F, Oja E (1978) Biol Cybern 33:9Hebb D O (1949) The organization of behavior. Wiley, New YorkHopfield J (1982) Proc Natl Acad Sci USA 79:2554Kohonen T (1970) Helsinki University of Technical ReportTKK-F-A130Kohonen T (1972) IEEE Trans C-21:353-359Kohonen T (1984) Self-organization and associative memory.Springer, Berlin Heidelberg New YorkKohonen T, Oja E (1976) Biol Cybern 21:85-95Kohonen T, Lehti6 P, Rovamo J (1974) Annales AcademiaeScientiarum Fennicae A: V Mediea 167Koh onen T, Lehti6 P, Rovamo J, Hyv~irinen J, Bry K , Vainio L(1977) Neuroscience 2:1065-1076Little WA (1974) Math Biosci 19:101

Nakano K (1972) IEEE Trans SMC-2:380-388Peretto P (1984) Biol Cybern 50:51Peretto P, Niez JJ (1985) Collective properties of neural net-works. In: Proceedings of the winter school on "DisorderedSystems and Biological Organization"Personnaz L, Guyon I, Dreyfus G (1985) J Phys Lett 46:359Personnaz L, Guyon I, Dreyfus G, Toulouse G (1986) J StatPhys 43:411Rosenblatt F (1958) Psych Rev 65:386Weichbuch G, Fogelman-Soulie F (1985) J Phys Lett 46:624Received: February 26, 1987Accepted in revised form: July 11, 1987Dr. Marie CottrellUA 743 CNRS Statistique Appliqu6eLaboratoire de Math6matique BSt. 425Universit6 Paris XIF-91405 Orsay CedexFrance

Documents

Stability and Attractivity in Associative Memory Networks - Cottrell - En