Strange attractors in chaotic neural networks

IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS—I: FUNDAMENTAL THEORY AND APPLICATIONS, VOL. 47, NO. 10, OCTOBER 2000 1455

Strange Attractors in Chaotic Neural NetworksLuonan Chen, Senior Member, IEEE,and Kazuyuki Aihara

Abstract—This paper aims to prove theoretically that tran-siently chaotic neural networks (TCNNs) have a strange attractor,which is generated by a bounded fixed point corresponding to aunique repeller in the case that absolute values of the self-feedbackconnection weights in TCNNs are sufficiently large. We providesufficient conditions under which the fixed point actually evolvesinto a strange attractor by a homoclinic bifurcation, whichresults in complicated chaotic dynamics. The strange attractorof -dimensional TCNNs is actually the global unstable set ofthe repeller, which is asymptotically stable, sensitively dependenton initial conditions and topologically transitive. The existenceof the strange attractor implies that TCNNs have a globallysearching ability for globally optimal solutions of the commonlyused objective functions in combinatorial optimization problemswhen the size of the strange attractor is sufficiently large.

Index Terms—Attracting set, chaos, homoclinic orbit, neuralnetwork, snap back repeller, strange attractor, topological tran-sitivity.

NOMENCLATURE

Output of neuron.Internal state of neuron.Connection weight from neuronto neuron .Input bias of neuron.Damping factor of nerve membrane.Self-feedback connection weight.Connection weight matrix.Steepness parameter of output function ( ).Self-recurrent bias.Composition of a map with itself times.Jacobian matrix of at a point .Determinant of at a point .Closed ball in of radius centered at apoint .Transpose of a column vector.Usual Euclidean norm of a vector or the in-duced matrix norm of a matrix.Zero matrix with appropriate dimensions.Local stable set (or manifold) of a fixed point

.Local unstable set (or manifold) of a fixed point

.Global stable set of a fixed point.Global unstable set of a fixed point.Term in the order of .

Manuscript received October 19, 1999; revised May 17, 2000.L. Chen is with the Faculty of Engineering, Osaka Sangyo University, Daito,

Osaka 574-8530, Japan (e-mail:[email protected]).K. Aihara is with Graduate School of Complex Science and Engineering,

University of Tokyo, Tokyo 113-8656, Japan, and CREST, Japan Science andTechnology Corporation (JST), Kawaguchi, Saitama 332, Japan.

Publisher Item Identifier S 1057-7122(00)08344-6.

Attracting set of .A fixed point of .Schwarzian derivative for with respect to .Empty set.

I. INTRODUCTION

T RANSIENT CHAOTIC NEURAL NETWORKS(TCNNs) have rich spatio-temporal dynamics with var-

ious coexisting attractors, not only of fixed points, but also ofperiodic and even more complex attractors [6]–[8]. Numericalcomputations have verified that TCNNs have a very high abilityto find globally optimal solutions for combinatorial problems,although they are deterministic models without stochasticfluctuations [8]. To examine the characteristics of TCNNs, thelocal convergence and the generic bifurcation conditions havebeen analyzed in [9] where the relations between the asymp-totically stable points and the minima of an objective functionhave also been investigated in detail for both asynchronousand synchronous update schemes. In addition, existence of thetopologically chaotic structure (or formal chaos) in TCNNs hasbeen theoretically proven [9] with the Marotto Theorem [22].In [11], we have shown that TCNNs have aglobal attractingset, which encompasses all the global minima of an objectivefunction when certain conditions are satisfied, thereby ensuringthe global searching of TCNNs. Furthermore, the conditionsthat a transversal homoclinic orbit is included in an attractingset are also provided in [11]; this property is generally viewedas important evidence for the existence of a strange attractor.

However, from the viewpoint of dynamical system theory, anattracting set can generally contain multiple attractors, and someparts of an attracting set may not be actually attracting. Althoughit has been numerically observed that TCNNs have complex dy-namics with positive Lyapunov exponents in [8] and [9], andfurther [11] has proven the existence of a global attracting setwith a transversal homoclinic orbit, it is not theoretically clearwhether or not TCNNs have a strange attractor. If a strange at-tractor identical to the attracting set exists, such an attractor en-ables TCNNs to escape from local minima and search for glob-ally optimal solutions of optimization problems.

As a part of a continuing series of works to elucidate the prop-erties of TCNNs following [8], [9], and [11], this paper aims toprove theoretically that TCNNs have a strange attractor, whichcovers all global minima for the commonly used objective func-tion [9], [11] when certain conditions are satisfied. The strangeattractor defined in this paper not only has a complicated topo-logical structure, such as existence of a homoclinic orbit andtopological transitivity, but also is asymptotically stable. In ad-dition, the system is unpredictable because of the sensitive de-pendence on initial conditions.

1057–7122/00$10.00 © 2000 IEEE

1456 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS—I: FUNDAMENTAL THEORY AND APPLICATIONS, VOL. 47, NO. 10, OCTOBER 2000

This paper is organized as follows: In the next section, sev-eral definitions for an attracting set and a strange attractor arebriefly described. In Section III, we first give a general formof TCNNs, and then summarize the previous results. In SectionIV, we prove that the one-dimensional (1–D) version of TCNNshas a strange attractor, which is asymptotically stable. Section Vfirst shows that the global unstable set of a repeller in a-dimen-sional TCNN is topologically transitive, and then proves thatthere exists a strange attractor in TCNNs, whose size is suffi-ciently large such that all minima of the objective function areinvolved. Section VI describes two numerical examples. Finally,in Section VII we make some general concluding remarks.

II. NOTATION

The following notation is used throughout the paper. Definea discrete-time system as

(1)

where , and .Let denote the composition of with itself times,

then a point is a -periodic point if butfor . If , that is, ,

is called a fixed point. Let and be theJacobian matrix of at a point and its determinant,respectively. Let be a closed ball in of radius( 0) centered at the point . Furthermore, let be the trans-pose of a column vector . Define to be the usual Euclideannorm of a vector or the induced matrix norm of a matrix. Let

stand for empty set. is the zero matrix with appropriatedimensions. means that this term is in the order of. Aneighborhood of a point or a set is an appropriate openset including or , respectively. A small letter stands for ascalar, and a capital letter represents a vector, a matrix or a setin this paper.

The local stable and unstable sets (or manifolds) of a hyper-bolic fixed point are defined for a neighborhoodof where

is locally invertible for , as follows:

as

and

as

and

Then, we have the definitions of the global stable and unstablesets as follows:

as

An invariant set is a subset , such that .Next, we summarize necessary definitions [5], [13], [14], [23],used in this paper.

Definition 1: A closed invariant set is called an attractingset, if there is some neighborhoodof , such that

and as , for all . The setas is the domain (or basin)

of attraction of , which is an open set.Note that the attracting set is the largest invariant set in,

which contains not only due to , but also otherinvariant sets in , such as fixed points, periodic points, etc.

Definition 2: The -limit set of for is the set ofaccumulation points of the forward orbit as ,or .

We use Lyapunov stability to describe the dynamical stabilityfor an invariant set.

Definition 3: A closed invariant set is Lyapunov stable if,for every open neighborhoodof , there exists an open neigh-borhood of such that for all nonnegative .

Definition 4: A closed invariant set is said to be asymptot-ically stable, if it is Lyapunov stable and there is an open neigh-borhood of , such that for all .

Obviously, an attracting set is asymptotically stable becauseit is the largest invariant set in the region.

Definition 5: A closed invariant set is topologically tran-sitive if for any two open sets , , a positive integerexists such that , or equivalently a pointexists such that its orbit is dense in.

If a set is a topologically transitive set, is indecompos-able.

Definition 6: An orbit starting at a point iscalled homoclinic to the fixed point if as

. Moreover, if and intersect transversallyat , the orbit is called a transversal homoclinic orbit.

Next, we give a definition for sensitive dependence on initialconditions.

Definition 7: is sensitively dependent on initial conditionson an invariant set if there exists such that, forany and any neighborhood of , there existsand such that .

Then, we have the definitions of the attractor and the strangeattractor [14], [13], [23].

Definition 8: An attractor is an attracting set, which istopologically transitive. An attractor is strange, if it containsa transversal homoclinic orbit and is sensitively dependent oninitial conditions.

Topological transitivity means indecomposability for an in-variant set. It is evident that an attractor in Definition 8 is bothasymptotically stable and indecomposable. Therefore, attractorsdefined in this paper are asymptotically stable attractors. Theyare sometimes referred to as open-basin attractors, although thisterminology may be misleading [5]. To describe the topologicalproperties of chaos based on Marotto Theorem [22], next wealso give the definition of a snap-back repeller.

Definition 9: A fixed point is said to be a snap-back re-peller of (1) if there exists a real number( 0) and a snap-backpoint with , such that all eigenvaluesof exceed unity in norm for all and

with for some positive in-teger .

In this paper, the dynamics of is based on syn-chronous updating [8], [9].

CHEN AND AIHARA: STRANGE ATTRACTORS IN CHAOTIC NEURAL NETWORKS 1457

III. T HE TRANSIENTLY CHAOTIC NEURAL NETWORK AND

ATTRACTING SET

In this section, we briefly summarize several previous theo-retical results as well as the TCNNs.

A. A Model of Chaotic Neural Networks

The model of the TCNNs can be described in terms of scalarvariables as follows [6]–[9]:

(2)

(3)

where all variables and parameters are real numbers andoutput of neuron;internal state of neuron;connection weight from neuronto neuron ;input bias of neuron;damping factor of nerve membrane;steepness parameter of the output function ( );self-feedback connection weight;self-recurrent bias of neuron.

Originally, is a function of in TCNNs [8], [9], [11]. Forthe sake of the simplification of analysis, we assume that all theself-feedback connection weights are fixed and take the samevalue in this paper, i.e., for .

Therefore, TCNNs can simply be rewritten in terms of vectorvariables as follows:

(4)

where

......

......

......

Clearly, in (4) is a class function. If andare constants, (4) coincides with the original chaotic neural net-works (CNNs) [4], [8], [9]. For more particular cases, iffor all , TCNNs reduce to the difference equational version ofthe Amari–Hopfield neural networks [3], [17], [18], on the basisof the Euler method; if , and are constants,TCNNs are equivalent to the mean field equations (mean fieldIsing model) [10], [24].

B. Formal Chaos

Let

where for and 0 for . Let

which is independent of and bounded becausedue to . Then, (2) or (4) can be

rewritten as

(5)

In [11], the following theorem on the existence of topologicalchaos for TCNNs in the sense of Marotto [22] has been proven(see [11, Theorem 8]).

Theorem 1: Assume for andthat is sufficiently large. Then, (4) has a fixed point

which can be expressed asfor . If the parameters and

further satisfy one of the following three conditions for each:

1) ;2) ;3) ;

then is a snap-back repeller and (4) has a transversal homo-clinic orbit for . In other words, (4) is chaotic in the sense ofMarotto [22], i.e., there exist

A) a positive integer such that for each integer, (4) has -periodic points;

B) a scrambled set, i.e, an uncountable setcontainingno periodic points such that

a)b) for every and any periodic point of (4)

c) for every , with

C) an uncountable subset of such that for every ,

Strange attractors in Definition 8 mainly emphasize the prop-erty of attracting (or asymptotic stability) for chaos. In contrast,


Marotto’s theorem describes chaos from the topological view-point. Actually, the Marotto chaos is a generalization of theLi–Yorke chaos in a 1-D difference equation [20] to multi-di-mensional systems, for which there exists a topological struc-ture, including a scrambled set and infinitely many periodicpoints as indicated in A, B, and C of Theorem 1.

Remark 1: Assume the conditions of Theorem 1. Then, thereexists at least one snap-back point such that .

C. Attracting Set with Homoclinic Orbits

Theorem 1 ensures that TCNNs have a complicated structure,including a scrambled set and a transversal homoclinic orbit.However, the Marotto chaos is formal chaos, which may not benumerically observed when the scrambled set or the homoclinicorbit is not appropriately included in an attractor. The followingtheorem [11] gives sufficient conditions under which a homo-clinic orbit is at least contained in an attracting set.

Theorem 2: Assume for andis sufficiently large. Assume that, and satisfy one of thefollowing three conditions for each

1) and;

2) and ;3) and .

Then, there is an attracting setsatisfying for (4)and containing the transversal homoclinic orbits of, where

•for condition

1 or 2;•

for condition 3.

IV. STRANGE ATTRACTORS INONE-DIMENSIONAL TCNN

Although the points representing the system behaviorshould eventually converge to an attracting set, there may existattractors, which are smaller sets in the attracting set, i.e., an at-tracting set may contain one or several attractors. In this sec-tion, extending the results of Theorem 2, we will first showthat the homoclinic orbits of are actually included in an at-tractor, which is indecomposable, by proving the topologicaltransitivity for 1-D version of TCNNs. Then, in the next sec-tion, we will prove the existence of a strange attractor in-di-mensional TCNNs.

From (2) and (3), a 1-D version of TCNNs can be interpretedas a single neuron model, and is described by

(6)

where , and all parameters arereal numbers.

A. Negative Schwarzian Derivative in

The Schwarzian derivative plays an important role in this sec-tion to show the topological transitivity for 1-D maps.

Definition 10: The Schwarzian derivative of a functionat is

where .Remark 2: If , then

1) for any positive integer ;2) cannot have a positive local minimum or a negative

local maximum.

The proof of Remark 2 is given in [12]. Then, from (6), we havethe following theorem [26].

Theorem 3: If either or, then .

Proof of Theorem 3:From the calculations of ,and

where

and .Therefore, if , then

thereby by noting .In the same way, it can be proven that if

.Theorem 3 and Remark 2 indicate that for all positive

integer has negative Schwarzian derivative for allprovided that the conditions of Theorem 3 are satisfied.

B. Strange Attractor in

In the following, we show that there is a strange attractor for1-D TCNNs. Fig. 1 qualitatively shows mapping offor sat-isfying either condition 1 or condition 2 of Theorem 2 where

in (6). Next, we will prove that the attracting setin TCNNs is actually topologically transitive provided that theSchwarzian derivative is negative and .

In Fig. 1, the extreme points and of forsatisfy , i.e.,

(7)


Fig. 1. Mapping of a 1-D TCNN and subintervals. Repellery; J = [f(y ); y); attracting set� = [y ; y ]; global unstable setW (y) = �.

where

and

Hence, the minimal value and the maximal value ofin the attracting set are

(8)

(9)

Clearly, the attracting set is the closed interval, which is equal to . Theorem 1

and Remark 1 indicate that there exists a snap-back repellerand a snap-back point , such that , as shown inFig. 1. and are the two distinct inverse maps

(or preimages) of within the attracting set. andare also two distinct inverse maps (or preimages)

of within the attracting set.Theorem 1 and Definition 9 show that there is a positive

number , such that all eigenvalues of exceed unityin norm for all and , which im-plies that any region in not only is expanded underthe iterations of but also belongs to the local unstable man-ifold. Therefore, , as shown in Fig. 1.In other words, expands any interval in . Let

. maps onto the interval , and fur-ther maps onto the interval and

itself. Hence, some points in are mapped back into by. We define as a “first return map of an interval” on

the interior of by analog of [12, Proposition 11.11], suchthat , where is the smallest integer for which

and . As shown in Fig. 2, there are two in-tervals and in which are mapped onto by

. Hence, for . In Fig. 2,and . No-

tice that there always exists an interval orsuch that covers for any . As long asthe conditions of Theorem 2 hold, the orbit ofstarting fromeventually returns to .


Fig. 2. Mapping of a 1-D TCNN and decomposition.

On the other hand, since the interval belongs to thelocal unstable manifold of , any interval including pointis eventually expanded to cover , which then coversby . Let be the smallest integer for which and

. In other words, is the largest intervalfor which is first mapped onto . Then, the first returnmap of an interval is defined as

(10)

For instance, on .Fig. 3 shows the graph of , where has infinitely many

points of discontinuity [12]. Note that at any point of disconti-nuity or , . Next we will prove afundamental lemma related to .

Lemma 1: Assume condition 1 of Theorem 2 whereand also that . Then, for each when

is sufficiently large.Before proving Lemma 1, we first derive the condition for

, which is the case graphically demonstrated inFigs. 1 and 2. Assume and . Then, accordingto (6) and (9), we have

On the other hand, from Theorem 1,is bounded when . Therefore, when

, holds for sufficiently large .Otherwise, yields for suffi-ciently large . Thus, depending on and , iseither more or less than.

Proof of Lemma 1:Without loss of generality, we considerthe case where (or ), for which

is located to the left of . Note that although, due to condition 1 of Theorem 2. We divide

into four subintervals, i.e.

We only prove the Lemma for the subintervalsince the results for the other three subintervals follow in thesame manner. Let and . For wedefine as

(11)

which is an interval bounded by and . Let bethe time of the first return from onto , where

. Then, maps homeomorphically onto , and mapsto cover . Since the length of

is less than that of , there exists such that


Fig. 3. First return map of an interval for a 1-D TCNN.

. Now must cover . Sinceholds, it is easy to show

(12)

(13)

which imply that the length of is less than that ofprovided that is sufficiently large, noting

. Therefore, it follows that there existswith .

It is easy to check that the conditions of Theorem 3 holdwhen condition 1 of Theorem 2 is satisfied and is suffi-ciently large. Since each has negative Schwarzian derivativefor all positive , the minimum value of occurs at oneof the endpoints of any interval according to Remark 2. There-fore, due to .

To prove that , let be the time ofthe first return from onto , where .Then, noting , we have

(14)

(15)

due to both and for theinterval .

For intervals and ,note that

(16)

due to and (7) [11]. Then, the assumedcondition implies that fromcomparing (12) and (16).

For the case where or ,should be to the right of, because it is easier to construct theintervals and which are all in the right of .

The next lemma shows topological transitivity for the interval.

Lemma 2: Assume condition 1 of Theorem 2 withand . Then, the interval is topologicallytransitive for 1-D TCNNs when is sufficiently large. In par-ticular, for any interval in the basin of attraction of , thereexists a positive integer such that .

Proof of Lemma 2:Let be any interval in . Since anypoint in is eventually mapped to, there is an and asubinterval with . Now expandsthe lengths of intervals in by Lemma 1. Therefore, there isa and a subinterval such that containsa discontinuity point of . In other words, there is ansuch that . On the other hand, sinceis a snap-backrepeller, any neighborhood of is eventually expanded underiterations to cover . In particular there exists a, such that , which

proves topological transitivity.By graphical analysis of Fig. 1, for any intervalin the basin

of attraction of there exists a such that ,because is eventually mapped into. Since any interval inwill cover under iterations, there exists asuch that due to .

Now we are in the position to show the existence of a strangeattractor in 1-D TCNNs.

Theorem 4: Assume and condition 1 of Theorem 2with . Then, there exists a strange attractor which co-


incides with and in 1-D TCNNs, provided thatis sufficiently large.Proof of Theorem 4:It is straightforward to show that

is an attractor, which includes a homo-clinic orbit by considering Theorem 2 and topological transi-tivity of Lemma 2. The sensitive dependence on initial condi-tions is actually proven in Theorem 5 for more general-di-mensional TCNNs.

For 1-D TCNNs, the interval is a strange at-tractor which is asymptotically stable, sensitively dependent oninitial conditions and topologically transitive. In addition, since

is a snap-back repeller, this strange attractor has the structureof Marotto chaos according to Theorem 1.

Remark 3: The condition possibly can be relaxed toin Theorem 4, e.g. by using a different snap-back point

to reconstruct intervals of in Figs. 1 and 2 with a moreappropriate decomposition.

Actually, the numerical simulations have shown that TCNNscommonly have an attractor with a positive Lyapunov exponentwhen for sufficiently large .

Remark 4: [26] has indicated that the original chaotic neuronmodel [4] with is chaotic in Devaney’s sense [12]under certain conditions. Theorem 4 has shown how to extend[26] by providing concretely sufficient conditions in a more de-tailed way.

V. STRANGE ATTRACTORS IN -DIMENSIONAL TCNN

Recall the -dimensional TCNN of (5)

(17)

where in (17) is the th element of for the -dimen-sional TCNN.

In this section, we will focus on analyzing the dynamics of theth element of . Hence, we define two different 1-D TCNNs

associated with theth element of

(18)

(19)

where and with respect toand of (5) or (17). We assume that all parameters satisfyconditions of Theorem 4 with .

Then, iterating (17)–(19) from any initial condition , wehave

(20)

(21)

(22)

According to condition 1 or 2 of Theorem 2, the global un-stable set and the attracting set of are covered by

. Therefore, any point in orcan be expressed by where .

A. Recurrent Points in

We will concentrate on the dynamics of theth element ofgiven by (17), i.e., , to compare with and .

Note that the dynamics of in this sectionis not solely dependent on theth equation of (17), but also on

for and , due to the interconnectionterm .

Lemma 3: Assume that for each ,for an initial

condition [i.e., where]. Then, for each

and sufficiently large

1) , and can be expressed as follows:

(23)

(24)

(25)

where and ;2) .Lemma 4: Assume that

first becomes equal to zero at for an initialcondition with the th element bounded[i.e., ]. Then, there exist and

such that for sufficientlylarge , where is a bounded fixed point.Moreover whereand .

Lemma 3 and Lemma 4 are proven in Appendix A. Next, wepresent the main lemma of this paper, by using Lemma 3 andLemma 4.

Let intervals and be the basins of attracting sets forand , respectively. Define an interval

where , which is a restrictedsubset of for . When condition 1 or 2 of Theorem 2 issatisfied and is sufficiently large, according to [11]

(26)

Let be any open set. Define

(27)

which is an interval for theth element of while otherelements where are in . Note that .

Lemma 5: Assume and condition 1 of Theorem 2.Then, for any open set , there exist andsuch that for all sufficiently large .

Proof of Lemma 5:Let be any open set andbe an interval for the th element of in defined

in (27). Then, and from (26). Let a


point where.

Case-1: Without loss of generality, assume (seethe following Case-2 for the case with , i.e.,

where ). Ifwith an initial condition for all , then

for all ac-cording to Lemma 3. Hence, according to Lemma 7 of Ap-pendix B, due to and

for . On the other hand, sinceand are 1-Dmodels and and , according to Lemma 2 thereexists a positive integer such that andcover all of the bounded regions (invariant sets). In particular

and because or is boundedaccording to Theorem 1. Therefore

which implies that there exists a such thatfor all sufficiently large .Case-2: If first be-

comes equal to zero for with an initial condition ,then can be expressed as

from (20), where is independent of . Letbe a new initial condition where

. Then, due to, and for . Therefore we

have the following two cases:Case-2.1: If with an

initial condition for all , then this is the sameas Case-1, but with the different initial condition . FromLemma 2 and analogous to the proof of Case-1, there exists

such that for all sufficiently large . Inother words, which impliesthat there exists a such that for allsufficiently large .

Case-2.2: If first be-comes equal to zero for with the initial condition

. According to Lemma 4, there exist andsuch that and

for all sufficiently large . That is, or

Therefore, we conclude that there exists a such thatfor all sufficiently large .

Taking completes theproof.

B. Strange Attractor in

From the results of the previous subsection, we first show thatis topologically transitive, and then give sufficient con-

ditions for the existence of a strange attractor in-dimensionalTCNNs.

Lemma 6: Assume and condition 1 of Theorem 2.Then

1) there exists a integer such that forany open set ;

2) is topologically transitive;

in -dimensional TCNNs when is sufficiently large.Proof of Lemma 6:Let be any open set in

and .Then, according to Lemma 5, there exist a and a

such that for all sufficiently largeand each , respectively. Hence, byletting for all sufficiently largeand each . Therefore, we have for each

. In other words,

(28)

which implies that any open set in eventually covers .On the other hand, since is a snap-back repeller, any neigh-borhood of is eventually expanded under iterations to coverthe global unstable set including . In particular, thereexists a such that

, which proves topological transitivity.In general, the topologically transitivity of a set does not

imply that its orbits are also sensitively dependent on initial con-ditions [12]. Therefore, next we give a proof for sensitive depen-dence on initial conditions.

Theorem 5: Assume and condition 1 of Theorem2. Then, is sensitively dependent on initial conditions on

.Proof of Theorem 5:From the conditions of this theorem,

there is an unstable fixed point in . According to 1of Lemma 6, for any point where is anopen set containing , there exists an integer such that

. Without loss of generality, assume .Therefore there exists and , such that

.Since is an unstable fixed point, which has an unstable

local set (manifold) , any point on the unstable localset , including , is eventually expanded under itera-tions along the unstable set of. Therefore, noting that

, there exists and an integer such that, which proves the theorem.

Theorem 6: Assume and condition 1 of Theorem 2.Then, there exists a strange attractor in-dimensional TCNNs,which is equal to and , provided thatis sufficiently large.

Proof of Theorem 6:Theorem 2 has ensured that there isan attracting set , which is stable. Now we show that there isonly one attractor in , namely . Assume that thereis an attractor , but does not include . Then,

because and are different invariantsets which are both topologically transitive. On the other hand,according to Lemma 5 and (28), for any open set

, there exists such that . Since, we have which contradicts the

assumption. Therefore, there is only one attractor in, which isaccording to Lemma 6, i.e., .

It is straightforward to show that a transversal homoclinicorbit is actually included in by Theorem 2, and that issensitively dependent on initial conditions on by The-orem 5.


The strange attractor in this paper has the following proper-ties.

Remark 5: Assume that is the strange attractor of satis-fying the conditions of Theorem 6. Then

1) is asymptotically stable;2) is topologically transitive on ;3) is chaotic in the sense of Marotto;4) is sensitively dependent on initial conditions on.In Remark 5, 1 and 2 can be easily derived from the def-

initions of asymptotical stability and a strange attractor; 3 isfrom Theorem 1; and 4 describes the unpredictable property ofa strange attractor, which is proven in Theorem 5. Therefore,for -dimensional TCNNs, the global unstable set isa strange attractor which is asymptotically stable, sensitivelydependent on initial conditions and topologically transitive ac-cording to Remark 5. In addition, due to the snap-back repeller, this strange attractor has the structure of Marotto chaos, ac-

cording to Theorem 1. Since the strange attractor in TCNNsis actually , which contains all of the bounded region,there are sufficiently large snap-back points for the-di-mensional system, which number is equal to the combinationnumber of -binary variables for the combinatorial optimiza-tion problem [11].

In applications of neural computation, a Lyapunov function,or a computational energy function, plays an important role, be-cause minima of such an energy function correspond to possiblesolutions of combinatorial optimization problems and storedpatterns of associative memory, as demonstrated in [16]–[18]. Itis shown in [11] that TCNNs have the following objective func-tion as a computational energy function undercertain conditions with :

(29)

where the third term of the right-hand side of (29) can be ignoredwhen is sufficiently small.

Unlike locally convergent dynamics reined by a Lyapunovfunction [11], globally chaotic dynamics of TCNNs shown inthis paper is not restricted to any local area, but able to visitevery possible state in the strange attractor, i.e., TCNNs havethe global searching ability. For instance, a TCNN can be usedas an optimizer to solve a combinatorial optimization problembecause the strange attractor of (2) and (3) encloses all globalminima of when is sufficiently large. Furthermore,such global dynamics of the strange attractor also providea basis for dynamical associative memory, which can searchall of stored patterns or generate rich spatio-temporal patternscomposed of spatial patterns stored with synaptic connectionweights [1], [2].

VI. NUMERICAL EXAMPLE FOR STRANGE ATTRACTOR

In this section, 1-D and 2-D TCNNs are used for numericalsimulation to verify our theoretical results.

Fig. 4. Internal statey(t) of the neuron and the Lyapunov exponent withincreasingj!j in a 1-D TCNN. (a) Internal statey(t) of the neuron and (b) theLyapunov exponent.

Example 1: Consider a 1-D case of (6). Let, and . Fig. 4 shows the

internal state of the neuron and the Lyapunov exponent withincreasing .

The parameter values of this example have been chosen tosatisfy the conditions of Theorem 4. Therefore, there exists astrange attractor when is sufficiently large due toTheorem 4. According to Theorem 1 or (8) and (9), the in-terval of the strange attractor is approximately equal to the in-terval , i.e., . Thesnap-back repeller whose global unstable set constructs theattractor, should be in a neighborhood of

from Theorem 1.In this example, when , the bounded chaotic be-

havior constantly appears, which is numerically verified by pos-itive Lyapunov exponents in Fig. 4. The interval for is alsoproportional to where the orbit of seems tobe densely scattered. Here, the Lyapunov exponentis definedas follows:

(30)

For the numerical simulation, we use sufficiently large itera-tion number to approximate (30). In this example, each Lya-punov exponent, with respect to the fixed, is calculated by (6)and (30) for , after discarding the first 200 iterationsto eliminate the influence of initial values.

The parameter values of this example also satisfy both con-dition 1 and condition 2 of Theorem 1, which means that thereare at least four sequences of the transversal homoclinic orbits[11]. As a matter of fact, it was numerically found thatas wellas its homoclinic orbit is in the attractor.

According to (29), the objective functionof this example is

(31)


Fig. 5. Internal states(y (t); y (t)) of neurons and the maximum Lyapunov exponent with increasingj!j in a 2-D TCNN. MLE : Maximum Lyapunov Exponent;t = 10000.

where anddue to small and . The global

minimum of (31) is

for variable or for variable .When is sufficiently large

for variable , which is clearly in the interval of the strangeattractor for .

Example 2: Consider a 2-D case of (2) and (3). Let

. Fig. 5 shows theinternal states of neurons and the maximum Lyapunov exponentwith increasing .

The parameter values of this example satisfy condition1 of Theorem 2. Therefore, there exists a strange attractor

, when is sufficiently large due to Theorem 6.According to Theorem 2, the attractor is covered bythe region , i.e.,

. The snap-back

repeller , whose global unstable set constructs the strangeattractor, should be in a neighborhood of

from Theorem 2. Furthermore, when is sufficiently large,there are eight repellers, which locate in neighborhoods of

respectively [11].In this example, when , the bounded chaotic be-

havior constantly appears, which is numerically verified by pos-itive maximum Lyapunov exponents. In Fig. 5, MLE means theMaximum Lyapunov Exponent, which is numerically calculatedwith algorithm proposed by Shimada and Nagashima [28].


The parameter values of this example also satisfy both con-dition 1 and condition 2 of Theorem 1, which means that thereare at least four sequences of the transversal homoclinic orbits.

In addition, [8], [9], and [11] also provided several one and2-D examples of TCNNs, which demonstrate the dynamicalproperties, such as asymptotical stability, local bifurcations andnumerical chaos.

VII. CONCLUSION

It has been theoretically proven in this paper that TCNNshave a strange attractor, which generates spatio-temporal andunpredictable dynamics by proving the topological transitivityin its global unstable set. TCNNs have a globally searchingability, which has been numerically observed in combinatorialoptimization problems [8]. Furthermore, this paper gives suf-ficient conditions for existence of a strange attractor, whichensures that the neural networks carry out the global search. Asignificant characteristic of TCNNs is that the strange attractoris constructed from a bounded fixed point corresponding toa unique repeller by homoclinic bifurcations, which generatescomplex behavior of neural networks when the absolute valueof the self-feedback connection weight in TCNNs is sufficientlylarge. In addition, this paper has also shown that the strange at-tractor not only contains the transversal homoclinic orbits withthe structure of Marotto chaos, but also is sensitively dependenton initial conditions, which results in complicated chaotic be-havior. Moreover, the strange attractor in TCNNs is asymptoti-cally stable. These theoretical results are useful for applicationsof TCNNs to practical problems, e.g., combinatorial optimiza-tion [8], [9], [11], and dynamical associative memory [1], [2],because they indicate that the attractor of the neural networkscovers all the bounded regions, including all minima of thecommonly used objective function [11]. Although this paperfocuses only on the analysis on TCNNs, theoretical results canalso be applied to more general discrete-time recurrent neuralnetworks (DRNNs) [9], [11].

APPENDIX ARECURRENTORBIT

In this Appendix, we will focus on analyzing the dynamicsof the th element of , and use the same notation as SectionV. Using (17), we will concentrate on the dynamics of thethelement of , i.e., , to compare with and . Notethat the dynamics of in this Appendixis not solely dependent on theth equation of (17), but also on

for and , due to the interconnectionterm .

Proof of Lemma 3:Letwhere , then

For the case of , from (17)

(32)

(33)

As the same manner, from (18) and (19) we have

(34)

(35)

where which is nonzero by the as-sumption, and , and

. Hence,due to . Therefore the theorem holdsfor .

Suppose that this theorem holds for , i.e.

(36)

where andor .

Then, let us show that it is also true for . When ,from (17)

(37)

(38)

where (36) is substituted.Since and

, then is given by follows

(39)

for sufficiently largeTherefore

(40)

Hence, .In the same way, and can also be exactly ex-

pressed by (40) for the first and third terms. However, for thesecond term, and

. Therefore, , and all have the sameterm related with which is also nonzero according to the as-sumption of the theorem and (20)–(22). Furthermore,

due toand , which means that

.


Actually,and inLemma 3, according to (21) and (22).

Proof of Lemma 4:If first be-comes equal to zero for , then from(20) where is a bounded real number. Furthermore

for eachwhere and are bounded real numbers independent of

and according to Lemma 3.Next, we use Urabe’s proposition (Appendix C) to prove the

theorem. Let

(41)

Then

(42)

Since both and are bounded real numbers,

(43)

which means that (or ) is an approximate so-lution of for sufficiently large . Therefore, wetake as the 1-D algebraic equation with variable

. Then, by (41)

(44)

(45)

due to from (17).For and each

, it follows that

(46)

However, for because is a boundedreal number.

Therefore due to , we have, i.e.

which implies that there exists a positive numbersuch that

for any sufficiently large . Furthermore, for a positive number, there exists a positive numbersuch that

for all

Therefore, there exists a such that, and according to (43), holds for

all sufficiently large . Then, the following three conditionshold for all sufficiently large :

1) ;2)

3) , where and.

According to Urabe’s Proposition (see Appendix C or [29]),has the unique solution in for all

sufficiently large , That is, for all sufficiently large

(47)

where .In addition to the bounded fixed point , we can show

that there exist and suchthat for sufficiently large , where

is any bounded point of , as faras the conditions of Lemma 4 are satisfied. Moreover

where and.

APPENDIX BINTERSECTION OFSETS

Lemma 7: Assume that andall belong to for , and

for each . Then, .Proof of Lemma 7:Without loss of generality, assume that

is not empty. Let be any point in. Then, there exist two points and

such that . Now, we show that .By assumption, and . In

other words, and where .Since is continuous on , there must exist a suchthat , or , thereby proving thislemma.

APPENDIX CURABE’S PROPOSITION

Let be a continuously differentiable functionin some region . Assume that the equation

has an approximate solution for which isnot zero, and there is a constant and a constantsuch that

1) ;2) for any ;3) ;

where and are positive numbers such thatand .

Then, has one and only one solution inand .


ACKNOWLEDGMENT

The authors thank Prof. S. Amari for his encouragement ofthis work and the anonymous referees for many valuable com-ments on the manuscript.

REFERENCES

[1] M. Adachi and K. Aihara, “Associative dynamics in a chaotic neuralnetwork,”Neural Networks, vol. 10, no. 1, pp. 83–98, 1997.

[2] , “An analysis of instantaneous stability of an associative chaoticneural network,”Int. J. Bifurcation and Chaos, vol. 9, no. 11, pp.2157–2163, 1999.

[3] S. Amari, “Characteristics of random nets of analog neuron-like ele-ments,” IEEE Trans. Syst., Man, Cybern., vol. SMC–2, pp. 643–657,1972.

[4] K. Aihara, T. Takabe, and M. Toyoda, “Chaotic neural networks,”Phys.Lett. A, vol. 144, no. 6/7, pp. 333–340, 1990.

[5] J. Buescu, Exotic Attractors from Liapunov Stability to RiddledBasins. Basel: Birkhaüser Verlag, 1997.

[6] L. Chen and K. Aihara, “Transient chaotic neural networks and chaoticsimulated annealing,” inToward the Harnessing of Chaos, M. Yamaguti,Ed. Amsterdam, The Netherlands: Elsevier, 1994, pp. 347–352.

[7] , “Chaotic simulated annealing for combinatorial optimization,” inDynamic Systems and Chaos, Aoki et al., Eds. Singapore: World Sci-entific, 1995, vol. 1, pp. 319–322.

[8] , “Chaotic simulated annealing by a neural network model withtransient chaos,”Neural Networks, vol. 8, no. 6, pp. 915–930, 1995.

[9] , “Chaos and asymptotical stability in discrete-time neural net-works,” Physica D, vol. 104, pp. 286–325, 1997.

[10] L. Chen, H. Suzuki, and K. Katou, “Mean field theory for optimal powerflow,” IEEE Trans. Power Syst., vol. 12, no. 4, pp. 1481–1486, 1997.

[11] L. Chen and K. Aihara, “Global searching ability of chaotic neural net-works,” IEEE Trans. Circuits Syst. I, vol. 46, no. 8, pp. 974–993, 1999.

[12] R. L. Devaney,An Introduction to Chaotic Dynamical Systems, 2nded. Reading, MA: Addison-Wesley, 1989.

[13] J. P. Eckmann and Ruelle, “Ergodic theory of chaos and strange attrac-tors,” Rev. Mod. Phys., vol. 57, no. 3, pp. 617–656, 1985.

[14] J. Guckenheimer and P. Holmes,Nonlinear Oscillations, Dy-namical Systems, and Bifurcations of Vector Fields. New York:Springer-Verlag, 1983.

[15] M. W. Hirsch and S. Smale,Differential Equations, Dynamical Systems,and Linear Algebra. New York: Academic, 1974.

[16] J. Hopfield,Neural Networks and Physical Systems with Emergent Col-lective Computational Abilities: Proc. Nat. Acad. Sci., 1982, vol. 79, pp.2554–2558.

[17] , Neurons with Graded Response Have Collective ComputationalProperties Like Those of Two-state Neurons: Proc. Nat. Acad. Sci., 1984,vol. 81, pp. 3088–3092.

[18] J. Hopfield and D. Tank, “Neural computation of decisions in optimiza-tion problems,”Biol. Cybern., vol. 52, pp. 141–152, 1985.

[19] Y. A. Kuznetsov,Elements of Applied Bifurcation Theory. New York:Springer-Verlag, 1995.

[20] T. Y. Li and J. A. Yorke, “Period three implies chaos,”Amer. Math.Monthly, vol. 82, pp. 985–992, 1975.

[21] C. M. Marcus and R. M. Westervelt, “Dynamics of iterated-map neuralnetworks,”Phys. Rev. A, Gen. Phys., vol. 40, pp. 501–504, 1989.

[22] F. R. Marotto, “Snap-back repellers imply chaos inR ,” J. Math. Anal.Appl., vol. 63, pp. 199–223, 1978.

[23] C. Mira, L. Gardini, A. Barugola, and J. Cathala,Chaotic Dynamics inTwo-Dimensional Noninvertible Maps. Singapore: World Scientific,1995.

[24] C. Peterson and J. R. Anderson, “A mean field theory algorithm forneural networks,”Complex Systems, vol. 1, pp. 995–1019, 1989.

[25] C. Peterson and B. Soderberg, “Artificial neural networks,” inModernHeuristic Techniques for Combinatorial Problems, C. R. Reeves,Ed. London, U.K.: Blackwell Scientific, 1993.

[26] E. A. Razafimanantena, “Analysis of the dynamics of Aihara’s model ofchaotic neural networks,”, preprint, 1997.

[27] D. E. Rumelhart, J. L. McClelland, and PDP Research Group,ParallelDistributed Processing: MIT Press, 1986, vol. 1 & 2.

[28] I. Shimada and T. Nagashima, “A numerical approach to ergodicproblem of dissipative dyanmical systems,”Prog. Theor. Phys., vol. 61,pp. 1605–1616, 1979.

[29] M. Urabe, “Galerkin’s procedure for nonlinear periodic systems,”ARCH. Rat. Mech. Anal., vol. 72, pp. 121–152, 1965.

Luonan Chen (M’94–SM’98) received the B.S.E.E.degree from Huazhong University of Science andTechnology, Wuhan, China, in 1984, and the M.E.and Ph.D. degrees electrical engineering fromTohoku University, Sendai, Japan, in 1988 and 1991,respectively.

He was with KCC Ltd. until 1997, when he joinedthe Faculty of Osaka Sangyo University, Osaka,Japan, where he is currently Associate Professorin the Department of Electrical Engineering andElectronics. His fields of interest are nonlinear

dynamics, optimization, neural networks, and chaos.

Kazuyuki Aihara received the B.E. degree in elec-trical engineering in 1977 and the Ph.D. degree inelectronic engineering 1982 from the University ofTokyo, Tokyo, Japan.

Currently, he is Professor in the Departmentof Complexity Science and Engineering andDepartment of Mathematical Engineering andInformation Physics, the University of Tokyo. He isalso Chairman of the Biochaos Research Committeein the Japan Electronic Industry Development Asso-ciation. His research interests include mathematical

modeling of biological neurons, parallel distributed processing with chaoticneural networks, and time series analysis of chaotic data.

Documents

Strange attractors in chaotic neural networks