Adaptability and Diversity in Hiroyuki Iizuka Simulated Turn-taking Behavior€¦ · · 2015-07-29Adaptability and Diversity in Simulated Turn-taking Behavior Hiroyuki Iizuka Takashi

Adaptability and Diversity inSimulated Turn-taking Behavior

Hiroyuki IizukaTakashi IkegamiDepartment of General

Systems SciencesThe Graduate School of Arts

and SciencesUniversity of Tokyo3-8-1 KomabaTokyo [email protected]@sacral.c.u-tokyo.ac.jp

KeywordsTurn taking, adaptive behavior,diversity of behaviors, cognition,embodiment

Abstract Turn-taking behavior is simulated in acoupled-agents system. Each agent is modeled as a mobilerobot with two wheels. A recurrent neural network is used toproduce the motor outputs and to hold the internal dynamics.Agents are developed to take turns on a two-dimensionalarena by causing the network structures to evolve. Turntaking is established using either regular or chaotic behaviorof the agents. It is found that chaotic turn takers are moresensitive in response to inputs from the other agent.Conversely, regular turn takers are comparatively robustagainst noisy inputs, owing to their restricted dynamics. Frommany observations, including turn taking with virtual agents,we claim that there is a complementary relationship betweenrobustness and adaptability. Furthermore, by investigatingthe recoupling of agents from different GA generations, wereport the emergence of a new turn-taking behavior.Potential for synthesizing a new form of interaction is anothercharacteristic of chaotic turn takers.

1 Introduction

Dynamical systems can theoretically simulate behavior produced over time with interac-tions between various entities. This approach, based on embodied cognition [13, 16, 17],has a different perspective from the traditional AI approaches. Namely, representationsare not given as symbols in advance, but are only realized, by the dynamics, over time[1, 18, 21]. Cognitive structure is characterized by geometrical and flow patterns in anadequate phase space. As well as being characterized by attractor types (e.g., fixedpoints, limit cycles, and strange attractors), they are also characterized by chaotic itiner-ancy and other novel concepts, such as open-ended evolution/dynamics, that describetheir inherent behavior.

Beer [1, 2] provides clear dynamical perspectives from the ongoing interaction be-tween brain, body, and environment in simulations of hexapod locomotion, sequencelearning, and categorical perception. The behaviors are characterized by the dynami-cal systems terms with trajectories attracted by point attractors, which are derived fromthe environment dynamics. For learning grammars, Pollack proposed second-orderrecurrent neural networks, which are dynamical systems, as being able to act as dy-namical recognizers for formal languages [18]. The dynamical recognizer can imitatethe behavior of some finite automaton while learning to hold a rich internal dynam-ics, which is characterized by fractal structure. Extending Pollack’s framework, Ikegamiand Morimoto proposed a coupled dynamical recognizer system for analyzing cognitivebehaviors among multiple players [11]. They showed that coupled unstable learning isnecessary for continuing novel interactions as the source of the itinerancy of dynamics.

c© 2004 Massachusetts Institute of Technology Artificial Life 10: 361–378 (2004)

H. Iizuka and T. Ikegami Adaptability and Diversity in Simulated Turn-taking Behavior

This chaotic itinerant dynamics is discussed for communications and predictions amonghumans. In this way, investigating the structure of the state space of coupled equa-tions (e.g., environment-body-nerve, environment-agent, or agent-agent) is the mostimportant use of dynamical systems.

Richness and the potential of the dynamical systems approach encourage us to gobeyond merely adaptive behavior. The higher functions, such as intention, motivation,emotion, and consciousness, are within the scope of this kind of approach. Walterstarted the discussion of emotional and playlike behavior by synthesizing artificial crea-tures [25, 26]. A wheeled vehicle containing a simple electric circuit can show unex-pected and complex behavior, comparable to that of living creatures. Without makingreal robots, Braitenberg made conceptual robots to discuss the higher functions [3]. Inhis thought experiments, he designed vehicles using simple hardwired electrical con-nections from sensory inputs to motor outputs. His vehicles gradually showed morecomplex cognitive behaviors to start with as they were provided with more complexinternal structures. For example, his most primitive behavior is a sense of “aggression,”which is simply given by attraction to a light source with a crossed sensory-motor con-nection. However, to simulate more complex behavior, such as association and conceptformation, he has to implement new wires, such as “mnemotorix” and “ergotorix” wires,with some Darwinian-type selection. Walter and Braitenberg have one thing in com-mon in claiming that any apparently complex cognitive behavior can be built up fromsimple sensory-motor coordination. That is, agents can be rendered cognitive by im-posing physical constraints. We basically agree that any meaningful cognition shouldbe embodied, but focus on different aspects.

In this article, we try to understand the cognitive behaviors of turn taking and imita-tion, caused by interactions between two or more humans, for which it is thought thatthe sharing of mental states and intentions with others is important. There are manyways to understand psychological phenomena by computer simulations and robot ex-periments rather than by studying human behavior directly [5, 6, 20]. We conductedcomputer simulations of two agents with internal dynamics which were implementedby an artificial recurrent neural network as a model of turn-taking behavior. In ourprevious works, cognitive behaviors were explained from the dynamical systems per-spective by coupling between agents with rich internal dynamics [9–11]. Here, wegeneralize from turn-taking behavior to autonomous role changing, as in games of tagamong children, and investigate the generic underlying mechanisms using the dynami-cal systems method. Therefore, this study focuses on different perspectives from thoseof fixed role-playing games (e.g., a pursuit-evasion game [4]). Here we take turn takingas the simplest example that shows the diversity of dynamics. For turn taking behaviour,it is necessary for roles to be exchanged autonomously within a context constructed bythe entities’ behaviors, such as chaser-evader and speaker-listener. When taking turnsin a two-person conversation, people usually avoid overlapping or interrupting eachother’s speech without any explicit cues to switch speakers. Some cues for this includeeye contact and the detection of intonation changes. It is considered that turn takingis established by coordination between predictions and the internal neural networksthat compute the output from the inputs. Therefore, coupling between agents meansa coupling of anticipatory systems with intrinsic dynamics.

By introducing neural architecture, an evolutionary algorithm, and a turn-takinggame in Sections 2 and 3, we explore four topics in the simulation. The first topic isdynamic repertoire. We describe how turn taking is established with different formsof interaction. In particular, we argue in Section 4.1 that regular interaction evolvesinto chaotic behavior. The second topic is predictability. Each agent has to predict theother’s behavior one step ahead. Interestingly, prediction precision decreases whenthe turn-taking role switches from one player to the other. This will be discussed in

362 Artificial Life Volume 10, Number 4


Section 4.2. The third topic is ongoingness of interactions. Agents become robustagainst sensor noise; however, the turn-taking performance is established only whenagents synchronize their dynamics precisely. This point is discussed in Section 4.3.The last topic is adaptability. As discussed in the section on dynamic repertoires, theturn-taking pattern appears to be different for different evolutionary generations. InSection 4.4, we investigate the emergence of new spatiotemporal patterns by couplingagents from different generations. In Section 5, we discuss a possible linkage betweenthese simulation results and the psychological experiments conducted by Trevarthen[23]. A concept of intersubjectivity is also discussed.

2 The Model

We modeled the playing of a tag game in which the role of chaser or evader is notgiven to players in advance. There are some other game models in which the rolesare not prespecified. Reynolds also showed that the abilities of chasing and evadingevolve simultaneously by genetic programming in a game of tag, which is a symmetricalpursuit-evasion game [19]. The variety in the behavior of agents adapting to their en-vironments is worth noting. In Reynolds’ game, switching between evader and chaseris specified as happening when both agents come into physical contact. The differ-ence between Reynolds’ model and ours is in the spontaneous emergence of behavior.Whether an agent plays the role of a chaser or an evader is dynamically determined inour model. On the other hand, Di Paolo modeled and studied social coordination withagents interacting acoustically [7]. To avoid misperceiving the acoustical signals, theiremission timings were entrained in an antiphase state; the resulting behavior resemblesa turn-taking process.

There is a difference between Di Paolo’s turn taking and ours. Both turn-taking be-haviors are established by the coordination of agents through their interactions. How-ever, Di Paolo modeled turn taking as the result of antiphase signals to avoid signalinterference, whereas we model it as a result of coupling between richer internal dy-namics. Therefore, in this article we pay more attention to the diversity of behaviorpatterns.

2.1 Game and EnvironmentHere each agent has a circular body of radius R , with two diametrically opposed motors(Figure 1). The motors can move the agent forward in a two-dimensional unstructuredand unlimited arena. The motion is described by the following equation of motion interms of an agent’s heading angle (θ) and its speed (v) in that direction:

Mv̇ + D1v + f1 + f2 = 0, (1)

I θ̈ + D2θ̇ + τ( f1, f2) = 0, (2)

where f1 and f2 are the forward driving forces, and τ denotes the torque. D1 andD2 express the resistance coefficients, and the agents have mass (M ) and inertia (I ).We solve the equations iteratively using the Runge-Kutta method. At each time step,the agents compute the forces from the inputs using the internal neural nets describedbelow.

We assume there is no collision between agents, because we focus on the internalstates of the agents that generate turn taking. Two agents try to coordinate their turn-taking behavior, each trying to get behind the other. Because they cannot get behindeach other simultaneously, the turn-taking cannot be achieved if both agents play

Artificial Life Volume 10, Number 4 363


Figure 1. Left: A schematic view of the mobile robot with two wheels (actuators). It computes the forward forcevector and the torque strength from the force vector (f1, f2) on each actuator. Right: Two mobile robots interactto perform turn-taking behavior by sensing each other’s position, relative distance, and heading angle. It is robot A’sturn when A enters the area that is B’s rear side (RS) position. The shape of this RS is parameterized by r and φ.(We set r = 150, φ = 30 in our simulation.)

chaser. Naturally, mutual turn-taking cannot be achieved if both agents play evadereither. Therefore, it is necessary to have spontaneous symmetry breakdown so that oneplays the role of chaser and the other plays the role of evader. However, mere symmetrybreakdown is insufficient: temporal role changing is also required. By using recurrentneural networks, we focus on how the turn-taking dynamics are self-organized.

2.2 Agent DesignWe designed the agents to have recurrent neural networks (Figure 2). Inputs to an agentare the other agent’s position, distance, and heading angle, relative to the agent. Agentsmove freely in the arena using two motors, the outputs of which are computed at everytime step of the game. Each agent predicts the other’s next relative position, which isassigned to three output neurons. The dynamics of the recurrent neural network areexpressed by the following equations at each time step t :

hj (t) = g

(∑i

wij yi(t) +∑

l

w ′l j cl (t − 1) + bj1

), (3)

zk(t) = g

(∑j

ujkhj (t) + bj2

), (4)

cl (t) = g

(∑l

u′j l hj (t) + bj3

), (5)

g(x) = 1

1 + e−x, (6)

where yi, zk, hj , and cl represent input, output, hidden, and context nodes, respectively.The context nodes encode internal context information. They do not receive directinput from the external context. The respective numbers of nodes in these layers areset to (I , K , J, L) = (3, 5, 10, 3) throughout this article. The symbols wij , ujk, w ′

l j , andu′

j l denote the weights from input to hidden, hidden to output, context to hidden, andhidden to context neurons, respectively, and the parameter b is a bias node. This



Figure 2. Recurrent neural networks with three layers. Input nodes receive the other agent’s relative position. Thefinal layer consists of three types of node: context, prediction, and motor output. Context nodes feed back to theinput layer. Prediction nodes output the other’s relative position in the next time step. Motor nodes output theforce vector (f1, f2).

network architecture evolves using a genetic algorithm as explained in the followingsection.

3 Genetic Algorithm and Noisy Environment

3.1 Genetic AlgorithmWe update the weights according to the turn-taking performance. In practice, theweight set of the neural networks has a vector representation of the real weight values,which evolve using a genetic algorithm (GA).

We use a GA to evolve two separate populations, to prevent agents of a single geno-type from dominating, in which case turn taking would be played between geneticallysimilar agents. As a result, a player would have to play against itself, which we wishto avoid. Each population contains P individuals. The performance of all P 2 agentpairs from the separated populations is evaluated at each generation. Agents that cantake turns equally are evaluated as having greater fitness. At first, individuals in eachpopulation are initialized with random weight values. Then we calculate the fitness ofeach individual, based on its performance.

The highest value is given when both agents take their turn alternately and the agentscan predict each other’s behavior. A one-sided (i.e., role-fixed) behavior is associatedwith lower fitness values. Practically, the fitness of an agent a from a population (A)against an agent b from the other population (B) is calculated as follows. Below, wedefine a total fitness F as the sum of two fitnesses associated with prediction and turntaking, respectively (Equation 7). When one agent gets behind the other, by definitionthe other agent has its turn, and the rear scope is specified as RS , which is parameterizedby two parameters r and φ (see Figure 1). The agent in this state is said to be havingits turn and is rewarded (Equations 8 and 9). A spatial position of agent b at time stept is represented by Posb(t). This is compared with agent a’s prediction value Posa→b .Therefore the squared difference (Equation 11) is the measure of the error in agent a’sprediction. As described above, it is undesirable for a one-sided agent to be able totake turns or predict its partner’s behavior. To avoid this, we use the product terms in



Equations 8 and 10, which encourage symmetric fitness rather than one-sided fitness:

Fa = 1

P

P∑ (s1 × F turn

a + s2 × F predicta

), (7)

F turna =

T∑t

ga (t) ×T∑t

gb (t) , (8)

ga(t) ={1, Posa(t) ∈ RSb(t),0, Posa(t) /∈ RSb(t),

(9)

F predicta = −

T∑t

Pa (t) ×T∑t

Pb (t) , (10)

Pa(t) = (Posb(t) − Posa→b(t))2. (11)

The performance of turn taking is evaluated for different lengths of time (T = 500,1,000, and 1,500), so that agents cannot tell when the evaluation time is over. Evaluatingthe turn-taking performance at each GA generation, we leave the best E individuals ineach population (elitism) and reconstruct the other individuals from the best individualswith a point mutation rate m. The GA proceeds by repeating this procedure, and therecurrent neural networks evolve. The ratio of weights s1 and s2 in Equation 7 is 5 : 2through the simulation. However, during the first 100 GA generations, we set s2 = 0,so that the prediction accuracy does not contribute to the fitness during this period.We set the other parameters, P , E , and m, to 15, 4, and 0.1, respectively.

An initial configuration of the agents is determined by the distance and the rela-tive angle between agents, because they sense nothing but each other. For each GAevaluation, the initial distance is fixed at 250, but the head directions of the agents israndomly assigned from 0 to 2π radians.

In addition, the following points should be noted.

3.2 Two Time ScalesTwo time scales exist: the vehicle navigation time scale (�T1), and the neural compu-tation time scale (�T2). The time evolution of the vehicle navigation is computed usingthe fourth order Runge-Kutta method, where �T1 is set to 0.01. The basic process is thatthe neural net receives the sensor inputs and computes the motor outputs. We makethe assumption that the vehicle navigation motion is faster than the internal neural timescale. For simplicity, the neural net produces the outputs every 100 Runge-Kutta timesteps (100�T1 = �T2). When the network structure evolves by GA, the time scale ratiois implicitly reflected in the net structure. Therefore, we believe that the same behaviorstructure can be obtained, at least qualitatively, for a different scale ratio.

3.3 Noisy EnvironmentLiving systems are involved in a fundamentally noisy environment. We know that ourperception has to deal with noisy inputs. However, it is not usually possible to fullydiscriminate noise from other signals. As living systems, we are somehow handlingthe problem. Therefore, we simulated the agents interacting with each other in a noisyenvironment. Noises are added to the input neurons at every time step during each runin the GA. The strength of noise is provided by uniform random numbers between zeroand almost the maximum distance the agent can move during one time step. In thenext sections, spatial patterns of turn taking are studied as simulation results. Unless



otherwise noted, those patterns are generated in a noise-free environment to clarify theintrinsic dynamics of the agents.

4 Simulation Results

Simulation was performed with a GA using 15 individuals. After several thousand GAgenerations, turn taking is established between the two agents. The basic dynamicsof the turn taking was observed as follows. Two agents adjust their speeds and taketurns automatically to switch from the role of evader to chaser and vice versa. In thefollowing subsections, we investigate the realized turn-taking pattern with respect todynamic repertoire, predictability, and adaptability. These issues are related to thenotion of evolvability. (These are well discussed also in [15, 24].)

4.1 Diversity of Dynamic RepertoireFirst, the evolutionary algorithm effectively functions to improve the turn-taking per-formance. The development of the performance as a function of GA generations isdepicted in Figure 3. The resulting turn-taking patterns are sensitive to some of thesettings. In particular, they are sensitive to the division of the agent population intotwo. In previous work, we encoded the pair of agents’ structures on the same gene [8].Then we encoded them separately but used a single population. That algorithm canalso develop turn-taking behavior, but with much less diversity than the present algo-rithm. When the agents are on the same gene, it is difficult to show diversity, as theirnet structures are too correlated. With a single population, development of an agentthat can take turns with itself (and its genetically close relatives) is enhanced. There-fore, there is a strong probability that the dynamics of the turn taking may be tuned forself-turn-taking. To avoid this situation, we used the two-population structure.

Figure 4 shows examples of the spatial trails of an agent from different GA genera-tions with different initial population structures. For the sake of clarity, a single agent’strail is depicted. The paired agent tends to show the same trail with different phases.

0

2

4

6

8

10

12

0 2000 4000 6000 8000 10000 12000

generation

fitne

ss

Figure 3. Fitness values of the best agents in two populations at each GA generation for a single run.



-1000

-500

0

500

1000

1500

2000

2500

-2000 -1500 -1000 -500 0 500 1000 1500 2000 2500-200

-100

0

100

200

300

400

100 200 300 400 500 600 700 800 900

100

150

200

250

300

350

400

450

100 150 200 250 300 350 400 450 500 550 600

(a) (b)

(c) (d)

200

300

400

500

600

700

800

900

1000

1100

1200

-200 -100 0 100 200 300 400 500 600 700-1500

-1000

-500

0

500

1000

1500

-2000 -1500 -1000 -500 0 500 1000

(e) (f)

-6000

-4000

-2000

0

2000

4000

6000

-6000 -4000 -2000 0 2000 4000 6000 8000 10000

(g) (h)

-3000

-2000

-1000

0

1000

2000

3000

4000

0 2000 4000 6000 8000 10000

0

200

400

600

800

1000

-800 -600 -400 -200 0 200 400 600

Figure 4. Spatial trails of turn-taking behavior observed in the simulations. To clarify the qualitative difference ofturn-taking structures, a spatial trail of only one of the two agents is shown. The other agent moves around thesetrails, generating similar trails. All games in these graphs are started from (550, 300). (a) and (b) are examples ofregular turn-taking behavior, while the others are examples of chaotic turn-taking behavior.

We can classify these trail patterns approximately into regular, chaotic, and others,based on their appearance in space and time. When spatial trails consist of regularcurves, and turns are exchanged almost periodically (which corresponds to a turningpoint in the figures), we speak of regular turn taking. On the other hand, if spatial trailshave irregular curves with nonperiodic turn taking, we speak of chaotic turn taking.

In the earlier GA generations, agents with regular turn taking evolve to yield higherperformance (Figure 4a, b). The behavior structure is as follows. One agent followsthe other and passes it; then it slows, as does the other agent; then both agents simul-



-400

-200

0

200

400

600

800

1000

1200

1400

-1500 -1000 -500 0 500 1000 1500 100

150

200

250

300

350

400

450

100 150 200 250 300 350 400 450 500 550 600

(a) (b)

Figure 5. Differences of spatial trails between adaptive agents without noise (solid) and with noise (dotted) areplotted. They start from the same initial points, (550, 300). (a) Chaotic turn taker; (b) regular turn taker.

taneously turn around quickly. This returns the agents to the first phase. A series ofbehavior patterns repeats almost periodically, and the envelope curves of these trailsconstitute a circular pattern. In the later GA generations, more chaotic patterns emerge(Figure 4c–h). In contrast to the regular patterns, the turns are exchanged in differ-ent places at irregular time intervals. Therefore, the spatiotemporal pattern becomeschaotic, and agents move around the entire space.

The evolution of turn-taking type from regular to chaotic is explained as follows. Theevolutionary pressure of the GA at first allows the agents to move stably in the noisyenvironment. A structured turn-taking behavior can only be built up on stable motiondynamics that are insensitive to random noise. As argued briefly in the introduction,noise and “intentional” action are difficult to distinguish when the agents’ motionsbecome chaotic. However, when their actions appear regular, we can conclude thatthe agents can more easily distinguish noise from the other agent’s intentional motion,as they show different performance with and without partners’ adaptive motions (seeSection 4.3). Therefore, the regular type emerges earlier than the chaotic. As shown inFigure 5, regular turn taking occurs at almost the same spatial location with differentnoise series. However, the chaotic type is sensitive to the noise series. The totalperformance of turn taking remains high in both cases.

Thus, a regular turn-taking pattern suppresses a variety of dynamic repertoires. Bydoing so, it becomes robust against sensory noise. On the other hand, a chaotic turn-taking pattern has the potential to develop a dynamic repertoire, and thereby becomemore adaptive, as studied in Section 4.4.

Intuitively, agents that can take turns in the presence of noise can take turns perfectlywithout noise. However, this does not hold for some agents found in later GA gen-erations. Indeed, in Figure 6, agents can take turns only when there is sensory noise.We call this phenomenon noise-induced turn taking. As shown in the figure, there is astrong attraction to a circular motion without exchanging turns. The two agents havedifferent neural structures, and the resulting turn-taking behavior is generally asymmet-rical. Without noise, one agent is never able to take its turn. In addition, it forms anattractor in the sense that adding a small noise cannot break this one-sided behavior.True turn taking only emerges above a certain noise level (Figure 7). In another case,there exist three attractors when there is no sensory noise. In one attractor, agent Afollows the rear side of agent B closely. Another attractor assigns the opposite role toagents A and B; and the last attractor is that both agents chase each other. Each of thethree attractors consists of circular orbits. The transition between attractors is causedby noise. Without noise, agents are trapped by one of the attractors.



without noise with noise

attractor

100

200

300

400

500

600

-200 -100 0 100 200 300 400 500 600 0

100

200

300

400

500

600

700

800

900

-400 -200 0 200 400 600

Figure 6. Noise-induced turn-taking behavior. There is an attractor of role-fixed behavior (left). By adding noise tothe agents, an agent can slip out of the attractor and successfully perform turn taking.

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0 5 10 15 20

strength of noise

perf

orm

ance

of t

urn

taki

ng

Noise level used in evolution

Figure 7. The performance of turn-taking behavior as a function of noise strength. Below a certain noise level,agents cannot perform turn taking. Above that level, agents take advantage of noise to perform turn taking. Thecritical noise level is lower than that used in evolution.

Compared with these noise-induced behaviors, chaotic turn takers can spontaneouslyestablish turn-taking behavior without noise. Even if noise is introduced into the system,chaotic turn takers can establish turn-taking behaviors independent of the low noiselevel. That is, they do not utilize noise, but suppress the effect of noise to perform turntaking. In contrast, noise-induced turn takers need noise to perform turn-taking.

4.2 Prediction Capability and Role SwitchingThese observations were analyzed in terms of the prediction capability of agents. Theagents, after thousands of GA generations, are able to predict their partners’ futuremovements while turn taking. Three outputs of the recurrent network simulate the otheragent’s future location and heading from the current input. Figure 8 shows the precisionof predictions and the associated turn-taking patterns. In earlier GA generations, oneagent’s prediction is far better than the other’s. In later generations, both predictions areimproved. However, through entire GA generations, the predictions almost periodicallybreak down when the turns (roles) are exchanged. As indicated in the figure, theprediction is also perturbed by noisy inputs. However, the effect is much smaller thanthat of the other agent’s action.

It should be noted that these prediction outputs are not designed explicitly to do any-thing in generating action sequences. However, because they depend on the common



-1

-0.9

-0.8

-0.7

-0.6

-0.5

-0.4

-0.3

-0.2

-0.1

0

1000 1100 1200 1300 1400 1500 1600 1700 1800

game time step

perf

orm

ance

of p

redi

ctio

n

-1

-0.9

-0.8

-0.7

-0.6

-0.5

-0.4

-0.3

-0.2

-0.1

0

1000 1100 1200 1300 1400 1500 1600 1700 1800

perf

orm

ance

of p

redi

ctio

n

game time step Agent A Agent B

Figure 8. Prediction (top spiky lines) and turns (line segments) are drawn for each agent from 10,000 GA generations.A horizontal line expresses time steps for two agents moving in the two-dimensional arena. The top two line segmentscorrespond to turns of the coupled agents. The bottom segments correspond to times when neither agent has aturn. This shows that the prediction precision decreases sharply when a turn is switched.

context neurons that also control the motion patterns, simulating each other’s behaviorand generating the motor outputs have indirect correlations. The correlation betweenprediction breakdown and the turn-taking performance will be reported elsewhere.

4.3 Ongoingness of InteractionsThe inherent adaptability of each turn-taking pattern can be studied using its stabilityin the presence of noise. In other words, we study an agent’s ability to discriminatebetween noise and the adaptive behavior of the other agent. In this section, we comparethe behavior of live interaction with recorded interaction. The live interaction is normalinteraction between evolved agents, and the recorded interaction is that between anagent and a virtual agent, defined below.

First, we selected the best two agents, A and B, from each population. Turn takingbetween these agents was studied without noise. This is what we term “live interaction.”The trails of the agents were recorded during the run. Then, turn taking between agentA and the recorded trail of agent B (a virtual agent) was conducted. This is what weterm “recorded interaction.” We perturb the recorded trail and simulate the changes inthe turn-taking dynamics.

Figure 9(a) shows the growth of a discrepancy between A with virtual B and Awith perturbed virtual B (chaotic turn takers). During the initial few hundred steps, nodiscrepancy was observed. The behaviors are similar, as shown in the figure. However,a small noise was amplified and the orbit drastically changed from the original orbitat approximately 800 time steps. In terms of the turn-taking behaviors, the adaptiveagent can no longer recover harmonization with the perturbed virtual agent. The agentapproaches the trail and tries dynamically to resume the original turn-taking behavior.

Another example (the agents at 3,000 generations) is shown in Figure 9b. Theseagents established regular turn taking. In this case, the agents could cope with the per-turbed virtual agent. Note that agents that have achieved regular turn-taking behaviordo not always, but frequently do, have a tendency to cope with a perturbed virtualagent, although this varies with the timing and strength of the perturbation. Some-times turn-taking behavior breaks down when more noise is added to the recordedtrail. However, there are some examples in which turn taking recovers after a periodof discrepancy.



Figure 9. Differences of orbits between agents’ trails in a game with an adaptive agent and a recorded trail. A smallnoise is introduced at 340 time steps. If there is no noise, no difference is observed. Agents used in (a) and (b)correspond to those in Figure 4e and 4a, respectively. The difference is amplified if agents fail to establish turn taking.

4.4 Evolution of AdaptabilityAnother novel feature of adaptability was examined. We show here that adaptability cangenerate novel dynamics by constructing new couplings. We examine the behaviors ofnew couplings between two agents from different GA generations as follows. After theturn-taking performance had attained a satisfactory plateau, we selected two individualsfrom different generations to play. This was to examine how they performed turn takingwithout having the common experience of coevolution. Taking agents from generations10,000 and 3,000 as examples, we evaluated the performance of the new pairs foreach generation (Figure 10). In fact, the novel pairs often failed to sustain the sameperformance as the original pairs. However, the synthesized dynamics often showednovel structures. The examples can be found in Figure 11. Agents that perform chaoticturn taking after 10,000, 8,000, and 7,000 generations (Figure 11a,c,e) are coupled withagents from each different generation. As is seen in the figure, the newly coupled agentsalso show chaotic turn taking, but with a different kind of motion pattern (Figure 11d).Coupling an agent from generation 10,000 and one from generation 7,000, or an agentfrom generation 8,000 and one from generation 7,000, shows a similar pattern to thatby the agents from generation 7,000, which is shown in Figure 11b,f.

Figure 10. The best agents from the 3,000 (left) and the 10,000 (right) GA generations are examined with regardto coupling them with the best agents from different GA generations. The performance of turn taking of the newlycoupled pair is evaluated for each generation. Generally the performance is lower than the original performance ofthe best pair from each generation, which is approximately 0.6.



-1000

-500

0

500

1000

1500

-1000 -500 0 500 1000 1500 2000 2500

200

300

400

500

600

700

800

900

1000

1100

1200

-400 -200 0 200 400 600 800

-1000

-500

0

500

1000

-2000 -1500 -1000 -500 0 500 1000

100

200

300

400

500

600

700

800

-200 -100 0 100 200 300 400 500 600

-4000

-3000

-2000

-1000

0

1000

2000

-5000 -4000 -3000 -2000 -1000 0 1000 2000 3000 4000

-400

-300

-200

-100

0

100

200

300

400

500

-400 -200 0 200 400 600 800

(a)

(f)(e)

(c) (d)

(b)

Figure 11. Spatial trails of the original pairs and newly coupled agents at 7,000, 8,000, and 10,000 GA generations.(a), (c), and (e) show the trails of the original pairs at 10,000, 8,000, and 7,000 generations, respectively. On theright, newly coupled agents’ trails are shown. The arrows indicate pairing of partners from different generations.(b), (d), and (f) are generated by the best agents at 10,000 versus 7,000, 10,000 versus 8,000, and 8,000 versus 7,000generations, respectively. (b) and (f) are similar to the trails generated by the original paired agents from generation7,000. On the other hand, (d) shows a new trail.

In summary, (i) novel structures sometimes inherit the original pattern of one of theagents, but not always, (ii) agents that readily exhibit chaotic turn-taking patterns losethe original pattern and adapt to the other agent’s pattern, and (iii) conversely, regularturn takers simply retain their original pattern and show little adaptability to a newpartner.

The last point is clearly shown in Figure 12. The regular turn takers can only achievehigher performance with agents from near generations (Figure 10). Our hypothesis isthat chaotic turn takers are more adaptive than regular ones. The observation hereconfirms the hypothesis, but we should note that performance sometimes differs sig-nificantly between populations A and B from the same GA generation. Figure 13illustrates how turn-taking performance varies from generation to generation. We de-duce from this figure that they are basically symmetrical for populations A and B.Sometimes there are notable exceptions—for example, population A from generations8,000–10,000 compared with population B from generations 10,000–12,000. It shouldalso be noted that genetically closer agents can collaborate better than more distantlyrelated agents. Qualitatively however, beyond generation 6,000 agents become moreadaptive than those of earlier generations.

The turn-taking patterns result from the collaboration of two agents. Therefore, aneural structure in the body of a single agent alone cannot explain the turn-takingdynamics. This is an interesting part of the present study, but at the same time, a gap



-1200

-1000

-800

-600

-400

-200

0

200

400

600

-500 0 500 1000 1500

300

350

400

450

500

550

600

650

700

750

100 200 300 400 500 600 0

5000

10000

15000

20000

25000

30000

35000

40000

-9000 -8000 -7000 -6000 -5000 -4000 -3000 -2000 -1000 0 1000

-2000

-1000

0

1000

2000

3000

-3000 -2000 -1000 0 1000 2000 3000 4000

(a) (b)

(c) (d)

Figure 12. Spatial trails of new couplings of regular and chaotic turn takers: (a) 3,000 versus 7,000, (b) 3,000 versus8,000, (c) 3,000 versus 10,000, (d) 3,000 versus 27,280. A spatial trail of the regular turn taker is shown here. Oneagent always chased the partner, and role changing did not occur. Convergence of agents’ sensors and motors causesthe decrease in behavioral diversity and the interruption of role changing for turn taking.

between the two agents may develop. That is, when one population becomes veryadaptive against many others, it is not necessary for the other population to becomevery adaptive; it may simply become a test data set for the former population to becomeuniversal turn takers. As far as we know, such universal turn takers are yet to evolve.Here we notice that a chaotic turn taker is better at eliciting coordinated behavior fromthe partner.

It is also worth noting that the evolutionary history, or experience, of two agentsinteracting with each other is a prerequisite for better turn taking. The experience ofhow agents have collaborated to perform turn taking determines with whom an agentcan take turns. In new pairs, responses of one agent to the other often occur at thewrong time, whereas the original pairs show complete synchronization of turn taking.That is, we infer that it is not the neural structure but the collaboration of timing andpatterning that is responsible for the better turn-taking behavior. This may be truenot only for this special type of interaction—that is, turn taking—but for cognitiveinteraction in general. We will argue this point in the final section of this article.

5 Discussion

It was found in the virtual-agents experiment (Section 4.3) that chaotic turn takersare much more sensitive to the difference between live and recorded inputs. Theirturn-taking patterns are driven by the ongoing interaction. On the other hand, regularturn takers are relatively insensitive to the difference. As evolution continues, chaoticturn taking replaces regular turn taking in the GA simulations. This may be due toregular turn takers’ being less adaptive than chaotic turn-takers in the sense that theycan only cope with fewer agents. This is clearly seen in the new coupling experimentof Section 4.4. The coupling with regular turn takers only generates circular patterns,



0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0 2000 4000 6000 8000 10000 12000 0

2000

4000

6000

8000

10000

12000

generation (population A)

gene

ratio

n (p

opul

atio

n B

)

perf

orm

ance

of t

urn-

taki

ng

Figure 13. The performance in turn taking by new couplings with the best agents from all generations of twopopulations. Higher performance is represented by darker color. Beyond 6,000 generations, patterns change fromregular to chaotic. The agents after 6,000 generations show a tendency to be able to perform turn taking withagents from different generations.

but chaotic turn takers show various patterns. In summary, we claim that the spatialpatterns of chaotic turn takings are less robust in the presence of noise the patterns buthave more adaptability than the patterns of regular turn takers.

This complementary relationship between adaptability and robustness has also beendisplayed in some empirical experiments [12]. We compare the present result withTrevarthen’s double-monitor experiments between a baby and its mother [22, 23], andwith Nadel and Revels’ mutual-imitation experiments [14]. In Trevarthen’s experiment,mother and baby (infant) only communicate through videos that display their faces toeach other. It is reported that the mother’s response to a baby has to be of the rightstyle and the right timing. Otherwise, the baby does not engage with her mother. Forexample, if a recorded video of the mother was displayed to the baby, the baby becamewithdrawn and depressed. This corresponds to our simulations of turn taking betweena real agent and a virtual one. We stressed that an agent must become sensitive tothe ongoing dynamics of the other agent in order for them to be mutually adaptive inturn-taking behavior. Nadel studied how a mutual imitation game progresses betweenchildren. Children regularly switch between the roles of imitating and being imitatedby having new imitation patterns. This corresponds to the behavioral diversity of thechaotic turn takers in our simulation. The behavioral diversity of each agent providesnovelty to the other agent.

Trevarthen’s experiments show that it is not necessarily important for the infant thatthe mother be displayed on the monitor. It can be assumed that the most importantclue during interactions is the ongoing anticipation of a partner. The infant performs



some actions and anticipates the mother’s reactions reflecting the infant’s actions, andthis is also true with respect to the mother’s anticipation of the infant. Interactionsin social behavior, including turn taking, can be established when these anticipationsare mutually formed dynamically. Furthermore, it is shown by Nadel’s experiment thata non-affordant way of using objects can maintain interaction—that is, some form ofnovelty/unpredictability is required. In our simulations, when an agent calculates out-puts, this calculation simultaneously affects the internal dynamics. That is, the actionsperformed form its internal dynamics as much as actions form anticipations. The agentreceives a partner’s actions as inputs that reflect the agent’s own actions. We main-tain that turn taking is established when these structures are mutually organized. Turntaking is therefore broken in the simulation with virtual agents. However, our simula-tions also show that unpredictability is found when turn taking occurs. We thereforeclaim that mutually adaptive coupling of actions and internal dynamics between agentsis essential for the establishment of cognitive interaction, which may be related tointersubjectivity.

It is widely observed in communication systems ranging from play to turn takingthat temporary transitions occur between each salient phase and the next. We callsuch itinerant behavior chaotic itinerancy. Nonrecursivity and open-endedness areexpected in lively communication, and indeed this can be taken as an example ofchaotic itinerancy [11]. In addition to the notion of chaotic itinerancy, we here stress thenotion of adaptability. Simulating turn-taking performance with and without chaoticinstability revealed that adaptability was suppressed in the latter case. According toTrevarthen’s pioneering work and Nadel and Revels’ recent work, we conclude thatturn taking is a cooperative and cocreative activity of the participants. That is, turntaking is not a simple entrainment of two coupled nonlinear pendulums, but requiresmutual adaptation of the participants and has to be maintained intentionally. Studyingtransitions from regular to chaotic turn taking, we interpret them as establishing thecooperative activity first and the cocreative activity second, because agents that showchaotic turn taking can create new turn-taking patterns with other agents.

According to our dynamical systems approach, we don’t claim that human com-munication is a simple physical phenomenon. Rather, we claim that it is complexand that sophisticated ways of interaction are still difficult to simulate. However, thenotions of adaptability and chaotic itinerancy, for example, characterize some facetof human communication. Simulating physiological phenomena can lead to the dy-namical systems approach itself. We now try to formalize open dynamical systems,that is, a dynamical system in an ever-changing environment. We hope creatingnew dynamical systems will lead to a deep understanding of human communicationsystems.

AcknowledgmentsWe thank Ryoko Uno and Gentaro Morimoto for their fruitful discussions. This work ispartially supported by grants-in-aid (No. 09640454 and No. 1310950) from the Ministryof Education, Science, Sports, and Culture, and also by a grant-in-aid from the 21stCentury COE (Center of Excellence) program (Research Center for Integrated Science)of the Ministry of Education, Culture, Sports, Science, and Technology, Japan.

References1. Beer, R. D. (1997). The dynamics of adaptive behavior: A research program. Robotics and

Autonomous Systems, 20, 257–289.

2. Beer, R. D. (2003). The dynamics of active categorical perception in an evolved modelagent. Adaptive Behavior, 11(4), 209–243.



3. Braitenberg, V. (1984). Vehicles: Experiments in synthetic psychology. Cambridge, MA: MITPress.

4. Cliff, D., & Miller, G. F. (1996). Co-evolution of pursuit and evasion II: Simulation methodsand results. In Animals to Animats 4 (pp. 506–515). Cambridge, MA: MIT Press.

5. Dautenhahn, K. (1995). Getting to know each other—Artificial social intelligence forautonomous robots. Robotics and Autonomous Systems, 16, 333–356.

6. Dautenhahn, K. (1999). Embodiment and interaction in socially intelligent life-like agents.In C. L. Nehaniv (Ed.), Computation for metaphors, analogy and agents, Vol. 1562 ofSpringer Lecture Notes in Artificial Intelligence (pp. 102–142). New York: Springer-Verlag.

7. Di Paolo, E.A. (2000). Behavioral coordination, structural congruence and entrainment in asimulation of acoustically coupled agents. Adaptive Behavior, 8(1), 25–46.

8. Iizuka, H., & Ikegami, T. (2002). Simulating turn-taking behaviors with coupled dynamicalrecognizers. In R. K. Standish, M. A. Bedau & H. A. Abbass (Eds.), Proceedings of ArtificialLife 8 (pp. 319–328). Cambridge, MA: MIT Press.

9. Ikegami, T., & Taiji, M. (1998). Structures of possible worlds in a game of players withinternal models. Acta Polytechnica Scandinavica Mathematics and Computer ScienceSeries, 91, 283–292.

10. Ikegami, T., & Taiji, M. (1999). Imitation and cooperation in coupled dynamical recognizers.Advances in Artificial Life (pp. 545–554). New York: Springer-Verlag.

11. Ikegami, T., & Morimoto, G. (2003). Chaotic itinerancy in coupled dynamical recognizers.Chaos: An Interdisciplinary Journal of Nonlinear Science, 13(3), 1133–1147.

12. Ikegami, T., & Iizuka, H. (2004). Simulated turn-taking and dynamics repertoire. InC.L. Nehaniv & K. Dautenhahn (Eds.), Imitation and social learning in robots, humans andanimals: Behavioural, social and communicative dimensions. Cambridge, UK: CambridgeUniversity Press.

13. Marocco, D., & Floreano, D. (2002). Active vision and feature selection in evolutionarybehavioral systems. In J. Hallam, D. Floreano, G. Hayes, & J. Meyer (Eds.), From Animals toAnimats 7 (pp. 247–255). Cambridge, MA: MIT Press.

14. Nadel, J., & Revel, A. (2003). How to build an imitator? In K. Dautenhahn & C.L. Nehaniv(Eds.), Proceedings of Second International Symposium on Imitation in Animals andArtifacts (AISB’03) (pp. 120–124).

15. Nehaniv, C. L. (2003). Evolvability. Special issue. BioSystems, 69, 77–81.

16. Nolfi, S., & Floreano, D. (2000). Evolutionary robotics: The biology, intelligence, andtechnology of self-organizing machines. Cambridge, MA: MIT Press.

17. Pfeifer, R., & Scheier, C. (1999). Understanding intelligence. Cambridge, MA: MIT Press.

18. Pollack, J.B. (1991). The induction of dynamical recognizers. Machine Learning, 7, 227–252.

19. Reynolds, C.W. (1995). Competition, co-evolution and the game of tag. In R. Brooks & P.Maes (Eds.), Artificial Life IV (pp. 59–69). Cambridge, MA: MIT Press.

20. Scassellati, B. (1999). Imitation and mechanisms of joint attention: A developmentalstructure for building social skills on a humanoid robot. In C.L. Nehaniv (Ed.), Computationfor metaphors, analogy and agents, Vol. 1562 of Springer Lecture Notes in ArtificialIntelligence (pp. 176–195). Springer-Verlag.

21. Tani, J. (1996). Model-based learning for mobile robot navigation from the dynamicalsystems perspective. IEEE Transactions on Systems, Man and Cybernetics Part B, SpecialIssue on Robot Learning, 26(3), 421–436.

22. Trevarthen, C. (1977). Descriptive analyses of infant communicative behaviour. In H. R.Schaffer (Ed.), Studies in Mother-Infant Interaction. London: Academic Press.

23. Trevarthen, C. (1993). The self born in intersubjectivity: The psychology of an infantcommunicating. In U. Neisser (Ed.), The perceived self (pp. 121–173). Cambridge, UK:Cambridge University Press.



24. Wagner G. P., & Altenberg, L. (1996). Complex adaptations and the evolution ofevolvability. Evolution, 50(3), 967–976.

25. Walter W. G. (1950). An imitation of life. Scientific American, 182(5), 42–45.

26. Walter W. G. (1951). A machine that learns. Scientific American, 185(2), 60–63.


Documents

Adaptability and Diversity in Hiroyuki Iizuka Simulated Turn-taking Behavior€¦ · · 2015-07-29Adaptability and Diversity in Simulated Turn-taking Behavior Hiroyuki Iizuka Takashi