Generic Architecture for Reactive Animation

Technical Report

David Harel and Yaki SettyThe Weizmann Institute of Science, Rehovot, Israel

January 21, 2007

This technical report describes in detail a specific mechanism for linking anynumber of tools, including an animation tool and reactive system engines ofdifferent kinds. We illustrate the technique and the underlying principles bylinking up the Rhapsody tool, which supports statecharts and object diagrams,and the Play-Engine, which supports live sequence charts (LSCs), to 3D GameStudio (3DGS) for animation and MATLAB for mathematical analysis. The tworeactive engines, Rhapsody modeling in a state-based spirit and the Play-Enginein a scenario-based spirit, are connected up through a central mechanism withthe 3DGS animation tool, and this mechanism is general enough to support andmaintain any number of such links. Moreover, we also illustrate that reactiveanimation can be applied in a three-dimensional setting too. We also describe indetail how the architecture is employed for a 3-participant gym training system.

1 Reactive Animation

Reactive Animation (RA) [4] is a technology aimed at combining state-of-theart reactivity with state-of-the-art animation. RA links the effort of reactivesystem design and the front-end design by bridging the power of tools in thetwo separate areas. In essence, RA has two arms: One comprises powerful toolsand methods for reactive systems development, the heart of which is a rigorousspecification of the systems reactivity. The other comprises powerful animationtools to represent the specification as an intuitive, controllable, animated front-end. The animated front-end serves as a communication channel for betterhuman understanding of the simulation. Technically, RA is based on the viewthat says that a system is a closely linked combination of what it does and whatit looks like. From this stem two separate but closely linked paths: reactivebehavior design and front-end design.

Implementing a tool for RA poses a number of challenges: accuracy, per-formance (e.g., CPU usage, memory management, smoothness of the resultinganimations), distribution, ease of interaction, openness and platform indepen-dence. These should be considered at the architecture level and with the specific

1

functionality chosen.

2 Implementation Architecture

The architecture of our running example includes two reactive engines support-ing two different modeling approaches: a state-based, intra-object approach(which we shall denote symbolically by ST), and a scenario-based, inter-objectapproach (denoted by SC). The reactive engines are linked up to a three-dimensional animated front-end (denoted by FE), using a central routing server(denoted by CRS). This architecture can be extended by using any number ofadditional components of any related kind (e.g., a mathematical analysis tool,such as Matlab) (see Fig. 1).

The various architectural components are organized in a star topology, com-municate through a TCP socket and transmit XML based messages[13], in away similar to [11]. Each component is equipped with a communication plug-in(CPI), which is responsible for the TCP communication and the parsing of in-coming and outgoing messages. The CPI makes no changes to its component.Rather, it runs as an independent thread to enable modularity, extendabilityand efficiency. In addition, using a TCP connection enables a fully distributed,platform-independent implementation.

Below we describe each of the architectural components and its CPI imple-mentation in more detail.

2.1 The Central Routing Server (CRS)

In our architecture, the CRS was implemented as a multi-threaded executableapplication (written in C++). Each thread of the CRS serves as a CPI for onearchitectural component. A TCP socket is initialized upon registration to theCRS, enabling message transmission. The messages are XML documents of apredefined form: the header of the documents carries the source and destination,while the function to be generated is coded in the document’s body. The CRSanalyzes the header of a received message and directs it to its final destination(see Fig. 1). We have added a viewer (written in Visual Basic) that displays theinter-component communication by highlighting message transmission. More-over, the user can track message transmission on the CRS log file.

Elsewhere we discussed InterPlay, a previous effort we carried out to set up astar-like topology for connecting reactive system engines [2]. InterPlay requiresa 2-way mapping for message transmission between each pair of communicatingengines. This is less flexible and requires the CRS to recognize the participatingcomponents.

2.2 The State-Based Specification (ST)

In our example we use the language of statecharts [5] and the Rhapsody tool[9] to implement ST. Statecharts are naturally suited for the specification of

2

objects that have clear internal behavior, an attribute we call intra-object. To-gether with object model diagrams, they provide a graphical representation ofthe dynamics of objects using states, transitions, events, and conditions [7]. Thelanguage makes it possible to visualize the behavior of an object in a way thatemphasizes the elements in its life-cycle. Rhapsody is a model-driven develop-ment environment supporting statecharts and object model diagrams (see [7, 9]),and can be viewed also as a UML tool [12]. It enables object-oriented design,with full execution of the statechart-rich models, and full code generation.

To enable communication of ST with external components (e.g., our CRS),the ST model can contain specially designated sets of incoming and outgoingstates. An incoming state features a set of reactions that can be triggered byexternal events ( i.e., a message that was received in the CPI). Similarly, anevent is sent from ST to an external component (e.g., the CRS) upon enteringan outgoing state. The destination and the contents of such a message areencoded inside the appropriate outgoing state itself.

The CPI for ST was implemented as an external library (written in C++)that extends Rhapsody to support communication and XML parsing.

2.3 The Scenario-Based Specification (SC)

We use the language of LSCs, live sequence charts [3] and the Play-Engine[8] to implement SC. LSCs are scenario-based, and inter-object in nature, andare particularly fitting for describing behavioral requirements. LSCs extendclassical message sequence charts (MSCs) with logical modalities, depicted ashot and cold elements in the charts. The language thus achieves far greaterexpressive power than MSCs, and is comparable to that of temporal logic [10].In particular, LSCs can specify possible, mandatory and forbidden scenarios,and can be viewed as specifying multi-modal restrictions over all possible systemruns.

An LSC typically contains a prechart and a main chart. The semanticsis that if the scenario in the prechart executes successfully, then the systemis to satisfy the scenario given in the main chart. An LSC can also containscoped forbidden elements, listed in a separate area beneath the main chart. Forexample, a hot forbidden condition that becomes true within its scope causes therequirements to be violated. The Play-Engine is the tool built to support LSCs.It enables a system designer to capture behavioral requirements by ‘playing in’the behavior of the target system, and to execute the specified behavior by‘playing out’. In the play-out phase the user or an external component (e.g.,CRS) executes the application as if it were the real system.

To enable communication of SC with external components, the SC modelcan contain specially designated sets of incoming and outgoing LSCs. A messagereceived in SC triggers an incoming LSC, which initiates a particular scenario.Likewise, SC sends a message to an external component (e.g., the CRS) whenthe system execution triggers an outgoing LSC.

The CPI for SC was implemented as a Visual Basic application, and is partof the model’s graphic interface.

3

2.4 The 3D Animated Front-End (FE)

FE was implemented using a three dimensional authoring tool (3D Game Studio(3DGS) [1]), which supports real time rendering of 3D animation. In 3DGS,objects can have associated actions, which appear as part of its attributes. Thescripting language of 3DGS, C-Script, enables control of animation objects (e.g.,changing an attribute). It is a simplified version of C++, and supports objectoriented programming.

Here too, to enable communication, FE can contain specially designatedsets of incoming and outgoing C-Script functions. FE sends a message upontriggering an outgoing function by the user (or an external component), and areceive message triggers an incoming function in FE that generates an animationchange. This is the heart of the link between the animation and the reactivedriving engines.

The CPI for FE was implemented as an external dll, extending the C-scriptlanguage to support communication and XML parsing.

We choose to add a controlling GUI to our system as part of FE (as wasdone also in the thymus model [6, 4]). However, in more complex systems sucha GUI could very well complicate the FE and should be designed as a differentarchitectural component, possibly using a GUI-building tool.

2.5 Mathematical Analysis GUI:

We designed a mathematical analysis GUI using MATLAB from MathWorks(http://www.mathworks.com/), which is a high-level language and interactiveenvironment for computational tasks. This GUI generates various graphs andstatistics based on data received from the simulation. The CPI for this GUIwas implemented as an additional script and supports communication and XMLparsing.Similar mathematical analysis tools such as Mathematica from Wol-fram Research, can also be plugged-in the architecture.

3 An Example: A Team Training Model

To demonstrate our architecture, we chose a simple, yet representative, modelof a gym team in training. The model implements FE, ST and SC, by using3DGS, Rhapsody and the Play-Engine, respectively. These were integrated asdescribed in the previous section (see Fig. 2).

3.1 Elements of the Model

Our model describes gym training sessions for a team of three: a team leaderand two team members. The team leader performs various actions at differentspeeds, and the team members follow suit, after a short ”comprehension” delay.Team members, however, are not as fit as the team leader, and need to restwhile performing certain fast actions. In addition, the team leader reacts to

4

environmental changes; when a Storm, a Flood, or a Volcano eruption occurs,the team leader performs an appropriate set of actions to handle the situation.

Three cameras cover the training scene. The main camera is focused on theteam leader, and each of the other cameras covers a team member. The userinstructs the team to perform an action or assigns it a training task. He or shecontrols the cameras’ activity and may query the running simulation.

3.2 Modeling Team Behavior Using Statecharts

We used the state-based approach (ST) to specify the team’s actual behavior.Our Rhapsody model includes three classes: the team leader, the team memberand the team. We demonstrate ST using two team members, however, the modelcan be easily extended to handle any number thereof.

The behavior of the team leader is specified by a statechart with four or-thogonal components (i.e., concurrent sub-statecharts); see Fig. 4. The mostimportant of these are Action and Speed, which specify the team leader’s cur-rent action (e.g., running, working) and current speed (slow, medium, fast),respectively. The statechart of a team member is similar, but it has an additionalbehavioral element: a team member rests when a fast action is taken. To addthis behavior, a superstate was added to a team member’s Action sub-chart,specifying whether he/she is in a resting state or an acting state. In addition,an internal statechart was added to the fast state, to capture the possibilitythat the team member is really resting from the effort and was just supposedto be moving fast but actually isn’t in motion at all. . . . This is triggered bya resting event that is generated after a predefined time from one member toanother. After some time, a rise event is generated and the statechart returnsto its pre-resting state, using the history connector.The other two orthogonal components are less important; they serve as internaland external communicators.

The team behavior is specified using four orthogonal components: a senderand receiver that handle interactions with external components (e.g., CRS), aninformation monitor that informs of a participant’s condition when requested,and a tick manager that generates time stamps during the model’s life span.

3.3 Specifying Team Training Tasks Using LSCs

We used the scenario-based approach (SC) to specify training tasks for the team.The Play-Engine model specifies several training task, which include a set ofinstructions for the team and for the environment. Four different tasks have beenspecified: Escape, Flood, Storm and Volcano Eruption. Each of these initiatesa different scenario. Triggering a new task while another is being executedcauses a violation and the new LSC terminates. An instruction message (e.g.,crawl, jump, swim) triggers an LSC that forces SC to execute a specified set ofmessages. Triggering a new instruction while another is executing terminatesthe latter (this is done using forbidden elements). In addition, LSCs specifycamera control and environmental changes.

5

For example, assigning a flood task triggers the flood LSC, which initiatesthe following scenario (see Fig. 4): the team participants run until the waterlevel rises, and then they swim until the water level is low enough to walk.When the flood is over, the team participants stop motion, and stand, readyfor a new task. During task execution, the run instruction triggers an LSC thatinstructs the team to run at an increasing speed. When the swim instruction istaken, the run LSC terminates.

Some of these scenarios can be seen in 3 and the video figure.

3.4 Designing the Animation Using 3DGS

The front-end for the model (FE) consists of a real-time 3D animation of thetraining. Animated renditions of a team leader and a team member were createdbased on the cbabe model of 3DGS.

The team is placed on an outdoor grass field, with some scattered housesand the team members are positioned behind the leader. Both the leader (bluishuniform and brunette hair) and the members (grayish uniforms and blond hair)can stand, walk, run, bend, crawl, swim, wade, jump and wave. Team memberscan also fall and rest.

The user can query the model by clicking an animated figure, and the rel-evant data (e.g., ID, current action) is displayed next to it. Environmentalchanges may occur: a storm makes the sky darker, a flood displays a layer ofwater, and a volcano eruption changes the ground to lava. Each environmentalchange is accompanied by matching sound effects.

The main camera is set in front of the team and is focused on the teamleader. To simplify camera usage, its movement was restricted to up, downand circular. In addition, the camera can zoom in and out. Each of the teammembers is covered by an additional camera; these are initially disabled, and areactivated by an external event (coming from the user or an external architecturalcomponent).

In addition, the GUI portion of the FE includes three panels: Action, Taskand Time. Clicking a button in the Action or Task panels triggers an instructionto the team (e.g., to jump fast ) or assigns it a task (e.g., to escape), respectively.The Time panel displays the running duration of the animation.

3.5 The Integrated Team Training Model

Using the centralized linking architecture described earlier, we linked up FE,SC and ST. The composed model provides a realistic simulation of the teamtraining. The user interacts with the model through FE, while ST and SC runin the background. Fig. 5 describes some of each component.

We illustrate the model with two sample runs (see Fig. 3 and the videofigure). The first concerns a jumping action at medium speed. When the usersets the speed scroll bar to medium and clicks the jump button, FE notifies(i.e., sends a message to) ST about the instruction (1). Accordingly, the teamobject in ST generates an inner event to set the statechart of the team leader to

6

jump at medium speed (i.e., the Action sub-statechart moves to the Jumpingstate and the Speed sub-statechart is set to Medium). After a predefined timeinterval, the team object generates another event for the statecharts of each ofthe team members. Upon entering the Jumping state, ST notifies FE to animatethe action (2). Consequently, the animated team leader jumps, and the teammembers follow suit. The user can change the camera’s position to view theteam from different angles. Camera relocation, however, has no effect on therunning simulation.

As a second example, consider the flood training task specified by the floodLSC. During its execution, other LSCs (e.g., run and floodAlert) are activated,and they notify SC and FE. The user assigns a flood task to the team by clickingon the appropriate button. Consequently, FE notifies SC and the flood LSC isactivated (3), initiating the scenario. The first message in the flood LSC is arunning instruction for the team. Consequently, the run LSC is activated, andit notifies ST to run at a slow speed (4). Later on, a message in the flood LSCinstructs the environment to initiate a flood. The FloodAlert LSC is activatedand notifies FE (5). Accordingly, a water layer is displayed and a correspondingsplashing sound is played. Immediately after this, the swim LSC is activatedand it notifies ST. The swim message enables a forbidden element in the runLSC, causing the run LSC to exit. Again, the swim LSC notifies ST, which inturn notifies FE(2).

The flood LSC completes after it instructs the team to walk, the environ-ment to end the flood, and the team to stand (i.e., to stop moving). At thispoint, there are no more LSCs active in SC, the statecharts of ST are all in thestanding state, and the animated figures in FE are standing, ready for the nexttask.

If a fast speed action is assigned to the team, each of the two team memberobjects will enter a resting state after some time. Consequently, ST notifiesSC(6) and FE(2). Concurrently, the team member in FE changes its appearanceto resting, an LSC in SC is triggered, and it notifies FE (5) to activate the teammember’s camera.

During a run of the system, the user may query the model or relocate thecamera. When a query is requested, FE notifies ST (1), which provides theappropriate information. Changes in the camera, however, do not interact withthe reactive engines.

4 Extensions

This implementation of RA is oriented towards academic research. It suggestsan intuitive methodology to simulate large scale complex systems, of whichbiological ones are a good example. RA may be adapted to other kinds ofsystems too, but further work would be needed. For example, one would haveto address IEEE standards such as distributed interactive simulation.

Synchronization is also an issue that our architecture should address. Eacharchitectural component advances asynchronously by the clock of its host com-

7

puter. Thus, loss of synchronization may occur. In some applications this maycause highly undesirable results. There are numerous proposals as to how tobest deal with time in a distributed environment, and there are several incentivesfor doing so using a centralized clock tick.

As a quick and simple solution for our team training example, we added twoclocks to the Time GUI panel, one displays the duration of the animation andthe other displays the duration of the simulation. A complete synchronizationcan be easily achieved by adding a periodic tick signal mechanism to the CRS.

References

[1] 3D game studio A6, Web page: http://www.conitec.net/a4info.htm.

[2] D. Barak, D. Harel, and R. Marelly. InterPlay: Horizontal Scale-Up andTransition to Design in Scenario-Based Programming. In Lectures on Con-currency and Petri Nets, volume 3098 of Lect. Notes in Comp. Sci., pages66–86. Springer-Verlag, 2004.

[3] W. Damm and D. Harel. LSC’s : Breathing life into message sequencecharts. In P.Ciancarini, A. Fantechi, and R. Gorrieri, editors, Proc. 3rdIFIP Int. Conf. on Formal Methods for Open Object-Based Distributed Sys-tems (FMOODS’99). Kluwer Academic Publishers, 1999.

[4] S. Efroni, D. Harel, and I. R. Cohen. Reactive animation: Realistic mod-eling of complex dynamic systems. IEEE Computer, 38(1):38–47, 2005.

[5] D. Harel. Statecharts: A visual formalism for complex systems. Science ofComputer Programming, 8:231–274, 1987.

[6] D. Harel, S. Efroni, and I. R. Cohen. Reactive animation. In FormalMethods for Components and Objects, pages 136–153, 2002.

[7] D. Harel and E. Gery. Executable object modeling with statecharts. Com-puter, pages 31–42, July 1997. Also in Proc. 18th Int. Conf. Soft. Eng.,Berlin, IEEE Press, March, 1996, pp. 246–257.

[8] D. Harel and R. Marelly. Specifying and executing behavioral requirements:The play in/play-out approach. Software and System Modeling (SoSyM),2:82–107, 2003.

[9] I-Logix,Inc., products web page. http://www.ilogix.com/fs prod.htm.

[10] Z. Manna and A. Pnueli. Time for concurrency. In A. Bensoussan andJ.-P. Verjus, editors, Future Tendencies in Computer Science, Control andApplied Mathematics, volume 653 of Lect. Notes in Comp. Sci., pages 129–153. Springer-Verlag, 1992.

[11] Simple Object Access Protocol specification, Web page:http://www.w3.org/TR/soap/.

8

[12] UML. Documentation of the unified modeling language (UML). Availablefrom the Object Management Group (OMG), http://www.omg.org.

[13] Web page http://www.xml.com.

9

Figure 1: Generic multi-party reactive animation (right): The reactive engines(red) and the animated front-end (blue) interact through a central component(yellow). The pre-defined form of the XML files (right): messages of this typeare recognized by the different architectural components. The header of thedocument specifies the source and destination (e.g., RHAPSODY to ANIMA-TOR), while the action to be taken is encoded in its body (e.g., message of typeACTION: COACH RUN FAST . . . VALN).

10

Figure 2: Implementing generic multi-party reactive animation: Rhapsody, thePlay-Engine and 3DGS linked using a central routing server (CRS).

Figure 3: Message transmission between the architectural components FE,STand SC.

11

Figure 4: The team leader’s statechart (left) and the flood LSC (right).

12

Figure 5: The Team Training Model: The statecharts of state-based model (ST,at top); part of the LSCs providing training task specifications (SC, at bottomleft); and snapshots from animated front-end and the GUI (FE, at bottomright).

13