1

ADAPT: The Agent Development andPrototyping Testbed

Alexander Shoulson, Nathan Marshak, Mubbasir Kapadia, and Norman I. Badler

Abstract—We present ADAPT, a flexible platform for designing and authoring functional, purposeful human characters in a rich

virtual environment. Our framework incorporates character animation, navigation, and behavior with modular interchangeable

components to produce narrative scenes. The animation system provides locomotion, reaching, gaze tracking, gesturing,

sitting, and reactions to external physical forces, and can easily be extended with more functionality due to a decoupled,

modular structure. The navigation component allows characters to maneuver through a complex environment with predictive

steering for dynamic obstacle avoidance. Finally, our behavior framework allows a user to fully leverage a character’s animation

and navigation capabilities when authoring both individual decision-making and complex interactions between actors using a

centralized, event-driven model.

✦

1 INTRODUCTION

Animating interacting virtual humans in real-time is acomplex undertaking, requiring the solution to tightlycoupled problems such as steering, path-finding, full-body character animation (e.g. locomotion, gaze track-ing, and reaching), and behavior authoring. This com-plexity is amplified as we increase the number andsophistication of characters in the environment. Nu-merous solutions for character animation, navigation,and behavior design exist, but these solutions areoften tailored to specific applications, making integra-tion between systems arduous. Integrating multiplecontrollers requires a deep understanding of eachcontroller’s design and combinatorial communicationbetween each other to avoid conflicts that lead tomotion artifacts. This is further exacerbated when us-ing hybrid control architectures (e.g., motion capturedata, inverse kinematics, physically based animation)in a single system. Monolithic, feature-rich characteranimation systems do not commonly support modu-lar access to only a subset of their capabilities, whilesimpler systems lack control fidelity. Realistically, nosub-task of character control has a “perfect” solution.An ideal character animation system would allowa designer to choose between preferable techniques,leveraging the wealth of established systems alreadyproduced by the character animation research com-munity and interface with robust frameworks forbehavior and navigation.

We present a modular system that allows for theseamless integration of multiple character animation

• All authors are with the Department of Computer and InformationScience, University of Pennsylvania, Philadelphia, PA, 19104.

• A. Shoulson: ashoulson@gmail.com• N. Marshak: nmarshak@seas.upenn.edu• M. Kapadia: mubbasir.kapadia@gmail.com• N. Badler: badler@seas.upenn.edu

controllers on the same model without requiring anycontroller to drastically change or accommodate anyother. Rather than tightly coupling a fixed set of char-acter controllers, ADAPT uses a system for blendingarbitrary poses in a user-authorable dataflow pipeline.Our system combines these animation controllers withan interface for path-finding and steering, as wellas a comprehensive behavior authoring structure forauthoring both individual decision-making and com-plex interactions between groups of characters. Ourplatform generalizes to allow the addition of newcharacter controllers and behavior routines with mini-mal integration effort. New control capabilities can beeasily integrated into ADAPT without being aware of,or requiring any changes in exisiting controllers. Oursystem provides a platform for experimentation incharacter animation, navigation, and behavior author-ing. We allow researchers to rapidly iterate on charac-ter controller designs with visual feedback, comparetheir results with other established systems on thesame model, and use features from other packages toprovide the functionality they lack without the needto deeply integrate or reinvent known techniques.

This paper makes the following contributions:

• An extensible, scalable platform for animatingcrowds of autonomous virtual characters in so-phisticated 3D virtual worlds.

• Automatic integration of diverse controllers, facil-itating data-driven, physically based, or IK basedsolutions to simultaneously operate on overlap-ping subsets of the character’s body without theneed for any conflict resolution.

• A graphical, hierarchical behavior authoring sys-tem empowering authors to design and orches-trate fine-grained character movements, as wellas complex, coordinated, and dynamic multi-character interactions.

2

Deliverable. We provide ADAPT: an open-source li-brary that comines a suite of modular character con-trollers including data-driven locomotion, proceduralreaching, gesturing, and physical reactions, with inte-grated navigation, and event-centric behavior author-ing for multi-actor interactions to create an extensibleend-to-end crowd authoring tool for research.

2 RELATED WORK

There exists a wealth of research [1], [2], [3] thatseparately addresses the problems of characteranimation, steering and path-finding, and behaviorauthoring, with many open challenges [4] that needto be addressed in an effort to arrive at a commonstandard for simulating autonomous virtual humansfor the next generation of interactive virtual worldapplications.

Character Animation. Data-driven approaches [5],[6] use motion-capture data to animate a virtualcharacter. Motion clips can be manipulated andconcatenated by using warping [7], blending [8],[9], layering [10], or planning [11], [12] to enforceparametric constraints on recorded actions. Interactivecontrol of virtual characters can be achieved bysearching through motion clip samples for desiredmotion as an unsupervised process [13], or byextracting descriptive parameters from motiondata [14]. Procedural methods are used to solvespecific tasks such as reaching, and can leverageempirical data [15], example motions [16], orhierarchical inverse kinematics [17] for more naturalmovement. Physically-based approaches [18], [19]derive controllers to simulate character movement ina dynamic environment. We refer to Pettre et. al. [20]for a more extensive summary of work in these areas.

Steering and Path-finding. For navigation, the en-vironment itself is often described and annotatedas a reduction of the displayed geometry to beused in path planning. Probabilistic roadmaps super-impose a stochastic connectivity structure betweennodes placed in the maneuverable space [21]. Nav-igation meshes [22] provide a triangulated surfaceupon which agents can freely maneuver. PotentialFields [23] generate a global field for the entire land-scape where the potential gradient is contingent uponthe presence of obstacles and distance to goal, but isprone to local minima. Dynamic potential fields [24]have been used to integrate global navigation withmoving obstacles and people, efficiently solving themotion of large crowds without the need for explicitcollision avoidance.

Steering techniques use reactive behaviors [25] orsocial force models [26], [27] to perform goal-directedcollision avoidance in dynamic environments.Predictive approaches [28], [29] and local perception

fields [30], [31] enable an agent to avoid others byanticipating their movements, while more complexscenarios such as group interactions and deadlocksare solved using hybrid techniques [32], space-time planning [33], or by externalizing steeringlogic [34]. Data-driven steering [35], [36] focuses ongenerating local-space samples from observations ofreal people which are used to create databases, orserve as training data to learn computational modelswhich are queried to emulate real-human behavior.Recast [37] provides an open-source solution togenerating navigation meshes from arbitrary worldgeometry by voxelizing the space, and the associatedDetour library provides path planning and predictivesteering on the produced mesh. Pelechano et. al. [1]provide a detailed review of additional work in thisfield.

Behavior Authoring. Animating behaviors invirtual agents has been addressed using multiplediverse approaches, particularly with respectto how behaviors are designed and animated.Early work focuses on imbuing characters withdistinct, recognizable personalities using goals andpriorities [38] along with scripted actions [39]. Oursystem makes use of parameterized behaviortrees [40] to coordinate interactions betweenmultiple characters. The problem of managing acharacter’s behavior can be represented with decisionnetworks [41], cognitive models [42], goal-orientedaction planning [43], [44], or via learning [45]. Verysimple agents can also be simulated on a massivescale using GPU processing [46]. Recent work [47],[48], [49] proposes an event-centric authoringparadigm to facilitate multi-actor interactions withcontextual awareness based on agent type and eventlocation.

Multi-Solution Platforms. End-to-end commercialsolutions [50], [51] combine multiple diverse charactercontrol modules to accomplish simultaneous taskson the same character, incorporting navigation,behavior, and/or robust character animation.SteerSuite [52] is an open-source platform fordeveloping and evaluating steering algorithms.SmartBody [53] is an open-source system thatcombines steering, locomotion, gaze tracking,and reaching. These tasks are accomplished with 15controllers working in unison to share control of partsof the body. SmartBody’s controllers are hierarchicallymanaged [54] where multiple animations, such asgestures, are displayed on a virtual character usinga scheduler that divides actions into phases andblends those phases by interpolation. The controllersmust either directly communicate and coordinate,or fix cases where their controlled regions of thebody overlap and overwrite one another, makingthe addition of a new controller a process that

3

affects several other software components. Ourplatform shares some qualities with SmartBody, butalso differs in several fundamental ways. Whilewe do provide a number of character controllersfor animating a virtual human, our work focusesmore on enabling high-level behavioral control ofmultiple interacting characters, the modularity ofthese character controllers, and the ease with whicha user can introduce a new animation repertoire tothe system without disturbing the other controllersalready in place.

3 FRAMEWORK

������� ������ ���

��������

������������ �����

�

�����������

���� ���������

�� ����� ���

�������

� ���

��������

���������� ���������

��� ��� ���� �

Fig. 1. Overview of ADAPT framework.

ADAPT operates at multiple layers with inter-changeable, lightweight components, and minimalcommunication between modules. The animationsystem performs control tasks such as locomotion,gaze tracking, and reaching as independent modules,called choreographers, that can share parts of thesame character’s body without explicitly communi-cating or negotiating with one another. These mod-ules are managed by a coordinator, which acts as acentral point of contact for manipulating the virtualcharacter’s pose in real-time. The navigation systemperforms path-finding with predictive steering andwe provide a common interface to allow users toreplace the underlying navigation library without af-fecting the functionality of the rest of the framework.The behavior level is split into two tiers. Individualbehaviors are attached to each character and ma-nipulate that character using the behavior interface,while a centralized control structure orchestrates thebehavior of multiple interacting characters in real-time. The ultimate product of our system is a pose

for each character at an appropriate position in theenvironment, produced by the animation coordinatorand applied to a rendered virtual character in thescene each frame. Figure 1 provides an illustrationof the framework from an architectural integrationperspective (the framework appears more hierarchicalfrom a behavior control perspective, as illustrated inFigure 6).

3.1 Full-Body Character Control

We divide the problem of character animation into aseries of isolated, modular components called choreog-raphers attached to each character. Each choreographeroperates on a shadow, which is an invisible clone ofthe character skeleton, and has unmitigated controlto manipulate the skeletal joints of its shadow. Eachframe, a choreographer produces an output pose con-sisting of a snapshot of the position and orientation ofeach of the joints in its private shadow. A coordinatorreceives the shadow poses from each choreographerand performs a weighted blend to produce a finalpose that is applied to the display model for thatframe. Since each choreographer has its own model tomanipulate without interruption, choreographers donot need to communicate with one another in orderto share control of the body or prevent overwritingone another. This allows a single structure, the coor-dinator, to manage the indirect interactions betweenchoreographers using a simple, straightforward, andhighly authorable process centered around blendingthe shadows produced by each choreographer. Thissystem is discussed in more detail in Section 4.

3.2 Steering and Path-finding

We use a navigation mesh approach for steering andpath-finding with dynamic obstacle avoidance. Eachdisplay model is controlled by a point-mass system,which sets the root positions (usually the hips) of thedisplay model and each shadow every frame. Char-acter choreographers do not directly communicatewith the navigation layer. Instead, choreographers aremade aware of the position and velocity of the charac-ter’s root, and will react to that movement on a frame-by-frame basis. A character’s orientation can followseveral different rules, such as facing forward whilewalking, or facing in an arbitrary direction, and wehandle this functionality outside of the navigation sys-tem itself. ADAPT supports both the Unity3D built-innavigation system and the Recast/Detour library [37]for path-finding and predictive goal-directed collisionavoidance, and users can supplement their own pre-ferred solution.

3.3 Behavior

ADAPT accommodates varying degrees of behaviorcontrol for its virtual characters by providing a di-verse set of choreographers and navigation capabili-ties. Each character has capabilities like ReachFor(),

4

GoTo(), and GazeAt() that take straightforwardparameters like positions in space and send messagesto that character’s navigation and animation compo-nents. To invoke these capabilities, we use Parame-terized Behavior Trees (PBTs) [40], which present amethod for authoring character behaviors that em-phasizes simplicity without sacrificing expressiveness.Having a single, flat interface for a character’s actionrepertoire simplifies the task of behavior authoring,with well-described and defined tasks that a charactercan perform. One advantage of the PBT formalism isthat they accommodate authoring behavior for multi-ple actors in one centralized structure. For example, aconversation between two characters can be designedin a single data structure that dispatches commandsto both characters to take turns playing sounds orgestural animations. For very specific coordination ofcharacters, this approach can be preferable over tradi-tional behavior models where characters are authoredin isolation and interactions between characters aredesigned in terms of stimuli and responses to triggers.The behavior system is discussed in more detail insection 5.

4 SHADOWS IN FULL-BODY CHARACTER

ANIMATION

General character controllers animate a virtual charac-ter using pre-recorded motions, or procedurally withphysical models or inverse kinematics. We addressthe problem of coordination between these controllersby allocating each character controller its own privatecharacter model, a replica of the skeleton or a subsetof the skeleton of the character being controlled. Ourmodular controllers, called choreographers, act ex-actly the same way as traditional character controllers,but do so on private copies of the actual renderedcharacter model. These skeleton clones (shadows),match the skeletal hierarchy, bone lengths, and initialorientations of the final rendered character (displaymodel), but have no visual component in the scene.Figure 2 illustrates a two-step blend process. First wecombine the pose of the locomotion choreographer(green, full-body) during a walk cycle with the gesturechoreographer (blue, upper-body) playing a wavinganimation, and then we apply the arm of the reachingchoreographer (red, upper-body) full blend weight,safely overwriting the previous step for the joints ofthe left arm. The partial blend is represented witha mix of colors in the RGB space. Blend ordering isdiscussed in greater detail in Section 4.2.

Character animation has two interleaving steps.First, each choreographer manipulates its personalshadow and outputs a snapshot (called a shadowpose) describing the position and orientation of thatshadow’s joints at that time step. Then, we use acentralized controller to blend the shadow pose snap-shots into a final pose for the rendered character.

For clarity, note that “shadow” refers to the invisibleskeleton allocated to each choreographer to manipu-late, while a “shadow pose” is a serialized snapshotcontaining the joint positions and orientations for ashadow at a particular point in time.

������� ����� ���������

����������� ����������

������������

����������������

Fig. 2. Blending multiple character shadows to pro-

duce a final output skeleton pose.

4.1 Choreographers

The shadow pose of a character at time t is given byPt ∈ R

4×|J|. where Pjt where is the configuration of

the jth joint at time t. A choreographer is a functionC(Pt) −→ Pt+1 which produces the next pose bychanging the configuration of the shadow joints forthat time step. Using these definitions, we define twoclasses of choreographers:

Generators. Generating choreographers produce theirown shadow pose each frame, requiring no externalpose data to do so. Each frame, the input shadowpose Pt for a generator C is the pose Pt−1 generatedby that same choreographer in the previous frame.For example, a sitting choreographer requires noexternal input or data from other choreographersin order to play the animations for a charactersitting and standing, and so its shadow’s pose isleft untouched between frames. This is the defaultconfiguration for a choreographer.

Transformers. Transforming choreographers expectan input shadow pose, to which they apply an offset.Each frame, the input shadow pose Pt to a transformerC is an external shadow pose P′

t+1 from anotherchoreographer C ′, computed for that frame. The co-ordinator sets its shadow’s pose to P′

t+1 and appliesan offset to the given pose during its execution,to produce a new pose Pt+1. For example, beforeexecuting, the reach choreographer’s shadow is setto the output pose of a previously-updated choreog-rapher’s shadow (say, the locomotion choreographerwith swinging arms and torso movement). The reachchoreographer then solves the reach position fromthe base of the arm based on the torso position it

5

was given, and overwrites its shadow’s arm andwrist joints to produce a new pose. This allows thereach choreographer to accommodate multiple torsoconfigurations without the choreographers directlycommunicating or even being fully aware of oneanother. A transforming choreographer can receive aninput pose, or blend of input poses, from any otherchoreographer(s).

4.2 The Coordinator

During runtime, our system produces a pose for thedisplay model each frame, given the character chore-ographers available. This is a task overseen by thecoordinator. The coordinator is responsible for main-taining each choreographer, organizing the sequencein which each choreographer performs its computa-tion each frame, and reconciling the shadow posesthat each choreographer produces. The coordinator’sfinal product each frame is a sequence of weightedblends of each active choreographer’s shadow pose.We compute this product using the pose dataflow graph,which dictates the order of updates and the flow ofshadow poses between choreographers. Generatorspass data to transformers, which can then pass theirdata to other transformers, until a final shadow poseis produced, blended with others, and applied to thedisplay model.

Blending is accomplished at certain points in thepose dataflow graph denoted by blend nodes, whichtake two or more input shadows and produce aweighted blend of their transforms. If the weightssum to a value greater than 1, they are automaticallynormalized.

B({(Pi, wi) : i = 1..n)}) −→ P′ (1)

Designing a dataflow graph is a straightforwardprocess of dictating which nodes pass their output towhich other nodes in the pipeline, and the graph canbe modified with minimal effort. The dataflow graphfor a character is specified by the user during theauthoring process. The weights involved in blendingare bound to edges in the graph and then controlledat runtime by commands from the behavior system.The order of the pose dataflow graph roughly dic-tates the priority of choreographers over one another.Choreographers closer to the final output node in thegraph have the authority to overwrite poses producedearlier in the graph, unless bypassed by the blendingsystem. We generally design the graph so that chore-ographers controlling more parts of the body precedethose controlling fewer.

Blended poses are calculated on a per-joint basis us-ing each joint’s position vector and orientation quater-nion. The blend function produces a new shadowpose that can be passed to other transformers, orapplied to the display model’s skeleton. Taking a lin-ear weighted average of vectors is a solved problem,but such is not the case with the problem of quickly

averaging n > 2 weighted quaternions. We discuss thetechniques with which we experimented, and the finalcalculation method we decided to use in Appendix A(supplemental). In addition, Feng et. al. [55] provide adetailed review of more sophisticated motion blend-ing techniques than our linear approach.

Figure 3 illustrates a sample dataflow graph. Threegenerating choreographers (blue) begin the pipeline.The gesture choreographer affects only the upperbody, with no skeleton information for the lower body.Increasing the value of the gesture weight wg placesthis choreographer in control of the torso, head, andarms. The sitting and locomotion choreographers canaffect the entire body, and the user controls themby raising and lowering the sitting weight ws. If wg

is set to 1 − ǫ, the upper body will be overriddenby the gesture choreographer, but since the gesturechoreographer’s shadow has no legs, the lower bodywill still be controlled by either the sitting or loco-motion choreographer as determined by the valueof ws. The first red blend node combines the threeproduced poses and sends the weighted average poseto the gaze tracker. The gaze tracking choreographerreceives an input shadow pose, and applies an offsetto the upper body to achieve a desired gaze target andproduce a new shadow pose. The second blend nodecan bypass the gaze tracker if the gaze weight wz isset to a low value (ǫ). The reach and physical reactionchoreographers receive input and can be bypassed ina similar way. The final result is sent and applied tojoints of the display model, and rendered on screen.

4.3 Using Choreographers and the Coordinator

The dataflow graph, once designed, does not need tobe changed during runtime or to accommodate ad-ditional characters. Instead, the coordinator providesa simple interface comprising messages and exposedblend weights for character animation. Messages arecommands (e.g., SitDown()) relayed by the coordi-nator to its choreographers, making the coordinatora single point of contact for character control, asillustrated in Figure 1. In addition to messages, theweights used for blending the choreographers at eachblend node in the dataflow graph are exposed, allow-ing external systems to dictate which choreographeris active and in control of the body (or a segment ofthe body) at a given point of time.

As an example, when the coordinator receives agesture command, it raises wg (in Figure 3), whichtakes control of the arms and torso away fromboth the locomotion and sitting choreographers andstops the walking animation’s arm swing. Givensole control, the gesture choreographer plays ananimation on the upper body, and then is faded backout to allow the walking arm-swing to resume. Sincethe gesture choreographer’s shadow skeleton has noleg bones, it never overrides the sitting or locomotion

6

�����������

������

������

�������

������������� ��������

��

����� 1���

��

1���

��

�1�������

�������

1���

��

Fig. 3. A sample dataflow graph we designed for evaluating ADAPT. Generating choreographers appear in blue,

transmuting choreographers appear in green, and blend nodes appear as red crosses. The final display model

node is highlighted in orange. The sitting weight ws, gesture weight wg, gaze weight wz, reach weight wr, and

physical reaction weight wp are all values between some very small positive ǫ and 1 − ǫ.

choreographer, so the lower body will still be sittingor walking while the upper body gesture plays. Allweight changes are smoothed over several frames toprevent jitter and transition artifacts. The division ofroles between the coordinator and choreographerscentralizes character control to a single externally-facing character interface, while leaving the detailsof character animation distributed across modularcomponents are isolated from one another and canbe easily updated or replaced.

Shadow Pose Post-Processing. Since shadow posesare serializations of a character’s joints, additionalnodes can be added to the pose dataflow graph tomanipulate shadows as they are transferred betweenchoreographer nodes or blend nodes. For instance,special filter nodes can be added to constrain thebody position of a shadow pose, preventing jointsfrom reaching beyond a comfortable range byclamping angles, or preventing self-collisions byusing bounding volumes. Nodes can be designed tobroadcast messages based on a shadow’s pose, suchas notifying the behavior system when a shadow isin an unbalanced position, or has extended its reachto a certain distance. The interface for adding newkinds of nodes to a pose dataflow graph is highlyextensible. This affords the user another opportunityto quickly add functionality to a coordinator withoutdirectly modifying any choreographers.

Cross-Choreographer Functionality. Conflicts be-tween choreographers are resolved using the inherentpriority of the pose dataflow graph – every trans-former has the option of overwriting any joint itreceives as input. Choreographers are unaware ofwhether or not their output poses are overwrittenlater or blended out of the final pose. Unless a particu-lar choreographer is the last dataflow node to affect allof the joints it requires, it cannot make any guaranteesabout the resulting pose it produces. For example,the reach choreographer cannot guarantee a successfulreach if its arm joints are partially overwitten bythe gesture choreographer. This is why the reachcontroller appears towards the end of the sample

dataflow graph in Figure 3, giving it high overwritepriority. There are two additional ways to addresspriority and guarantees at runtime. First, the messagepassing system can be used to tell a choreographerwhen it is in full control of a part of the body,allowing that choreographer to give accurate feedbackor request more control. Second, the pose dataflowgraph can be rearranged dynamically to adjust whichchoreographers have priority over one another whenneeded. In practice, priority is handled by a fewspecial cases in the behavior system, but the techniquecould be developed further.

4.4 Example Choreographers

ADAPT provides a number of diverse choreographersfor animating a fully articulated, expressive virtualcharacter. Some of these choreographers weredeveloped specifically for ADAPT, while others wereoff-the-shelf solutions used to highlight the ease ofintegration with the shadow framework. ADAPT isdesigned to “trick” a well-behaved character controlsystem into operating on a dedicated shadow modelrather than the display model of the character, andso the process of converting an off-the-shelf charactercontrol library into a character choreographeris straightforward. Since shadows replicate thestructure and functionality of a regular charactermodel, no additional considerations are required oncethe choreographer has been allocated a shadow. Notethat the choreographers presented here are largelybaseline examples. The focus of ADAPT is to allowa user to add additional choreographers, experimentwith new techniques, and easily exchange genericchoreographers with more specialized alternatives.

Locomotion. ADAPT uses a semi-procedural motion-blending locomotion system for walking and runningreleased as a C# library with the Unity3D engine [14].The system takes in animation data, analyzes thoseanimations, and procedurally blends them accordingto the velocity and orientation of the virtual character.We produced satisfactory results on our test modelusing five motion capture animation clips. The

7

user annotates the character model to indicate thecharacter’s legs and feet, which allows the locomotionlibrary to use inverse kinematics for foot placementon uneven surfaces. We extended this library to workwith the ADAPT shadow system, with some smallimprovements.

Gaze Tracking. We use a simple IK-based systemfor attention control. The user defines a subset ofthe upper body joint hierarchy which is controlledby the gaze tracker, and can additionally specifyjoint rotation constraints and delayed reaction speedsfor more realistic results. These parameters can bedefined as functions of the characters velocity orpose, to produce more varied results. For instance, arunning character may not be permitted to rotate itstorso as far as a character standing still.

Upper Body Gesture Animations. We dedicate ashadow with just the upper body skeleton to playinganimations such as hand gestures. We can playmotion clips on various parts of the body to blendanimations with other procedural components.

Sitting and Standing. The sitting choreographermaintains a simple state machine for whether thecharacter is sitting and standing, and plays theappropriate transition animations when it receives acommand to change state. This choreographer acts asan alternative to the locomotion choreographer whenoperating on the lower body, but can be smoothlyoverridden by choreographers acting on the upperbody, such as the gaze tracker.

Reaching. We implemented a simple reaching controlsystem based on Cyclic Coordinate Descent (CCD).We extended the algorithm to dampen the maximumangular velocity per frame, include rotationalconstraints on the joints, and apply relaxationforces in the iteration step. During each iterationof CCD (100 per frame), we clamp the rotationangles to lie within the maximum extension range,and gently push the joints back towards a desired“comfortable” angle for the character’s physiology.These limitations and relaxation forces are basedon an empirical model for reach control based onhuman muscle strength [56]. This produces morerealistic reach poses than naıve CCD, and requiresno input data animations. The character can reachfor an arbitrary point in space, or will try to do so ifthe point is out of range.

Physical Reaction. By allocating an upper-bodychoreographer with a simple ragdoll, we can displayphysical reactions to external forces. Once an impactis detected, we apply the character’s last pose to theshadow skeleton, and then release the ragdoll andallow it to buckle in response to the applied force.

By quickly fading in and out of the reeling ragdoll,we can display a physically plausible response andcreate the illusion of recovery without requiring anysprings or actuators on the ragdoll’s joints.

SmartBody Integration. To access its locomotion andprocedural reaching capabilities, we integrated theICT SmartBody framework into our platform, usingSmartBody’s Unity interface and some modifications.Since our model’s skeleton hierarchy differed fromthat of the default SmartBody characters, sample an-imations had to be retargeted to use on our model.Additionally, our animation interface needed to inter-act with SmartBody using BML. Since our coordina-tor is already designed to relay messages from thebehavior system, changing those messages to a BMLformat was a straightforward conversion. Overall, theSmartBody choreographer blends naturally with otherchoreographers we have in the ADAPT framework,though SmartBody has other features that we donot currently exploit. This process demonstrates theefficacy of integrating other available libraries and/orcommercial solutions.

4.5 Automatic Coordinator Derivation

Since a coordinator may be responsible for any num-ber of choreographers, each affecting an arbitrarynumber of contiguous joints, it is important for ageneration system to understand the relationshipsbetween joints and the layering nature of ADAPTchoreographers. We have designed a simple heuristicby which the hierarchical order of choreographerupdates can be automatically generated. The resultof our analysis is an ordering of choreographers touse in conjunction with two-input blend nodes forgenerating a pose dataflow graph. This is achievedby starting the graph with choreographers that affectbroader parts of the body, and placing narrower, morefocused choreographers later in the sequence. Sincechoreographers deeper in the pose dataflow graphhave higher priority over the parts of the body forwhich they’re responsible, we assume that choreog-raphers that control fewer parts of the body havea more specialized purpose requiring more explicitcontrol in the final generated pose. A choreographerfor locomotion might swing the character’s arms asthe model walks, but the target constraints of a reachchoreographer are priortized by placing it later in thedataflow graph.

For a choreographer C, we compute: (1) its max-imum depth Cd, defined as the maximal distancebetween any joint affected and the root (hips) of themodel and, (2) its coverage Cv which is the numberof joints affected. The specificity of a choreographerSC = Cd

Cv

is lower for choreographers that affect a highnumber of joints close to the hips (torso), and higherfor choreographers that affect a small number of joints

8

farther away from the root, such as by controlling theextremities. Sorting the choreographers by ascendingorder of specificity prioritizes more specialized chore-ographers, giving them more exclusive access to thejoints they focus on controlling. Figure 4 illustrates thedepth and coverage of four choregraphers, with theautomatically generated pose dataflow graph shownin Figure 5 (arranging the choreographers left-to-rightby ascending specificity).

Fig. 4. Coverage analysis of four choreographers on a

model with 32 total joints.

����������

��� ������ ��������������

Fig. 5. The automatically generated pose dataflow

graph from a four-choreographer coverage analysis.

This is a simple approach that may not apply toevery combination of choreographers, but provides auseful first-pass analysis to offer a suggestion to anauthor designing a new coordinator for a set of unfa-miliar choreographers. By design, choreographers arealready aware of the joints they affect on the body,making computing Cd and Cv possible without anymodification to the ADAPT system. However, otherheuristics with additional information from the chore-ographers could be designed to capture specificity orother elements of good dataflow graph design. Thisautomated technique accelerates the dataflow graphauthoring process, which occurs infrequently enoughto be tweaked by a human author without consumingprohibitive amounts of time.

5 CHARACTER BEHAVIOR

One of the most fundamental problems in interactivecharacter animation is converting simple commandslike “reach for that object”, or cooperative directiveslike “engage in a conversation” into a series of com-plicated joint actuations on one or more articulatedbodies. ADAPT accomplishes this task with a hier-archy of abstractions known as the ADAPT Character

Stack, illustrated in Figure 6. The stack is split intofour main tiers: Behavior, Actor, Body, and Animation.The higher two levels (“Behavior” and “Actor”) of thestack are designed for use by comparitively untrainedauthors, while lower levels levels offer more fine-graincontrol fidelity at the expense of simplicity, and canbe accessed by expert authors to ensure very specificconstraints on the character’s movements.

��������

���

���

�����������

��������

��������

��������

�

������������������

��������������

������������

�������

Fig. 6. The ADAPT Character Stack.

Commands from each layer of the stack are filtered,converted, and distributed to subcomponents, startingas behavior invocations, translating to messages sentto the navigation or animation system, and finallyconverting into joint angles and blend weightsused for posing the character on a frame-by-framebasis. Each layer in the character stack provides adifferent entry point for technical control over thecharacter. The “Behavioral” layers offer interfaces forcontrolling the character at a high level, suitable forinvocation by behavior trees and Smart Objects [57].

Animation (Navigation and Coordinator). Thislayer provides the lowest-level external access tothe character’s animation. A component accessingthis part of the character stack is concerned withsending messages directly to choreographers (suchas to change the reaching target position), ormodifying blend weights to adjust the influence of achoreographer.

Body. This layer converts abstract commands likeReachFor() into a series of messages passed to thereach choreographer and coordinator to set the reachtarget and raise the blend weight for the reachingpose. The Body layer is created to encapsulate thepose dataflow graph for a particular character, andassigns more semantic meaning to the blend weightsfor each blend node in the graph. An ADAPTcharacter’s list of capabilities is discussed in moredetail in Section 5.1.

9

Actor. This layer abstracts commands in the Bodylayer. However, unlike the Body layer, the commandsin the Actor layer will keep track of the durationof a task, and report success or failure. A call toReachFor() in the Body layer will return instantlyand begin the reaching process, whereas a call toReachFor() in the Actor layer will begin the reachprocess and then block until the reach has succeededor failed. Commands in the Actor layer are alsodesigned to respond to a termination signal forscheduling, as described in Section 6.1.1. This layer ofabstraction is necessary for controlling a character’sbehavior with behavior trees.

Behavior. This layer contains more sophisticated,contextual commands comprising multiple sequentialcalls to the Actor layer, such as playing a series ofgestures to convey approval in a conversation. TheBehavior layer also contains the character’s personalbehavior tree, and a BehaviorProcess node responsi-ble for scheduling multi-character actions using theADAPT behavior scheduler described in Section 6.1.1.Unless involved in a multi-actor event, the Behaviorlayer is responsible for directing the character’s goals,and external calls to the Behavior layer are usuallyconcerned with suspending or re-activating a charac-ter’s autonomy.

5.1 Body Capabilities

The navigation and shadow-based character anima-tion system provides a number of capabilities, enu-merated below. Passing an empty target position willend that task, stopping the gaze, reach, or navigation.The locomotion choreographer will automatically re-act to the character’s velocity, and move the legs andarms to compensate if the character should be turn-ing, walking, side-stepping, backpedaling, or running.Note that only sitting and navigating are mutuallyexclusive. All other commands can be performedsimultaneously without visual artifacts.

Commands DescriptionReachFor(target) Activates the reaching choreogra-

pher, and reaches towards a posi-tion.

GazeAt(target) Activates the gaze choreographer,and gazes at a position.

GoTo(target) Begins navigating the character to aposition.

Gesture(name) Activates the gesture choreographerfor the duration of an animation.

SitDown() Activates the sitting choreographerand sits the character down.

StandUp() Stands the character up and thendisables the sitting choreographer.

Adding a New Body Capability. Adding a newbehavior capability with a motion component, suchas climbing or throwing an object, requires a chore-ographer capable of producing that motion. Since

choreographers operate on their own private copiesof the character’s skeleton, they can be designed inisolation and integrated into the system separately.Once the choreographer is developed, the processof adding a new behavior capability to take advan-tage of the choreographer requires two steps. First,the choreographer must be authored into the posedataflow graph, either as a generating or transformingnode, with appropriate connections to blend nodesand other choreographers. Next, the behavior inter-face can be extended with new functions that ei-ther modify the blend weights relevant to the newchoreographer, and/or pass messages to that chore-ographer by relaying them through the coordinator.The sophistication of character choreographers varies,but ADAPT is specifically designed for integratingnew choreographers into the character behavior andanimation pipeline.

6 CHARACTER INTERACTIONS

Using a character’s body capability repertoire, wecan produce more sophisticated actions as charac-ters interact with one another and the environment.Authoring complex behaviors requires an expressiveand flexible behavior authoring structure granting thebehavior designer reasonable control over the char-acters in the environment. To accomplish this task,we use parameterized behavior trees (PBTs). PBTsare an extension of the behavior tree formalism thatallow behavior trees to manage and transmit datawithin their hierarchical structure without the use ofa blackboard.

6.1 Characters Interacting with Each Other

A useful advantage of PBTs is the fact that theycan simultaneously control multiple characters in asingle reusable structure called an event. Events arepre-authored behavior trees that sit uninitialized in alibrary until invoked at runtime. When instantiated,an event takes one or more actors as parameters, andis temporarily granted exclusive control over thoseparticipants. The event treats these characters as limbsof the same entity, dispatching commands for agentsto navigate towards and interact with one another.Once the event ends, control is yielded to the char-acters’ own individual decision processes, which arealso designed using PBTs. A conversation event canbe authored as a simple sequential and/or stochasticsequence of commands directing agents to face oneanother and take turns playing gesture animations.

Figure 7 illustrates a sample behavior tree eventconducting two characters through a conversationusing our action repertoire. The characters, a1 and a2,are passed as parameters to the tree, along with themeeting position. Using our action interface, the treedirects the two characters to approach one another

10

GoTo(MeetingPoint)

GoTo(MeetingPoint)

a2

a1

������

Gesture(“G1”)

a1

Gesture(“G2”)

a1

Gesture(“G1”)

a2

Gesture(“G2”)

a2

����

��������� ��

GazeAt(a1)

GazeAt(a2)

a1

�����������

Conversation(a1 : Actor, a2 : Actor, MeetingPoint : Position)

a2

������

Fig. 7. A simple conversation PBT controlling two char-

acters, a1 and a1, with a MeetingPoint parameter.

at the specified point, face each other, and alter-natively play randomly selected gesture animations.The gesturing phase lasts for an arbitrary durationdetermined by the configuration of the loop node inthe tree. After the loop node terminates, the eventends, reporting success, and the two characters returnto their autonomous behaviors. Note that this treecan be reused at any time for any two charactersand any two locations in the environment in whichto stand. This framework can be exploited to createhighly sophisticated interactions involving crowds ofagents, and its graphical, hierarchical nature makessubtrees easier to describe and encapsulate.

6.1.1 The ADAPT Behavior Scheduler

ADAPT provides a fully-featured schedulerfor managing and updating both the personalbehavior trees belonging to each character andhigher-level event behavior trees encompassingmultiple characters. Four basic principles in behaviordesign enable the scheduler to work effectively fororchestrating the behavior in an ADAPT environment.

PBT Clock. PBTs in ADAPT operate on periodic clockticks, where each tree refreshes itself, evaluates itscurrent state, sends messages through the characterstack, and transitions to its next node if necessary.The ADAPT scheduler keeps track of all of theactive trees in the environment, both personal treesfor individual characters, and event trees for multi-character interactions, and ticks them 30 times persecond.

Character Suspension. Each character in ADAPTowns a “BehaviorProcess” object in its behavior layer,which maintains that character’s state with regardsto autonomy. A character will not receive ticks to itspersonal tree if it has been suspended and placedunder the control of a multi-character event tree.

Behavior Termination. PBTs in ADAPT are designedto respond to a termination signal. Terminationsignals can come at any time, and instruct a tree tointerrupt its current action and end. The result of atermination signal can last multiple PBT clock ticks,so that poses such as reaching can be smoothly fadedout before the tree reports that it has completedtermination.

Event Priority. Multi-character event trees areassigned a priority value, where all characterpersonal trees have minimal priority. When acharacter receives an event with a priority higherthan its current tree, the scheduler terminates thecurrent tree and suspends the character until allinvolved characters are ready. The event then beginsticking and dispatching commands to them. Once theevent terminates, the characters return from beingsuspended.

These four concepts allow very direct control overgroups of characters in the environment, with smoothtransitions between drastically different tasks. Sincetrees can be cleanly terminated at any point in theirexecution, groups of characters involved in wander-ing or conversing with one another in an environmentcould very quickly activate a new, higher priorityevent tree to respond quickly to an event such as aloud noise or a fire. Each tree operates as its owncooperatively multithreaded process with no directinter-communication. Since a character is only evercontrolled by one tree at a time, this presents anopportunity for parallelization in future work.

6.2 Characters Interacting with the Environment

Using the same four principles for the behavior sched-uler allows us to more easily implement smart objectsinto ADAPT to allow characters to interact with theenvironment. A smart object’s affordances can beencoded sequentially in a manner similar to that ofa behavior tree. Smart objects receive ticks from thescheduler clock, and will block until reporting eithersuccess or failure. For example, when a characterwants to sit in a chair, the behavior tree invokesthe chair’s “sit” affordance with the character as aparameter. From that point, the tree will divert anyticks it receives from the scheduler clock to the smartobject, which temporarily takes control of the char-acter and directs it to approach and sit properly onthe chair. Once the smart object chair determines thatthe character has succeeded in sitting down, it willreport success to the behavior tree responsible forthe character’s behavior, so that the tree can moveon to other actions. Parallel nodes in the tree can beused to synchronize actions, allowing a character togesture or gaze at a target while still approaching andsitting. Smart objects represent the primary means of

11

interaction with the environment, and are useful fora wide array of other interaction tasks.

Figure 8 illustrates a simple state machine examplefor a chair smart object that instructs a character toapproach and sit on it in the proper orientation.

�������������� ������

�������

���������� �����

������� �������

����

�������

��� ����

������������ � ������� ����� ������ ��������

���� ����������� �� ��

�� ���������� ��� ����� �!�"� �

Fig. 8. The state machine for a chair smart object’s

“sit” affordance. The object transitions between states

after evaluating predicates on the state of the charac-

ter, periodically sending commands to the actor and

reporting a status (running, success, or failed) to the

behavior tree controlling the affordance.

7 RESULTS

We demonstrate the ADAPT features in isolation, aswell as a final scene showcasing animation, navi-gation, and behavior working together to producea narrative sequence (Figure 1). We can create acharacter that can simultaneously reach, gaze, walk,and play gesture animations, as well as activate otherfunctionality like sitting and physically reacting toexternal forces. ADAPT characters can intelligentlymaneuver an environment avoiding both static anddynamic obstacles. These features are used for author-ing sequences like exchanging objects between actors,wandering while talking on a phone, and multiplecharacters holding a conversation.

7.1 Choreographer Extensibility

Adding a Kinect Choreographer. We created anadditional choreographer to interface with theMicrosoft Kinect and control a character with gestureinput. We allocated a choreographer to the inputof the Kinect, applying the captured skeleton fromthe Kinect’s framework directly to the joints ofthe dedicated shadow. This is demonstrated inFigure 9(a). Blending this choreographer with othersallowed us to expand the character’s agency in theworld. When the character stands idle, we give fullupper and lower body control to the Kinect input.When the user wishes to make the character move,we blend the legs of the locomotion choreographeron top of the Kinect input, displaying appropriatewalking or running animations and foot placementwhile still giving the Kinect control of the upper body.This allows a user to retain correct leg animationwhen exploring a virtual environment larger than

the Kinect’s capture area. The process of interfacingthe Kinect skeleton input with a new choreographertook minimal effort.

Jump Choreographer. Figure 9(b) illustrates theintegration of a simple jump choreographer whichuses a pre-recorded motion clip that is blendedonto the final display model when a jump istriggered. Jump motion is applied to the whole bodyand supersedes the locomotion choreographer whenactive. The root motion during a jump is procedurallycomputed to make the jump choreographer robustto varied environment configurations with differingjump distances.

Locomotion with Footstep Constraints. The previ-ously described locomotion controller precomputesparameters such as root displacement and speed formotions in the database which is used for blendingbetween motions to follow a given velocity command.In order to facilitate control policies with footstep-level precision [33], we extend the parameter space toinclude the relative transform of the current supportfoot based on the work in [58]. To offset the problem ofinsufficient coverage in the motion database, we useIK on the legs to enforce foot constraints. This allowsour animated characters to naturally place their feetto solve particularly challenging locomotion scenariosas illustrated in Figure 9(d).

7.2 Computational Performance

�

��

��

��

���

���

���

����

����

���

���

� � �� �� �� ��� ��� ���

������������ ������

������������ �

����������

�������

��� �����

Fig. 10. Update frequency for the character animation

and navigation components in ADAPT.

ADAPT supports approximately 150 agents withfull fidelity at interactive frame rates. Figure 10 dis-plays the update frequency for the animation andnavigation system (for our scenes, the computationalcost of behavior was negligible). This varies withthe complexity of the choreographers active on eachcharacter. The ADAPT animation interface and thepose dataflow graph has little impact on performance,

12

Fig. 9. (a) Controlling a character in ADAPT and physically interacting with the environment using the Kinect. (b)

A time-lapse view of the jump choreographer avoiding a moving obstacle. (c) Solving a challenging locomotion

problem with footstep-level precision.

and the blend operation is linear in number of chore-ographers. Each joint in a shadow is serialized with 74-byte float values, making each shadow 28 bytes perjoint. For 26 bones, the shadow of a full-body char-acter choreographer has a memory footprint of 728bytes. For 200 characters, the maximum memory over-head due to shadows is less than 1 MB. In practice,however, most choreographers use reduced skeletonswith only a limb or just the upper body, making theactual footprint much lower for an average character.

Separating character animation into discrete mod-ules and blending their produced poses as a post-processing effect also affords the system uniqueadvantages with respect to dynamic level-of-detail(LOD) control. Since no choreographer is architec-turally dependent on any other, controllers can beactivated and deactivated arbitrarily. Deactivated con-trollers can be smoothly faded out of control at anytime, and their nodes in the dataflow graph can bebypassed using the already-available blend weights.This drastically reduces the number of computedposes, and conserves processing resources neededfor background characters that do not require a fullrepertoire of actions. The system retains the abilityto re-activate those choreographers at any time if aspecific complex action is suddenly required. Sincechoreographers are not tightly coupled, no choreogra-pher needs to be made aware of the fact that any otherchoreographer has been disabled for LOD purposes.

7.3 Multi-Actor Simulations

The concluding narrative sequence shown in thevideo is simulated using several reusable authoredevents, which are activated using spatial and tem-poral triggers. Events once active, can be success-fully executed or interrupted by other triggers dueto dynamic events or user input. This produces arich interactive simulation where virtual characterscan be directed with a high degree of fidelity, withoutsacrificing autonomy or burdening the user with au-thoring complexity. Trigger management is handledby the ADAPT Behavior Scheduler described in Sec-tion 6.1.1. For our demo, most triggers were scheduledtemporally, and can be replaced by a more dynamic

event dispatcher. In the beginning, an event ensueswhere a character is given a phone and converseswhile wandering through the scene, gazing at objectsof interest. The phone conversation event successfullycompletes and the character hands back the phone.Spotting nearby friends invokes a conversation, whichis an extension of the event illustrated in Figure 7.The conversation is interrupted when a ball is thrownat one of the characters. The culprit flees from thecharacters, triggering a chasing event where the groupruns after the child. The chase fails as the child isable to escape through a crossing crowd of characters,which are participating in a group event to navigateto the theater and find a free chair to sit. We illustratesome of the trees used for this sequence in greaterdetail in Appendix B (supplemental).

7.4 Open Source Framework

We have released the software framework as an opensource library. The platform has seen interest frommultiple academic and industrial institutions, and isalready being used for a wide variety of applica-tions including development of narrative-driven in-teractive virtual worlds, enabling sound perceptionin virtual humans, the use of machine learning forcrowd simulation, high-fidelity character animation,multi-character navigation, simulating culturally ac-curate virtual populaces, and the development ofserious games for education and data collection. Theplatform can be downloaded as a Unity package athttp://cg.cis.upenn.edu/ADAPT/.

8 CONCLUSIONS

ADAPT is a modular, flexible platform whichprovides a comprehensive feature set for animation,navigation, and behavior tools needed for end-to-end simulation development. By allowing a userto independently incorporate a new animationchoreographer or steering system, and make thosecomponents immediately accessible to the behaviorlevel without modifying other existing systems,characters can very easily be expanded with newcapabilities and functionality. Our framework

13

enables, for the first time, seamless integration ofdiverse, modular, and easily extensible controllers foranimating autonomous virtual humans in complex3D environments. This system can be used toorchestrate sophisticated character movements aswell as complex multi-actor interactions for narrative-driven interactive virtual worlds using a graphicalbehavior authoring paradigm. All of this is enabledby a simple, powerful, and accessible core platform.

Limitations and Complications. Choreographers maysometimes need to be aware of major state changes inthe character’s pose caused by another choreographer.For example, we may wish to restrict the degree towhich the character can rotate its torso for gaze track-ing while the character is running. We accomplish thisusing the message broadcast system integrated intothe coordinator. When a character reaches a certainspeed, the locomotion choreographer can broadcastto all other choreographers that the character is in anIsRunning state. The gaze tracking choreographercan receive this message and restrict its maximumtorso rotation accordingly. This allows choreographersto cooperate without being explicitly aware of oneanother, and is a more extensible paradigm than deepintegration of controllers.

Interpolation between arbitrary poses generallyproduces smooth results in our system, with theexception of blends that linearly translate theposition of a character’s feet. This situation ariseswith our sitting choreographer, where the placementof a character’s feet while standing may not coincidewith the foot placement in the transition animationbetween standing and sitting. A linear blend hereresults in an unrealistic sliding of the foot despiteground contact. This can be resolved by usingbetter blending schemes or by using a more robustlocomotion system, as described in Section 7.1.

Future Work. Moving forward, we will continueto expand the animation and authoring capabilitiessupported by ADAPT. In addition to choreographersdescribed here, we want our platform to provide anarray of options for different kinds of motor skills,including climbing, and carrying objects with weight,as well as provide multi-modal sensory capabilitiessuch as agent hearing. We are also interested inimproving the virtual environment and developingextensible ways for characters to interact with theenvironment on a behavioral level. To ease the author-ing burden, we have developed an interface similarto smart objects for annotating the environment anddescribing the ways that characters can interact withit. We are particularly interested in extending theADAPT platform to develop solutions for the auto-mated scheduling of events to follow global narrativearcs. All of these improvements will allow us to applyour platform to other areas research, as ADAPT is

uniquely suited for producing the next generation ofnarrative-driven simulations.

ACKNOWLEDGMENTS

We acknowledge Dr. Ari Shapiro and Dr. BenSunshine-Hill for their discussions and contributionsto the ADAPT project. The research reported in thisdocument was performed in connection with Con-tract Numbers W911NF-07-1-0216 and W911NF-10-2-0016 with the U.S. Army Research Laboratory. Theviews and conclusions contained in this document arethose of the authors and should not be interpretedas presenting the official policies or position, eitherexpressed or implied, of the U.S. Army Research Lab-oratory, or the U.S. Government unless so designatedby other authorized documents. Citation of manufac-turers or trade names does not constitute an officialendorsement or approval of the use thereof. The U.S.Government is authorized to reproduce and distributereprints for Government purposes notwithstandingany copyright notation heron.

REFERENCES

[1] N. Pelechano, J. M. Allbeck, and N. I. Badler, Virtual Crowds:Methods, Simulation, and Control, ser. Synthesis Lectures onComputer Graphics and Animation, 2008.

[2] D. Thalmann and S. R. Musse, Crowd Simulation, Second Edition.Springer, 2013.

[3] M. Kapadia and N. I. Badler, “Navigation and steering forautonomous virtual humans,” Wiley Interdisciplinary Reviews:Cognitive Science, 2013.

[4] M. Kapadia, A. Shoulson, C. D. Boatright, P. Huang, F. Du-rupinar, and N. I. Badler, “What’s next? the new era ofautonomous virtual humans,” in MIG, 2012, pp. 170–181.

[5] L. Kovar, M. Gleicher, and F. Pighin, “Motion graphs,” ser.SIGGRAPH, 2002, pp. 473–482.

[6] O. Arikan and D. A. Forsyth, “Interactive motion generationfrom examples,” ACM TOG, vol. 21, no. 3, pp. 483–490, July2002.

[7] A. Witkin and Z. Popovic, “Motion warping,” ser. SIGGRAPH,1995, pp. 105–108.

[8] S. Menardais, F. Multon, R. Kulpa, and B. Arnaldi, “Motionblending for real-time animation while accounting for theenvironment,” ser. CGI, 2004, pp. 156–159.

[9] J. Min and J. Chai, “Motion graphs++: a compact generativemodel for semantic motion analysis and synthesis,” ACMTrans. Graph., vol. 31, no. 6, pp. 153:1–153:12, Nov. 2012.

[10] M. Dontcheva, G. Yngve, and Z. Popovic, “Layered acting forcharacter animation,” ACM Trans. Graph., vol. 22, no. 3, pp.409–416, July 2003.

[11] A. Shapiro, M. Kallmann, and P. Faloutsos, “Interactive motioncorrection and object manipulation,” in Interactive 3D graphicsand games, ser. I3D ’07. ACM, 2007, pp. 137–144.

[12] Y. Huang, M. Mahmudi, and M. Kallmann, “Planning hu-manlike actions in blending spaces,” in Intelligent Robots andSystems (IROS), 2011.

[13] J. Lee, J. Chai, P. S. A. Reitsma, J. K. Hodgins, and N. S. Pollard,“Interactive control of avatars animated with human motiondata,” ACM TOG, vol. 21, no. 3, pp. 491–500, 2002.

[14] R. S. Johansen, “Automated semi-procedural animation forcharacter locomotion,” Master’s thesis, Aarhus University,2009.

[15] Y. Liu and N. I. Badler, “Real-time reach planning for animatedcharacters using hardware acceleration,” ser. CASA, 2003, pp.86–93.

[16] A. W. Feng, Y. Xu, and A. Shapiro, “An example-based motionsynthesis technique for locomotion and object manipulation,”ser. I3D, 2012, pp. 95–102.

14

[17] P. Baerlocher and R. Boulic, “An inverse kinematics architec-ture enforcing an arbitrary number of strict priority levels,”Vis. Comput., vol. 20, no. 6, pp. 402–417, Aug. 2004.

[18] P. Faloutsos, M. van de Panne, and D. Terzopoulos, “Compos-able controllers for physics-based character animation,” ser.ACM SIGGRAPH, 2001, pp. 251–260.

[19] K. Yin, K. Loken, and M. van de Panne, “Simbicon: simplebiped locomotion control,” ACM TOG, vol. 26, no. 3, 2007.

[20] J. Pettre, M. Kallmann, and M. C. Lin, “Motion planning andautonomy for virtual humans,” in ACM SIGGRAPH classes,2008, pp. 42:1–42:31.

[21] L. Kavraki, P. Svestka, J.-C. Latombe, and M. Overmars, “Prob-abilistic roadmaps for path planning in high-dimensional con-figuration spaces,” IEEE-RAS, vol. 12, pp. 566 –580, 1996.

[22] M. Kallmann, “Shortest paths with arbitrary clearance fromnavigation meshes,” in Eurographics/SIGGRAPH SCA, 2010.

[23] C. Warren, “Global path planning using artificial potentialfields,” in IEEE-RAS, 1989, pp. 316–321 vol.1.

[24] A. Treuille, S. Cooper, and Z. Popovic, “Continuum crowds,”in ACM SIGGRAPH, ser. SIGGRAPH ’06, 2006, pp. 1160–1168.

[25] C. Reynolds, “Steering behaviors for autonomous characters,”1999.

[26] D. Helbing and P. Molnar, “Social force model for pedestriandynamics,” PHYSICAL REVIEW E, vol. 51, pp. 42–82, 1995.

[27] N. Pelechano, J. M. Allbeck, and N. I. Badler, “Controllingindividual agents in high-density crowd simulation,” in ACMSIGGRAPH/Eurographics SCA, 2007, pp. 99–108.

[28] S. Paris, J. Pettr, and S. Donikian, “Pedestrian reactive naviga-tion for crowd simulation: a predictive approach,” ComputerGraphics Forum, vol. 26, no. 3, pp. 665–674, 2007.

[29] J. van den Berg, M. C. Lin, and D. Manocha, “Reciprocalvelocity obstacles for real-time multi-agent navigation,” inICRA. IEEE, 2008, pp. 1928–1935.

[30] M. Kapadia, S. Singh, W. Hewlett, and P. Faloutsos, “Egocen-tric Affordance Fields in Pedestrian Steering,” in Interactive 3Dgraphics and games, ser. I3D ’09. ACM, 2009, pp. 215–223.

[31] M. Kapadia, S. Singh, W. Hewlett, G. Reinman, and P. Falout-sos, “Parallelized egocentric fields for autonomous naviga-tion,” The Visual Computer, pp. 1–19.

[32] S. Singh, M. Kapadia, B. Hewlett, G. Reinman, and P. Falout-sos, “A modular framework for adaptive agent-based steer-ing,” in ACM I3D, 2011, pp. 141–150.

[33] S. Singh, M. Kapadia, G. Reinman, and P. Faloutsos, “Foot-step navigation for dynamic crowds,” Computer Animation andVirtual Worlds, vol. 22, no. 2-3, pp. 151–158, 2011.

[34] M. Schuerman, S. Singh, M. Kapadia, and P. Faloutsos, “Situa-tion agents: agent-based externalized steering logic,” Comput.Animat. Virtual Worlds, vol. 21, pp. 267–276, May 2010.

[35] A. Lerner, Y. Chrysanthou, and D. Lischinski, “Crowds byexample,” CGF, vol. 26, no. 3, pp. 655–664, 2007.

[36] K. H. Lee, M. G. Choi, Q. Hong, and J. Lee, “Group behaviorfrom video: a data-driven approach to crowd simulation,” inACM SIGGRAPH/Eurographics SCA, 2007, pp. 109–118.

[37] M. Mononen, “Recast/Detour navigation library,” 2009. [On-line]. Available: http://code.google.com/p/recastnavigation/

[38] A. B. Loyall, “Believable agents: Building interactive person-alities,” Ph.D. dissertation, Carnegie Mellon University, 1997.

[39] K. Perlin and A. Goldberg, “Improv: a system for scriptinginteractive actors in virtual worlds,” ser. SIGGRAPH, 1996,pp. 205–216.

[40] A. Shoulson, F. Garcia, M. Jones, R. Mead, and N. I.Badler, “Parameterizing Behavior Trees,” in Motion in Games.Springer, 2011, pp. 144–155.

[41] Q. Yu and D. Terzopoulos, “A decision network framework forthe behavioral animation of virtual humans,” ser. SCA, 2007,pp. 119–128.

[42] M. Fleischman and D. Roy, “Representing intentions in acognitive model of language acquisition: Effects of phrasestructure on situated verb learning,” in AAAI ’07, pp. 7–12.

[43] J. Orkin, “Agent architecture considerations for real-time plan-ning in games,” in Interactive Digital Entertainment Conference.AAAI Press, 2005, pp. 105–110.

[44] M. Kapadia, S. Singh, G. Reinman, and P. Faloutsos, “ABehavior-Authoring Framework for Multiactor Simulations,”IEEE CGA, vol. 31, no. 6, pp. 45 –55, 2011.

[45] W. Li and J. M. Allbeck, “The virtual apprentice,” in IVA, 2012,pp. 15–27.

[46] U. Erra, B. Frola, and V. Scarano, “BehaveRT: a GPU-basedlibrary for autonomous characters,” in Motion in Games, ser.MIG’10, 2010, pp. 194–205.

[47] C. Stocker, L. Sun, P. Huang, W. Qin, J. M. Allbeck, and N. I.Badler, “Smart events and primed agents,” in Intelligent virtualagents, ser. IVA’10. Springer-Verlag, 2010, pp. 15–27.

[48] A. Shoulson and N. I. Badler, “Event-centric control for back-ground agents,” in International conference on Interactive DigitalStorytelling, ser. ICIDS, 2011, pp. 193–198.

[49] L. Sun, A. Shoulson, P. Huang, N. Nelson, W. Qin, A. Nenkova,and N. I. Badler, “Animating synthetic dyadic conversationswith variations based on context and agent attributes,” Com-put. Animat. Virtual Worlds, vol. 23, no. 1, pp. 17–32, Feb. 2012.

[50] Massive Software Inc., “Massive: Simulating life,” 2010,www.massivesofware.com.

[51] Autodesk, Inc., “Autodesk gameware - artificial intelligencemiddleware for games,” 2012. [Online]. Available:http://gameware.autodesk.com/

[52] S. Singh, M. Kapadia, P. Faloutsos, and G. Reinman, “An openframework for developing, evaluating, and sharing steeringalgorithms,” in MIG, 2009, pp. 158–169.

[53] A. Shapiro, “Building a character animation system,” ser. MIG,2011, pp. 98–109.

[54] M. Kallmann and S. Marsella, “Lncs ’05,” 2005, ch. Hierarchicalmotion controllers for real-time autonomous virtual humans,pp. 253–265.

[55] A. W. Feng, Y. Huang, M. Kallmann, and A. Shapiro, “Ananalysis of motion blending techniques,” in MIG, 2012.

[56] D. Slonneger, M. Croop, J. Cytryn, J. T. K. Jr., R. Rabbitz,E. Halpern, and N. I. Badler, “Human model reaching, grasp-ing, looking and sitting using smart objects international sym-posium on digital human modeling,” ser. Proc. InternationalErgonomic Association Digital Human Modeling, 2011.

[57] M. Kallmann and D. Thalmann, “Direct 3d interaction withsmart objects,” in ACM symposium on Virtual reality softwareand technology, ser. VRST ’99, 1999, pp. 124–130.

[58] B. J. H. van Basten, P. W. A. M. Peeters, and A. Egges, “Thestep space: example-based footprint-driven motion synthesis,”CAVW, vol. 21, no. 34, pp. 433–441, May 2010.

Alexander Shoulson is Ph.D. student atthe University of Pennsylvania, supervisedby Norman I. Badler. His research focuseson interactive narrative, character animation,and behavior authoring for virtual humans.

Nathan Marshak is a Master’s student inComputer Graphics and Game Technologyat the University of Pennsylvania. He grad-uated with a BSE in Computer Science fromthe University of Pennsylvania in 2013. Hiscurrent interests include animation, control,and shape deformation.

Mubbasir Kapadia is an Associate Re-search Scientist at Disney Research Zurich.His research focuses on the simulation offunctional, purposeful autonomous virtualhumans. Kapadia has a PhD in ComputerScience from the University of California, LosAngeles.

Norman I. Badler is the Rachleff Professorof Computer and Information Science at theUniversity of Pennsylvania. His research in-volves developing software for human andgroup behavior modeling and animation. Heis the founding Director of the SIG Centerfor Computer Graphics and the Center forHuman