Dynamically Formed Human-Robot Teams Performing Coordinated Tasks

M. B. Dias, T. K. Harris, B. Browning, E. G. Jones, B. Argall, M. Veloso, A. Stentz, A. RudnickySchool of Computer ScienceCarnegie Mellon University

Pittsburgh, Pennsylvania 15213Email: {mbdias,tkharris,brettb,egjones,bargall,mmv,axs,air}@cs.cmu.edu

Abstract

In this new era of space exploration where human-robotteams are envisioned maintaining a long-term presence onother planets, effective coordination of human-robot teamsis paramount. Two critical research challenges that must besolved to realize this vision are the human-robot team chal-lenge and the pickup-team challenge. In this paper, we ad-dress these two challenges, propose a novel approach to solveboth challenges, and evaluate our approach in the newly in-troducedtreasure huntdomain.

IntroductionWe have entered a new era of Space exploration that requiresthe sustained presence of human-robot teams on other plan-ets. Meeting this requirement entails solving many researchchallenges. Two critical challenges to be solved are effec-tive coordination of human-robot teams and enablingpickupteams. In response to these challenges, the vision that drivesthe reported work is that humans and robots will dynami-cally form teams to solve complex tasks by efficiently join-ing their complementary capabilities.

The Pickup Team challenge is to dynamically form teamsof robots (and possibly humans) given very little a priori in-formation. That is, team members may have only minimalprior knowledge of each others’ behavior, the tasks at hand,and the environments they operate in, but are able to com-bine effectively. There are several additional reasons why anincreased understanding of pickup teams is needed. First,the development of large teams or teams of expensive ro-bots at the same site at the same time is impractical. Thislimitation currently hinders multi-robot research. Success-ful pickup teams will facilitate further research by allow-ing separate researchers to easily pool their robots to cre-ate teams for further study. Second, robots may be neededfor emergency tasks where there may be insufficient timeto hand-engineer the coordination mechanisms before taskexecution. Pickup teams enable robot teams to be formedon very short notice for such tasks. Third, as robots fail,get lost, or otherwise malfunction, it is often necessary tosubstitute or add new robots. Successful pickup teams willallow the integration of new robots into existing teams, and

Copyright c© 2005, American Association for Artificial Intelli-gence (www.aaai.org). All rights reserved.

also enable teams of heterogeneous robots to perform effi-ciently under dynamic and uncertain conditions. Thus, theoverall research challenge is to provide a principled method-ology for creating pickup teams. This paper presents a firstapproach to address this challenge.

The human-robot team coordination challenge further re-quires robust team operation across multiple environments,team capabilities applicable across humans and multiple ro-bot types, effective multi-human-multi-robot interaction ina team setting, and teams that improve over time. Compe-titions, such as RoboCup, have been effective in focusingefforts to overcome some of these challenges (Nodaet al.1998). However, these competitions focus on part of theoverall problem and do not generally address teams formedin an ad-hoc manner, complex environments beyond a well-defined soccer field, and the complexities of heterogeneousteams. This paper focuses on dynamically forming teamsof humans and heterogeneous robots to perform tasks thatrequire coordination. The robots have limited individual ca-pabilities, can sense different information about their envi-ronment, and can be assigned abstract tasks for execution.Robots can solve primitive tasks in different ways depend-ing on the robot capabilities and the prevailing environmen-tal conditions.

We propose a novel approach to solving both the pickupteam challenge and the human-robot team coordination chal-lenge. The proposed approach combines and extends threepreviously proven components: TheTraderBotsmarket-based coordination approach for efficient and flexible allo-cation of tasks to teams and roles to team-members,Playsfor synchronizing team-execution of coordinated tasks, andthe multi-agent multi-modal dialog systemTeamTalk. Thisapproach is implemented on a team consisting of a human, aSegway RMP robot, and a Pioneer IIDX robot, and demon-strated in a treasure hunt scenario.

The Treasure Hunt DomainTo investigate the coordination of human-robot pickupteams, we require a domain that will allow for dynamic andheterogeneous team formation, encourage coordination andtight coupling between team members, and provide a metricagainst which to compare team performances. We believethe treasure hunt domain, jointly proposed by researchers atThe Boeing Company and at Carnegie Mellon University,

provides for each of these characteristics.The treasure hunt domain consists of human-robot teams

competing in and exploring an unknown space. The robotsare heterogeneous, and the particular capabilities necessaryto accomplish hunt tasks are distributed throughout the team.The hunt tasks require a team to acquire specific objects,the treasure, within an unknown or partially-known environ-ment. Thus a representation of the world must be built asit is explored, and the treasure must be identified and thenlocalized within the built representation. Team coordinationfollows as a direct necessity, as the abilities required to per-form each of these tasks are distributed throughout the teammembers.

The ultimate goal with respect pickup team formation isspeed and flexibility. Within this domain, not only are teamscreated quickly and on the fly, but each member has no priorknowledge about the abilities of its potential teammates; arobot knows of its own capabilities only. Inherent to the defi-nition of a hunt task are the abilities necessary to accomplishit. Communication between potential pick-up team mem-bers is therefore carried out at the time of team formation, toensure the satisfaction of all capabilities requirements.

This domain can also be extended to include an adversar-ial environment in which to execute the hunt tasks. Teamscan compete against the clock, with the intent of collectingas much treasure as possible within the allotted time or com-pete against other dynamically formed teams. A metric forteam performance can be the amount of treasure collectedby the team in a given time period, or conversely, the timetaken by a team to collect a percentage of the total hiddentreasure.

In our specific treasure hunt implementation, potentialteam members include humans, Pioneer robots and the ro-botic Segway RMP platform provided by Segway, LLC (seeFigure 1). The Pioneer robot is equipped with a SICK laserand gyroscope, and is therefore able to both construct a mapof an unknown environment, and localize itself upon thatmap. The Segway robot has been outfitted with two cam-eras, by which it is able to visually identify both the Pioneerrobot and the treasure. The presented task is to search forand retrieve treasure. To explore an area while searching forthe treasure, the Pioneer is able to navigate and build a map,while the Segway is able to follow the Pioneer and searchfor treasure. Upon treasure identification, the robots mustreturn home with the treasure. By localizing on its createdmap, the Pioneer is able to determine the home location; theSegway then follows the Pioneer as it proceeds home. Teamcoordination between the robots during execution is accom-plished jointly via the visual identification of the Pioneer bythe Segway, and by communication between the two robotsshould this visual link be lost (in which case the Pioneer iscommanded to pause, until seen by the Segway). Commu-nication additionally occurs when the Segway informs thePioneer that treasure has been found.

The treasure hunt domain satisfies the criterion set for thestudy of the performance of human-robot pickup teams. Itoffers a number of challenging aspects, including robust andefficient operation in unconstrained environments, and ad-hoc team formation. Efficient execution requires a coordi-

Figure 1: The left figure shows a Segway robot, while theright figure shows the pioneer robots.

nated search of the space and the maintenance of an accu-rate shared knowledge about the space. As such, this domainprovides a rich environment in which to push the boundariesof adaptive, autonomous robotics.

Component TechnologiesIn this section we review our current approaches to team-work – Skills, Tactics, and Plays (STP) for team coordi-nation in adversarial environments, and TraderBots for ef-ficient and robust role assignment in multi-robot tasks.

STP: Skills, Tactics, and PlaysVeloso and colleagues (Bowling, Browning, & Veloso 2004)introduce a skills, tactics, and plays architecture (STP) forcontrolling autonomous robot teams in adversarial environ-ments. In STP, teamwork, individual behavior, and low-levelcontrol are decomposed into three separate modules. Rel-evant to our implementation are Plays which provide themechanism for adaptive team coordination. Plays are thecentral mechanism for coordinating team actions. Each playconsists of the following components:(a) a set of roles foreach team member executing the play,(b) a sequence of ac-tions for each role to perform,(c) an applicability evaluationfunction,(d) a termination evaluation function,(e) a weightto determine the likelihood of selecting the play.

Each play is a fixed team plan that describes a sequence ofactions for each role in the team toward achieving the teamgoal(s). Each of the roles is assigned to a unique team mem-ber during execution. The role assignment is based on thebelieved state of the world and is dynamic (e.g., role A maystart with player 1, but may switch to player 3 as executionprogresses). Note that the role assignment mechanism is in-dependent of the play framework.

The concept of plays was created for domains where tightsynchronization of actions between team members is re-quired. Therefore, the sequence of tactics to be performedby each role is executed in lock step with each other role inthe play. Hence, the play forms a fixed team plan wherebythe sequence of activities is synchronized between teammembers.

As not all plans are appropriate under all circumstances,each play has a boolean evaluation function that determinesthe applicability of the play. This function is defined on theteam’s belief state, and determines if the play can be exe-cuted or not. Thus, it is possible to define special purposeplays that are applicable only under specific conditions aswell as general-purpose plays that can be executed undermuch broader conditions. Once executed, there are two con-ditions under which the play can terminate. The first is thatthe team finishes executing the team plan. Each play in-cludes an evaluation function that determines whether theplay should be terminated. As with applicability, this evalu-ation function operates over the team’s belief state. Hence,the second means of ending a play is if the termination eval-uation function determines that the play should end, eitherbecause it has failed or is successful.

Team strategy consists of a set of plays, called a play-book, of which the team can execute only one play at anyinstant of time. A play can only be selected for executionif it is applicable. From the set of applicable plays, one isselected at random with a likelihood that is tied to the play’sweight. The plays are selected with a likelihood determinedby a Gibbs distribution from the weights over the set of ap-plicable plays, which means the team strategy is, in effect,stochastic. This strategy is desirable in adversarial domainsto prevent the team strategy being predictable, and thereforeexploitable by the opponent.

TraderBotsTraderBots, developed by Dias and Stentz (Dias 2004) is acoordination mechanism, inspired by the contract net pro-tocol by Smith (Smith 1980), is designed to inherit the ef-ficacy and flexibility of a market economy, and to exploitthese benefits to enable robust and efficient multi-robot co-ordination in dynamic environments. A brief overview ofthe TraderBots approach is presented here to provide con-text for the reported experimental results and analysis.

Consider a team of robots assembled to perform a par-ticular set of tasks. Consider further, that each robot in theteam is modeled as a self-interested agent, and the team ofrobots as an economy. The goal of the team is to completethe tasks successfully while minimizing overall costs. Eachrobot aims to maximize its individual profit; however, sinceall revenue is derived from satisfying team objectives, the ro-bot’s self-interest equates to doing global good. Moreover,all robots can increase their profit by eliminating excess cost.Thus, to solve the task-allocation problem, the robots runtask auctions, and bid on tasks in other robots task auctions.

If the global cost is determined by the summation of indi-vidual robot costs, each deal made by a robot will result inglobal cost reduction. Note that robots will only make prof-itable deals. Furthermore, the individual aim to maximizeprofit (rather than to minimize cost) allows added flexibilityin the approach to prioritize tasks that are of high cost andhigh priority over tasks that incur low cost but provide lowervalue to the overall mission. The competitive element of therobots bidding for different tasks enables the system to de-cipher the competing local information of each robot, whilethe currency exchange provides grounding for the compet-

Figure 2: TeamTalk system architecture

ing local costs in terms of the global value of the tasks beingperformed.

TeamTalkThe human interface to the robots, TeamTalk (Harriset al.2004), is a multi-modal multi-agent dialog system, whichaccommodates multiple dialog agents and supports multi-ple simultaneous conversations. TeamTalk is based on theRavenclaw dialog manager (Bohus & Rudnicky 2003), ageneralized framework that makes use of task-tree repre-sentations to manage interaction between the user and thecomputer (in the present case, multiple robots). In additionto Ravenclaw, TeamTalk makes use of various spoken lan-guage technologies developed at Carnegie Mellon. Theseinclude Sphinx-II (Huanget al. 1993) for automatic speechrecognition and Phoenix (Ward 1994) for context-free gram-mar parsing. Language generation is template based andsynthesis (the Kalliope module) uses Swift, a commercial-ized version of the Festival synthesizer (Blacket al. 2004).A diagram of the system architecture, as configured for onehuman and two robots is shown in Figure 2.

The TeamTalk system provides a framework for exploringa number of issues in human-robot communication, includ-ing multi-participant dialog and grounding. We will brieflydiscuss some of these issues and describe our current solu-tions.

Managing multi-participant dialogs Building systemswith multiple human and robot participants presents severalchallenges. After considering several alternatives, we settledon the following design. Each human participant is givena complete interface hosted on a single computer incorpo-rating the components that perform speech recognition, un-derstanding, generation, and synthesis. An alternative andseemingly natural solution would have been to give each ro-bot its own spoken language capability, but this anthropo-morphism is not necessary. A single transducer for the hu-man also allows us to mitigate the effects of environment anddistance. Associating the interface directly with the humanalso allows us to use a multi-modal interface that combinesspeech and gesture in a single device (a tablet computer).

Finally, there is the issue of cost as the number of robotsincreases.

TeamTalk creates a stateful dialog manager for each ro-bot; this manager handles all of the human-side communi-cation. TeamTalk automatically spawns a new dialog man-ager whenever the system detects a hitherto unknown robot.This mechanism allows the system to deal gracefully withthe appearance of new robots on the team.

To support flexible communication, user input is broad-cast to all robots on the team, much as one might encounteron a shared radio channel. Broadcasting also allows us tosupport more complex group addressing functions (see be-low) and creates the conditions for beneficial eavesdropping.

Grounding concepts and clarification dialog Robustmechanisms for grounding are an essential part of interac-tion between agents, particularly in open domains wherenovel events and entities can occur. The Treasure Hunt callsfor interactive techniques that allow humans and robots todevelop common shared understanding of the environmentthat they are operating in. For example, a newly activatedrobot needs to communicate its capabilities to the human bytransmitting an ontology (and associated language) that canbe used to configure understanding, generation and popu-late the sub-task library. Grounding is also performed dur-ing activities. For example, a shallow but useful form ofgrounding is agreeing on names for landmarks (with nameschosen from a closed set of labels). A more complex ver-sion would involve introducing new labels to the system(given that task-specific names make more sense than ab-stract ones). We can go beyond that and have a system thatcan generalize mappings between labels and features in theenvironment (”Call that a doorway”, ”Now go to the door-way”). Current capability includes only fixed labeling.

Clarification deals with confusions and ambiguities incommunication due either to processing (for example recog-nition errors) or actual ambiguities (”I know of two doors.Which one do I go to?”). In the case of recognition or un-derstanding errors clarification is triggered by confidencemetrics computed for recognition accuracy, grammar cov-erage, concept history, the task, and per-robot policies. Inthis way, the system can conservatively ground concepts forhigh risk tasks, and yet liberally and efficiently execute lowrisk tasks. An example, of a per-robot policy is as follows:the Pioneer robot with its laser-based collision detection willtranslate on command, only asking for command confirma-tion when there is a very noisy speech signal; conversely,the Segway SMP, with its high momentum, poor odometry,and total lack of collision detection, almost always asks fortranslation confirmation.

ImplementationWe have attempted to address two main challenges in ourimplementation. The first is the dynamic formation ofpickup teams efficiently given heterogeneous robots. Wehave adapted the Traderbots system to perform this func-tion. Tasks are matched with plays consisting of a numberof roles; the roles of the play, when performed, should sat-isfy the requirements of the task. Each role in a play contains

a sequence of action primitives that can actually be executedby the robots, but a role can also require a certain set ofcapabilities. Robots only bid on roles that they have the ca-pability to perform, thus accommodating the heterogeneityof the robots and providing an efficient way for new kinds ofrobots with different sets of capabilities to represent them-selves to the system. Traderbots also requires that robotshave the ability to estimate the cost of actions; for instance,a cost may be the total distance that a role requires a robotto move. By minimizing cost in performing role allocationswe hope to not only get feasible solutions but also to havehigh efficiency.

The heterogeneity of the pickup teams demands that muchcare be taken during play execution; roles may depend oneach other, and all robots need to do their part to actuallydiscover, localize, and retrieve treasure. Thus, once the allo-cation has been performed, the tight coordination subsystemmust monitor and direct play execution.

The following describes our implementation of the sub-systems for dynamic pickup team allocation and tight coor-dination. The first part introduces the main components ofimplementation, and the second part of the section illustratessystem performance by describing the life cycle of a treasurehunt task as it moves through allocation to execution.

Implementation Components

To realize the full functionality of our system, a number ofdifferent processes must be run on each of the different ro-bots, as well as on a human operator’s workstation. The fol-lowing describes only those processes essential to the func-tion of the system.

The top layer of our implementation consists of a num-ber of Traders. Each agent, including the human operator,is assigned a Trader. A Trader is the agent’s interface to themarket. The Trader can introduce items to be auctioned toother Traders by sending a call for bids, can respond to callsfor bids with bids, can determine which bids are most benefi-cial for the auctioning agent, and can issue awards to biddersthat have won auctions. Each robot agent has a trader calleda RoboTrader.

In our system the human operator has a trader, called theOpTrader. Each human team member uses TeamTalk as aclient from which to communicate with the OpTrader. Inthis implementation, TeamTalk runs on a tablet PC, and eachhuman team member carries the tablet PC with them. Thetablet runs an instance of the TeamTalk software, providesfor pen-and-tablet-based i/o with the robots, and provides forspeech-based i/o though an attached headset microphone.

A human team member (each with an instance ofTeamTalk) is a TeamTalk client of the OpTrader, and eachTeamTalk instance spawns a Ravenclaw dialog manager foreach active robot,n × m stateful dialogs are held, wheren is the number of human team members (and TeamTalkinstances) andm is the number of robotic team mem-bers. Each TeamTalk instance contains a single automaticspeech recognition engine, parser, natural language genera-tion module, text-to-speech engine, and pen-and-tablet inter-face. For each robot that comes on-line, the interface spawns

a separate dialog management system and confidence anno-tation module.

The PlayManager forms the next component module. Theprimary role of the PlayManager is to select useful plays fora task and to coordinate the execution of activities in a playacross a small sub-team to perform a won task. The Play-Manager can select plays to be bid upon for a specified task,coordinate the execution of a play with the play managersfor the other assigned roles, execute the sequence of tasksfor its particular robot, and synchronize the activities, whererequired, between the different roles.

A final important component, the Robot Server, providesan interface between the PlayManager and the componentson the robot responsible for controlling the robot. A strengthof our system is that neither the PlayManagers nor theTraders need to know very much about how the control ofthe system is actually implemented, as the Robot Serverserves as the single point of contact. Thus the Robot Serverson the robots must understand a standard packet, cause Play-Manager commanded actions to be performed, and to sendaction statuses, but beyond that robot platform developersare free to develop their system in any fashion they wish.

The Treasure Hunt Task Life Cycle

SEARCH (8,12) (8,18) (14,18) (14,12)

SEARCH−1

PlayManager

RoboTrader

TaskSEARCH−2

PlayManager

RoboTrader

TaskSEARCH−3

PlayManager

RoboTrader

TaskSEARCH−4

PlayManager

RoboTrader

TaskPlay Play Play Play

PIONEER 1 SEGWAY 1 SEGWAY 2 PIONEER 2

Task

Auction Call

OpTrader

Figure 3: Parts (a) and (b) of the task life-cycle as discussedin the Implementation section.

A human operator utters (or pen gestures) a task If themessage was a pen gesture, it is passed directly to Helios.If the task is an utterance, it is decoded by Sphinx into oneor more hypotheses along with associated confidence scores.The hypotheses are routed to Phoenix, which parses each hy-pothesized utterance, and for each parse, generates the con-cepts and additional confidence annotation.

These concepts are routed tom instances of Helios, onefor each robot. Helios examines the acoustic and language

model confidences from Sphinx, additional confidence an-notations imparted by Phoenix’s parser coverage of the hy-potheses, and the expectation agenda generated by Raven-claw based on that robot’s dialog state. From all of thisevidence, Helios picks concepts to pass to the dialog man-ager, and assigns a final combined confidence value for eachunique concept that is passed along.

At this point, each robot’s dialog manager has receiveda set of concepts with associated confidences. Each dia-log manager decides whether it is the addressed robot in thedialog or it is simply eavesdropping on a conversation be-tween the human and another dialog partner. We allow forthe possibility of opportunistic eavesdropping, so that an un-addressed robot could, for example, develop a better contextfrom which to understand the next utterance that may ac-tually be addressed to it. The dialog manager may act onthe concepts, or it may engage in a grounding action on oneor more of the concepts. This decision is made based on apolicy that depends on the concept confidence history in thedialog, and the action to be performed (more on this below).The dialog is a mixed-initiative dialog, so that if certain con-cepts are provided that only partially complete some agenda,the dialog system may take initiative to obtain the missinginformation. For example, ”how far do you want me to gobackwards?”. Based on grounded concepts, the dialog man-ager sends messages to the back-end manager, which in turninserts tasks to anOpTraderserver.

A task is issued and enters the Trading system Initially,the human operator designates a SearchArea task, embed-ding the points of a bounding polygon in the task data. Thetask is sent to the OpTrader. The OpTrader creates an auc-tion call with the task data, serializes it, and sends it via thewireless network using UDP to all RoboTraders (see Figure3).

Each RoboTrader receives the task auction call and getsa matching play from the PlayManager The RoboT-raders receive the call for bids from the OpTrader. Theydeserialize the call and pass the task specification to their in-dividual PlayManagers over a UDP socket connection. Eachcontacted PlayManager will compare the task string againstthe applicability conditions for each play in its playbook. Itwill then select a play stochastically amongst the set of playsthat are applicable, and return this play to the RoboTrader.

The RoboTrader assesses the playThe RoboTrader nowhas a play matching the task and a set of roles. It then mustselect one of the roles for itself. For each role in the play, theRoboTrader first considers whether or not it possesses all thecapabilities required to perform that role. For all roles forwhich it is capable, the Trader then performs a cost evalua-tion.

In the treasure hunt we are largely concerned with mini-mizing the time it takes to accomplish the task. As our ro-bots move roughly the same speeds during play execution,we try to minimize distance traveled as an approximation oftime. If robots were heterogeneous with respect to speed this

Play − SEARCH 1

Role 1 − Cover area and map − Cost $105

Role 2 − Follow and look for treasure.

RoboTrader

Auction Call

SEGWAY 1

RoboTrader

RoboTrader

SEGWAY 2

RoboTrader

PIONEER 2

PIONEER 1

Figure 4: The RoboTrader, upon receiving a play from thePlayManager, selects a play for itself and produces a costestimate. It auctions the other role in the play to the otherrobots.

should be reflected in their costing function. For most roles,cost is computed as total metric path cost for goal points tobe visited in the role. Costing in our system is modular, soadditional or different costing functions can be added easilyas required.

Once a cost has been assigned to each role, the RoboT-rader selects the lowest cost role for itself, then prepares anauction call with all the remaining roles. This auction call isserialized and sent to the other Traders, as shown on the leftin Figure 4.

The other RoboTraders bid on the role auction Afterthe other RoboTraders receive and deserialize the role call,they determine their own cost for each role in the call. Forany role in the call that requires capabilities the Trader doesnot possess it assigns an infinite cost. For all roles that theTrader can perform, it assigns the cost determined by thecosting function. Note that the Trader must bid on all rolesfor which it is capable. The role bids are then returned toauctioneering RoboTrader (shown in Figure 5).

The RoboTrader bids on the task Once all bids are re-ceived or a timeout has expired the auctioneer RoboTraderthen determines the winner or winners of the role auction.All role bids are considered, and the lowest cost bid is se-lected. If that bid is non-infinite, the role is designated asassigned to the bidding Trader. As a Trader can only wina single role in a play, that Trader’s other bids are nullified.Additionally, all bids for that role from any other Trader arenullified. Then the lowest cost remaining bid is considered;this continues until all roles in the play are assigned or thereare no remaining bids.

If all roles from the play have been assigned, the RoboT-rader constructs a bid for the original task, with a cost that

SEGWAY 1

RoboTrader

Cost $180

Role 2

Role Bid

RoboTrader

PIONEER 2

RoboTrader

SEGWAY 2

Play − SEARCH 1

Role 2 − Cost $140

Role 1 − Cost $105

RoboTrader

PIONEER 1

Cost $140

Role 2

Role BidRole 2

Role Bid

Cost $ ∞

Figure 5: Receiving the bids for the remaining role, the Op-Trader selects the lowest cost bid.

RoboTrader

PIONEER 2

RoboTrader

SEGWAY 1

RoboTrader

PIONEER 1

$280Task Bid

$260Task Bid

OpTraderTask

Award

$300Task Bid

RoboTrader

SEGWAY 2

$245Task Bid

Figure 6: The bids from all plays are returned to the Op-Trader, who selects the lowest cost task bid and sends a taskaward to the winning bidder.

is summed over all assigned role cost estimates. The bid issent to the OpTrader. This process is shown in Figure 6.

Task and role awards are granted Once the OpTraderhas received bids from all RoboTraders or a call timeout hasexpired it awards the task to RoboTrader with the lowest costbid. An award message is sent to the winning RoboTrader.All other RoboTraders are informed they lost the auction.The winning RoboTrader then sends role award messages toall RoboTraders that were assigned roles in the winning play.Note that at this point the RoboTrader still has not acceptedthe task award.

Awards are accepted and play execution beginsThe ini-tial role bids made by the RoboTraders were non-binding.However, a role award is binding; once accepted, a role mustbe performed. If a RoboTrader has received no other roleaward in the interval between bidding and the arrival of thisaward, it accepts the role award. Otherwise it rejects the roleaward. If any of the role awards are rejected, or one of theTraders awarded a role does not respond to the award by atimeout, then the task award is rejected and the OpTrader

TaskAccept

RoboTrader

PlayManager

RobotServer

Role 1Action 1

Role 2

OpTrader

PIONEER 1

Play Search 1

Role 1 − Assigned to Pioneer 1

Role 2 − Assigned to Segway 2

SEGWAY 2

PlayManager

RobotServer

Role 2Action 1

Figure 7: After the RoboTrader has confirmed the availabil-ity of the winning Role bidder, it can send notice of taskacceptance to the OpTrader and forward the role assign-ments to the PlayManager. The PlayManager contacts theplay managers of all robots assigned a role and sends them adescription of the actions in their role. Action primitives aresent to the robot servers and the play execution begins.

must perform another auction.If all role awards are accepted, the RoboTrader sends a

task acceptance message to the RoboTrader. Then it sendsan execution message to the PlayManager detailing the finalrole assignments. This stage is shown in Figure 7. The Ro-boTrader’s role in the pickup team allocation is concludedexcept for relaying status information to the OpTrader whenthe play completes. Future work, however, will examine dy-namic re-assignment of roles if and when required.

The PlayManager begins play execution Once informedof the play to execute, and the list of assigned roles, the Play-Manager forms the subteam to execute the play. It does thisby contacting each of the subteam members’ RoleExecutorsand transmits the play in compressed XML format to them.The RoleExecutor is responsible for executing the assignedrole in the play. If any subteam members are essential andfail to acknowledge the play reception, or are not able tobe contacted, the play is terminated and reported as such tothe RoboTrader. Otherwise, execution proceeds by the Play-Manager informing each RoleExecutor to start operation.

At this point, each RoleExecutor becomes loosely cou-pled to the PlayManager. The RoleExecutor will execute itssequence of actions and will only inform the PlayManager oftermination (success or failure), or when it needs to synchro-nize with another role according to the play. To synchronize,the RoleExecutor contacts the appropriate teammate’s Role-Executor and informs the PlayManager for status-keeping

purposes.When each RoleExecutor reaches the end of its sequence

of actions to perform, it informs the PlayManager of suc-cessful termination, and when the team is complete the Play-Manager reports this to the RoboTrader. Alternatively, if arobot fails, or the time limit for execution is reached (as en-coded in the play itself), the play is terminated and reportedas such. Taken together, execution is distributed and looselycoupled.

The robots report Task completion messages, as well asother messages (e.g., status reports) originated by the ro-bots, are captured by the TeamTalk back-end manager andthe messages are routed to each robot’s corresponding He-lios just as for the human utterances. Helios assigns highconfidence to information which originates from robots, onthe presumption that robot communications channel is notnoisy.

Generally, robot-originated messages are interpreted asrequests by the robots to communicate to the human, andRavenclaw passes those messages directly to Rosetta. Mes-sages may also complete task agencies, allowing Ravenclawto take these off of the focus stack and modify the expecta-tion agenda as appropriate (Bohus & Rudnicky 2003). Mes-sages are also routed to the GUI controller to allow the sys-tem to modify the display or perform some other signalingaction. In the case of a completed task, this may change thecolor of the robot’s icon.

Rosetta uses template-based natural language generationto render concept terms into well-formed sentences. Thesentences generated by Rosetta are routed to Kalliope forsynthesis. Kalliope selects a different Swift voice for eachrobot, thus each robot speaks in its own distinct voice. Thishelps the user to keep better track of information.

Related WorkWithin the field of robotics, there has been considerableresearch into multi-robot coordination for a variety of do-mains and tasks (Mataric 1994; Balch & Arkin 1998;Gerkey & Mataric 2003). Many groups have focused on re-search questions relevant to robot teams particularly (Balch& Parker 2001). RoboCup robot soccer has offered a stan-dardized domain in which to explore multi-robot team-work in dynamic, adversarial tasks (Nodaet al. 1998;Nardi et al. 2004) (see alsohttp://www.robocup.org). Seg-way Soccer (Browninget al. 2005) is a new league withinthis RoboCup domain, which addresses the coordination ofheterogeneous team members specifically. The emphasis ofthese human-robot teams is equality, both physically, as bothride the Segway mobility platform, and with respect to deci-sion making power and responsibility.

How to effectively coordinate heterogeneous teams hasbeen an ongoing challenge in multi-robot research (Scerriet al. 2004; Kaminka & Frenkel 2005). However, no onehas focused explicitly on the principles underlying the build-ing of such highly dynamic teams when the a priori inter-action between individual robot developers is so minimal.Much of the existing research implicitly assumes that the

robot team is built by a group of people working closelytogether over an extended period of time. While some pre-vious research within the software agents community hasaddressed the coordination of simulated agents built by dif-ferent groups (Pynadath & Tambe 2003), none has chosen toaddress this pickup challenge for the coordination of multi-ple robots. We believe this research direction of forming dy-namic teams will greatly advance the science of multi-robotsystems.

Dialog agents are almost always designed with the tacitassumption that at any one time, there is a single agent anda single human, and that their communication channel istwo-way and exclusive. As a consequence such systems donot deal gracefully with additional speakers or agents in thechannel. One approach constructs an aggregated spoken di-alog front-end for a community of under-specified agents,for example (Harris 2004). These systems, however, sacri-fice the integration of domain knowledge into the dialog andsupport only shallow levels of interaction.

ConclusionsIn this paper, we presented the concept of pickup teams,where teams are formed dynamically from humans and het-erogeneous robots with no a priori experience of one an-other. We have described an appropriate domain for explor-ing the research issues related to pickup teams: multi-robottreasure hunts. Based on our prior work with synchronizedactivities using STP with plays and tactics combined withrobust multi-robot role assignment using the TraderBots’market-based architecture, we have proposed a new tech-nique to address the problem of pickup teams. We have alsodescribed issues in language-based multi-actor communica-tion in mixed human-robot teams and have presented mech-anisms appropriate to managing interaction in such settings.

AcknowledgmentThis work is principally funded by the Boeing CompanyGrant CMU-BA-GTA-1. The content of the information inthis publication does not necessarily reflect the position orpolicy of the Boeing Company and no official endorsementshould be inferred. The authors would also like to thankthe Boeing Company for their contributions to this projectand this paper. This work was also enabled in part by fund-ing from the Qatar Foundation for Education, Science andCommunity Development and from the U.S. Army ResearchLaboratory, under contract Robotics Collaborative Technol-ogy Alliance (contract number DAAD19-01-2-0012).

ReferencesBalch, T., and Arkin, R. 1998. Behavior-based formationcontrol for multiagent robot teams.IEEE Transactions onRobotics and Automation.

Balch, T., and Parker, L., eds. 2001.Robot Teams: FromDiversity to Polymorphism. AK Peters.

Black, A.; Clark, R.; Richmond, K.; and King, S. 2004.The festival speech synthesis system.http://www.cstr.ed.ac.uk/projects/festival/ .

Bohus, D., and Rudnicky, A. I. 2003. Ravenclaw: Dialogmanagement using hierarchical task decomposition and anexpectation agenda. InProceedings of European Confer-ence on Speech Communication and Technology, 597–600.Bowling, M.; Browning, B.; and Veloso, M. 2004. Plays aseffective multiagent plans enabling opponent-adaptive playselection. InProceedings of International Conference onAutomated Planning and Scheduling (ICAPS’04).Browning, B.; Searock, J.; Rybski, P. E.; and Veloso, M.2005. Turning segways into soccer robots.Industrial Robot32(2):149–156.Dias, M. B. 2004.TraderBots: A New Paradigm for Robustand Efficient Multirobot Coordination in Dynamic Envi-ronments. Ph.D. Dissertation, Robotics Institute, CarnegieMellon University, Pittsburgh, PA.Gerkey, B. P., and Mataric, M. J. 2003. Multi-robot taskallocation: analyzing the complexity and optimality of keyarchitectures. InProceedings of ICRA’03, the 2003 IEEEInternational Conference on Robotics and Automation.Harris, T. K.; Banerjee, S.; Rudnicky, A.; Sison, J.; Bo-dine, K.; and Black, A. 2004. A research platform formulti-agent dialogue dynamics. InProc. of the IEEE In-ternational Workshop on Robotics and Human InteractiveCommunication, 497–502.Harris, T. 2004. The speech graffiti personal universal con-troller. Master’s thesis, Carnegie Mellon University, Pitts-burgh.Huang, D.; Alleva, F.; Hon, H. W.; Hwang, M. Y.; Lee,K. F.; and Rosenfeld, R. 1993. The Sphinx-II speechrecognition system: An overview.Computer, Speech, andLanguage7(2):137–148.Kaminka, G., and Frenkel, I. 2005. Flexible teamwork inbehavior-based robots. InIn Proceedings of the NationalConference on Artificial Intelligence (AAAI-2005).Mataric, M. J. 1994.Interaction and Intelligent Behavior.Ph.D. Dissertation, EECS, MIT, Boston, MA. Available astechnical report AITR-1495.Nardi, D.; Riedmiller, M.; Sammut, C.; and Santos-Victor,J., eds. 2004.RoboCup 2004: Robot Soccer World CupVIII (LNCS/LNAI). Berlin: Springer-Verlag Press.Noda, I.; Suzuki, S.; Matsubara, H.; Asada, M.; and Ki-tano, H. 1998. RoboCup-97: The first robot world cupsoccer games and conferences.AI Magazine19(3):49–59.Pynadath, D. V., and Tambe, M. 2003. An automated team-work infrastructure for heterogeneous software agents andhumans.Autonomous Agents and Multi-Agent Systems7(1-2):71–100.Scerri, P.; Xu, Y.; Liao, E.; Lai, J.; and Sycara, K. 2004.Scaling teamwork to very large teams. InAAMAS’04.Smith, R. G. 1980. The contract net protocol: High levelcommunication and control in a distributed problem solver.IEEE Transactions on ComputersC-29(12):1104–1113.Ward, W. 1994. Extracting information from spontaneousspeech. InProceedings of the International Conference onSpoken Language Processing, 83–87.