Reasoning about humans and its use in a cognitive controlarchitecture for a collaborative robot

Rachid Alami, AurélieClodic

CNRS, LAAS, 7 avenue ducolonel Roche, F 31400

Toulouse, FranceUniv de Toulouse, LAAS, F

31400 Toulouse, Francerachid.alami@laas.fr,aurelie.clodic@laas.fr

Raja ChatilaInstitut des Systèmes

Intelligents et de RobotiqueUniversité Pierre et MarieCurie-Paris6, CNRS UMR

72224, Place Jussieu 75252 Paris

Cedex 05 - Franceraja.chatila@isir.upmc.fr

Séverin LemaignanComputer-Human Interactionin Learning and Instruction

Laboratory (CHILI)Ecole Polytechnique Fédérale

de Lausanne (EPFL)CH-1015 Lausanne,

Switzerlandseverin.lemaignan@epfl.ch

KeywordsHuman-robot collaboration, Affordances, Human-aware taskplanning

1. INTRODUCTIONWe discuss here a decisional framework for human-robot in-teractive task achievement that is aimed to allow the robotnot only to accomplish its tasks but also to produce behav-iors that support its engagement vis-a-vis its human partnerand to interpret human behaviors and intentions. We haveadopted a constructive approach based on effective individ-ual and collaborative skills. The system is comprehensivesince it aims at dealing with a complete set of abilities ar-ticulated so that the robot controller is effectively able toconduct a collaborative task with a human partner in a flex-ible manner These abilities include geometric reasoning andsituation assessment based essentially on perspective-takingand affordances, management and exploitation by the robotof each agent beliefs (human and robot) in a separate cog-nitive model, human-aware task planning and human androbot interleaved plan achievement.

2. FRAMEWORKFigure 1 illustrates the main concepts on which the system isbuilt. Collaboration happens as a consequence of an explicitrequest of the human to satisfy a goal or because the robotfinds itself in a situation where it is useful if not manda-tory. In both cases, the robot has a goal to satisfy. Animportant issue is the notion of engagement, a process inwhich the robot will have to establish, maintain and termi-nate a connection with a human partner. This covers goalestablishment, selection of an incremental refinement of thetask that is intended to be achieved, and execution controlincluding monitoring, and even influencing, human task per-

??Hello!

Multi-modal Dialog

MutualActivity

Observation

Figure 1: The conceptual framework: Human andRobot share task and space. They are in a mutualobservation situation. Robot explicity reasons aboutthe fact that it is (or has to be) perceived by itshuman partner. Planning is performed at high-levelsymbolic as well as as geometric level

formance and his/her commitment to the goal. The humaninvolvement may range from a direct participation to thetask achievement, to a simple “acceptance” of robot activityin his/her close vicinity.

Our robot is controlled by a three layer architecture [1]. Wedescribe in the sequel the main activities performed by therobot controller (Figure 2.): (1) Situation assessment andcontext management (2) Goals and plans management (3)Action refinement, execution and monitoring. Other deci-sional activities, such as situated dialog ([9, 4], not presentedhere) have been developed that use the same set of compo-nents.

2.1 Situation Assessment andContext Management

Geometric reasoning plays a central role in our architec-ture. It is performed by a component called SPARK (Spa-tial Reasoning and Knowledge [12]). It is responsible forgeometric information gathering and it embeds a numberof decisional activities linked to abstraction (symbolic facts

Symbolic facts and beliefsmanagement

Human-aware Motion and Manipulation Planning

motion planrequests

Human-aware symbolic task planning

shared plans

Geometric & Temporal Reasoning

Symbolic Facts Production World Update ManagementManagement of environment

geometric model

world model and agents beliefs

symbolicfacts

action monitoringand management ofposition hypotheses

Goal / Plan ManagementAction Refinement,

Execution and MonitoringSituation assessment

and context management

12 3

Robot Controller

ActuationPan Tilt Unit, Gripper, Arm, Wheels

PerceptionTags ARToolKit, Kinect, Motion Capture

Sensorimotor layer

world model and agents beliefs

Figure 2: Architecture of the robot control system

production) and inference based on geometric and temporalreasoning. SPARK maintains all geometric positions andconfigurations of agents, objects and furniture coming fromperception and previous or a priori knowledge. Reasoningabout human perspective allows to compute facts such as:GREY TAPE isBehind HUMAN1, GREY TAPE isVis-ibleBy HUMAN1.Monitoring human activity is crucial to maintain a coherentstate of the world. Full human action and activity monitor-ing is a difficult task that requires knowledge and reasoningboth on high level facts like goals, intentions and plans, aswell as bottom-up data from agent and object motions. Sim-ple temporal and geometric reasoning on human hand tra-jectories and potential objects placements can provide someuseful clues for high level human monitoring processes.The facts produced by the geometric and temporal reason-ing component are stored in a central symbolic knowledgebase, called ORO. Besides acting as a facts database, theORO platform [3] exposes several functions: operations onknowledge statements relying on inference (through a con-tinuous first-order logic classification process), managementof per-agent symbolic models, and also higher cognitive andhuman-robot interaction related functionalities like catego-rization of sets of concepts and natural language ground-ing [4].

2.2 Goal and Plan ManagementIn order to devise how a given goal can be accomplished,the robot has to elaborate a plan, i.e. a set of actions to beachieved by itself and its human partners. This is the role ofHATP [2] (for Human Aware Task Planner). HATP is basedon a Hierarchical Task Network (HTN) refinement which

performs an iterative task decomposition into sub-tasks untilreaching atomic actions [7]. One key feature is that HATP isable to produce plans for the robot actions as well as for theother participants (humans or robots). The resulting plan,called “shared plan” is a set of actions that form a streamfor each agent involved in the goal achievement. Dependingon the context, some “shared plans” contain causal relationsbetween agents. It can be tuned by setting up different costsdepending on the actions to apply and by taking into accounta set of constraints called social rules. This tuning aims atadapting the robot behavior according to the desired levelof cooperation of the robot [2]. Depending on the contextand on the shared plan elaborated by HATP for a givengoal, the robot controller decides to execute an action or toask its human partner to do it. Actions feasibility by thehuman or the robot could be regularly reconsidered basedon reachability/visibility constraints.

2.3 Action refinement,Execution and Monitoring

Let us take a simple example to illustrate a full run of thesystem. We assume here that the robot (and the human)has been given the joint goal “CLEAN TABLE”. For HATP,this means putting all tapes that are currently on the tablein the trashbin. Depending on the state of the world andagent preferences, different plans are produced.There is only one tape on the table and it is is reachableonly by the robot while the trashbin is reachable only bythe human.Figure 2.3 illustrates the main processes occurring duringa multi-step human-robot collaborative goal achievement.The plan produced is quite straightforward and is shown in

Figure 3: The main processes occurring during a multi-step human-robot collaborative goal achievement

the third row called “Goal and Plan”. It consists in 4 suc-cessive actions involving the robot and the human. Robotgrasps the tape and then places it on the table at a positionwhere it is visible and reachable for the human. Human thenis asked to pick the tape and to throw it in the trashbin.The first row, named “Cameras”, shows several snapshotscorresponding to various execution steps. Snapshot 1 corre-sponds to the initial situation. Snapshots 2, 3, 4 and 5 givethe state after the successive achievement of the four actionsin the plan. The second row, named “3D Model”, shows thedisplay of SPARK at the same instants. The fourth row,called “Robot Speech Acts”, illustrates robot speech actsproduced along the execution to inform the human partnerabout goal and plan creation and status and to verbalizethe actions that the human is asked to execute. The fifthrow illustrates robot knowledge on itself and on the objects.The sixth row illustrates the robot knowledge about the hu-man state. The seventh row gives ongoing robot action withaction preconditions and effects assessment as well as mo-tion execution tasks. The eighth row gives ongoing humanaction with action preconditions and effects assessment andmonitoring activity.

3. ACKNOWLEDGMENTSThis work has been conducted within the ANR-CONTINTROBOERGOSUM (DECISION ANR-12- CORD-0030-02).

4. REFERENCES[1] R. Alami, R. Chatila, S. Fleury, M. Ghallab, and

F. Ingrand. An architecture for autonomy.International Journal of robotics Research, SpecialIssue on Integrated Architectures for Robot Control andProgramming, 17(4), 1998.

[2] S. Alili, V. Montreuil, and R. Alami. HATP taskplanner for social behavior control in autonomousrobotic systems for hri. In The 9th InternationalSymposium on Distributed Autonomous RoboticSystems, 2008.

[3] S. Lemaignan, R. Ros, L. Mosenlechner, R. Alami, andM. Beetz. Oro, a knowledge management module forcognitive architectures in robotics. In Proceedings of the2010 IEEE/RSJ International Conference onIntelligent Robots and Systems, 2010.

[4] S. Lemaignan, A. Sisbot, and R. Alami. Anchoringinteraction through symbolic knowledge. In Proceedingsof the 2011 Human-Robot Interaction Pioneersworkshop, 2011.

[5] S. Lemaignan, Raquel Ros, E. Akin Sisbot, Rachid

Alami, Michael Beetz BroquAlre, D. Sidobre, R. AlamiGrounding the interaction: anchoring situated discoursein everyday human-robot interaction Acceptable RobotMotions International Journal of Social Robotics,Volume 2, Issue 3, pp 329-343, April 2012

[6] J. Mainprice, E.A. Sisbot, L. Jaillet, J. Cortes,R. Alami, and T. Simeon. Planning human-awaremotions using a sampling-based costmap planner. InIEEE International Conference on Robotics andAutomation, 2011.

[7] D. Nau, T. C. Au, O. Ilghami, U. Kuter, J. W.Murdock, D. Wu, and F. Yaman. Shop2: An htnplanning system. Journal of Artificial IntelligenceResearch, pages 379–404, 2003.

[8] A.K. Pandey and R. Alami. Mightability maps: Aperceptual level decisional framework for co-operativeand competitive human-robot interaction. InIEEE/RSJ International Conference on IntelligentRobots and Systems, 2010.

[9] R. Ros, S. Lemaignan, E.A. Sisbot, R. Alami,J. Steinwender, K. Hamann, and F. Warneken. Whichone? grounding the referent based on efficienthuman-robot interaction. In 19th IEEE InternationalSymposium in Robot and Human InteractiveCommunication, 2010.

[10] E. A. Sisbot, A. Clodic, R. Alami, and M. Ransan.Supervision and motion planning for a mobilemanipulator interacting with humans. 2008.

[11] E.A Sisbot, L.F. Marin-Urias, X. BroquAlre, D.Sidobre, R. Alami Synthesizing Robot MotionsAdapted to Human Presence. A Planning and ControlFramework for Safe and Socially Acceptable RobotMotions International Journal of Social RoboticsVolume 2, Issue 3, pp 329-343, 2010

[12] E. A. Sisbot, R. Ros, and R. Alami. Situationassessment for human-robot interaction. In 20th IEEEInternational Symposium in Robot and HumanInteractive Communication, 2011.

[13] E. A. Sisbot, and Rachid Alami A Human-AwareManipulation Planner IEEE Transactions on Robotics

”ISSN : 1552-3098, Digital Object Identifier :10.1109/TRO.2012.2196303, 2012.