Humanoid Robots as Cooperative Partners for People

paper by Breazeal, C., et al.. (2003)MIT Media Lab, Robotic Life Group

presentation by Kósa Máté Ágostoncognitive robotics @ Rijksuniversiteit Groningen 2010

building socially intelligent robots

• important implications for how we will be able to communicate with, work with, and teach robots in the future

• it is a critical competence for robots that will play a useful, rewarding, and long-term role in the daily lives of people

socially intelligent robot

• robots that show aspects of human-like social intelligence, based on deep models of human cognition and social competence

• contrasted to socially evocative / receptive / situated / embedded

• brings research closer to the „hard” problem of artificial intelligence (in small steps…)

Fong, T., Nourbakhsh, I. & Dautenhahn, K. (2002)

socially intelligent robot

• Why?– We anthropomorphize by default– Personality lends coherence and consistence to

behavior (to know someone is to predict his actions)

– Natural learning– Scalability reflects in trust and sincerity (for when

it gets out of hand see Blade Runner, Ridley Scott 1982)

theory of mind

• Assumption:– each participant has a set of mechanisms and

representations for predicting and interpreting other’s actions, emotions, beliefs, desires, and other mental states

• Derived models:– joint attention, representation, empathy,

intersubjectivity, reason (mental states to behavior), inference, social reference etc.

Collaborative approach vs. ML

• supervised learning techniques – the learning algorithm has no a priori knowledge about the structure of the state and action spaces, must discover any structure that exists on its own

• needs data, time, relatively stable enviroment• problems with generalizing• hard to guide for the laic• bridges machine learning with HMC

Collaborative approach vs. Humans

• we are innate teachers• we have a well-established social signaling • we have infrastructure• we have an affinity towards interdisciplinarity

Social Skills

• reciprocal cooperation is achieved with the goal to:– help the instructor maintain a good mental model

of the learner– help the learner leverage from instruction and

guidance to build the appropriate task models, representations, associations, etc.

• test of abilities: the button task

Social Skills

• Communication skill• Deictic reference• Joint attention• Mutual beliefs

Communication

• Conversational policies– Cohen et. al. (1990) argue that much of task-

oriented dialog can be understood in terms of Joint Intention Theory

– Modeled after analysis of master-novice task

• Turn-taking skills– Modeled after human model, very robust– Envelope displays (para-linguistic cues)

Communication

• Conversational policies– Cohen et. al. (1990) argue that much of task-

oriented dialog can be understood in terms of Joint Intention Theory

– Modeled after analysis of master-novice task

• Turn-taking skills– Modeled after human model, very robust– Envelope displays (para-lingvistic cues)

- same goal and the same plan of execution- different abilities, tools, partial knowledge and different beliefs referring to the state of the goal- communication is necessary to mobilize the potential

- same goal and the same plan of execution- different abilities, tools, partial knowledge and different beliefs referring to the state of the goal- communication is necessary to mobilize the potential

Communication

• Conversational policies– Cohen et. al. (1990) argue that much of task-

oriented dialog can be understood in terms of Joint Intention Theory

– Modeled after analysis of master-novice task

• Turn-taking skills– Modeled after human model, very robust– Envelope displays (para-lingvistic cues)

- same goal and the same plan of execution- different abilities, tools, partial knowledge and different beliefs referring to the state of the goal- communication is necessary to mobilize the potential

- same goal and the same plan of execution- different abilities, tools, partial knowledge and different beliefs referring to the state of the goal- communication is necessary to mobilize the potential

Organizational markersElaborationsClarificationsConfirmationsReferential elaborations Confirmations of successful identification

Organizational markersElaborationsClarificationsConfirmationsReferential elaborations Confirmations of successful identification

Communication

• Conversational policies– Cohen et. al. (1990) argue that much of task-

oriented dialog can be understood in terms of Joint Intention Theory

– Modeled after analysis of master-novice task

• Turn-taking skills– Modeled after human model, very robust– Envelope displays (para-linguistic cues)

Deictic reference

• Estimating gaze via estimating head-pose– pan / tilt / rotation– objects in 3D spatial map projected on gaze vector– camera on the wall (panoramic view)

• Pointing– background and depth map extraction– candidates fit to ellipse, then presence of pointing

finger is analyzed (kurtosis)– stereo camera ceiling-mounted (bird’s eye view)

Deictic reference

• Estimating gaze via estimating head-pose– pan / tilt / rotation– objects in 3D spatial map projected on gaze vector– camera on the wall (panoramic view)

• Pointing– background and depth map extraction– candidates fit to ellipse, then presence of pointing

finger is analyzed (kurtosis)– stereo camera ceiling-mounted (bird’s eye view)

Deictic reference

• Estimating gaze via estimating head-pose– pan / tilt / rotation– objects in 3D spatial map projected on gaze vector– camera on the wall (panoramic view)

• Pointing– background and depth map extraction– candidates fit to ellipse, then presence of pointing

finger is analyzed (kurtosis)– stereo camera ceiling-mounted (bird’s eye view)

Joint Attention

– seeing vs. attending (in baby humans 7-9 months)– referential looking (in baby humans 6-18 months)– proto-declarative pointing (in b.h. 9-12 months)– exploiting all these at 14 months of age (in b.h.)

• two entities looking at the same thing is not necessarily joint attention (necessary-not-suff)

• updating mutual belief with a common referent is closer to the human-human model

Joint Attention

• To keep in mind:– Attention focus (what gets the attention)– Referent focus (the “subject” of communication)– Saliency determines a list, not a particular object– perceptual/internal/socially cued saliency– Decay of saliency– Leonardo’s model of own foci– Leonardo’s model of instructor’s foci

Beliefs

– humans around the age of 3 note difference between perception and belief

temporal integration of perceptual input(composite instances of real-world objects)

percept tree > snapshot > belief .classification > data structure > create/update

Beliefs

– humans around the age of 3 note difference between perception and belief

temporal integration of perceptual input(composite instances of real-world objects)

percept tree > snapshot > belief .classification > data structure > create/update

• when the robot shares a belief with a human, the belief gets labeled as “mutual belief”

• human’s attentional and referent focus are updated for the belief concerned

• when the robot shares a belief with a human, the belief gets labeled as “mutual belief”

• human’s attentional and referent focus are updated for the belief concerned

Learning

• From “internal” demonstration– telemetry suit – robot interpolates exemplars using a dynamically

weighted blend of the recorded button pressing trajectories

• Names of things– social cue feedback

Learning

• Task structure– task is either a (sub)task or an action, hierarchically

organized– constraints exist as actions (currently used for sequential

constraints but are expandable)– task goals are more than the sum of (sub)task goals– a goal can be either a

• state-change in world (attain a state)• performance (just do it)

• Natural instruction

Performing in collaboration

• possible because of the goal-oriented approach (and the turn-taking implementation)

• communication of robot’s perceived SoW and intention leads to common ground which is the basis of joint intention/attention/planning

• knowledge of own abilities, negotiation of task with human

• importance of gestural cues during collaboration

Video time

Discussion• Knowing what matters– restraining search-space by saliency– temporal cues + joint attention

• Knowing what to try– collaboration contrasted with imitation and experiment

• Knowing how to recognize success/faliure– goal types: change desired/performance, goal progress

• Knowing how to explore• Knowing how to leverage the provided structure– experienced demonstration, mo’ generally social context