n“PaDY,” a Robot

Co-worker Concept Kazuhiro Kosuge

Department of Bioengineering and Robotics Graduate School of Engineering

Tohoku University Sendai 980-8579, JAPAN

http://www.irs.mech.tohoku.ac.jp�

A Robot Co-worker�

http://www.dti.dk/services/robot-co-worker-for-assembly/32733�

Danish Technological Institute� Roadmap for robot helpers�http://cordis.europa.eu/ictresults/index.cfm?section=news&tpl=article&id=91400�

Frida (ABB)�http://news.cnet.com/8301-17938_105-20056649-1.html�

http://www.spacedaily.com/reports/DLR_presents_innovations_in_robotics_at_AUTOMATICA_2012_999.html�

DLR�

“Robot Co-worker” Images from Google�

A Robot Co-worker Concept�

Human-Robot Cooperation (Kosuge, 1993)

A Robot Co-worker Concept�

FLAMMARION (July 3 1997)�

A Robot Co-worker � Outline�

•  Overview of Robotics (Center for Research and Development Strategies, Japan Science and Technology Agency)

•  Multiple Robots Coordination •  Robot Helpers •  Dance Partner Robot •  PaDY •  Conclusions

Overview of Robotics�

http://crds.jst.go.jp/singh/wp-content/uploads/09wr041.pdf�

Past,&Today&and&&Future&of&Robo3cs�

Ill7defined&Problem�&

������ �� ����"�������#

�!����

���������

Industrial Robots�

Human Environment�

Well7defined&Problem�&Structured Environment�

Robots in Human Environments�

www.care-o-bot.de�

www.aldebaran-robotics.com�

www.paro.jp�

www.anl.gov/Science_and_Technology/History/�Master-Slave Manipulator

Robotics�

•  Social Level

•  Service Level�– Service enablers�

•  Fundamental Technologies Level�

Social'Value�

Services�

Founda1ons�

JST, CRDS, 2009, Modified by Kosuge, August, 2011�

Social Values�

•  For Individualss rQuality of Life�•  For Communities rIndustrial Competitiveness�

– For Families�– For Industries�– For Local Government�– For Nations�

•  For the Globe rGlobal Issues�

JST, CRDS, 2009, Modified by Kosuge, August, 2011�

Robotics Research Field�

Social Value�

Services�

Emerging Technology

Fundamental�

• Robot Kinematics and Dynamics • Manipulation • Mobility • Actuation • Physics-based Control

• Robot Systems Integration • Human Robot Interaction • Real-world Real-time Intelligence • Spatio-temporal System Design • Sensing and Machine Cognition

Global Level� Community Level� Quality of Life

• Software framework • Social Concerns • Functional Safety�• Nano-micro Robotics�• Human Modeling�• Wearable Technology�• Service Contents Design�

• Cyborg (Cybernetic organism) • Stochasticity in Robotics • Performance evaluation and Benchmarking • Ambient intelligence • Autonomous Robots�• Teleoperation�• Robotic Emotion (artificial emotion)

m E`h\da`_X`fT^>Ka`\fad\`Z�m LTfgdT^>OXeagdVXe>Eib^adTf\a`>�T`W>DXhX^ab_X`f�

m PbTVX>Eib^adTf\a`�m DXXb>R`WXdeXT>T`W>R`WXdZdag`W>�Eib^adTf\a`�

m B`f\?fXddad\e_>>>>�OXeVgX>MbXdTf\a`>>>>�m NdXhX`f\a`>aY>I`YXVf\age>D\eXTeXe�

m Rf\^\f\Xe�m OXfT\^Xd@S[a^XeT^Xd�m QdT`ebadfTf\a`�m Ca__g`\VTf\a`�m PXdh\VX>I`Wgefd\Xe�m KXW\V\`X�m EWgVTf\a`�m OXeXTdV[>T`W>�DXhX^ab_X`f�

m KXW\V\`X�m Q[XdTbj�m DT\^j>J\YX>Bee\ef�m HXT^f[VTdX�m OX[TU\^\fTf\a`�m KX`fT^>VTdX�m JXTd`\`Z�m C[\^W>VTdX�m HageX]XXb\`Z�

m PXVgd\fj�m KaU\^\fj�m P[abb\`Z�m HaUUj�m E`fXdfT\`_X`f�m Pbadfe�m Ca_Yadf>J\YX�m STfV[�m Ca__g`\VTf\a`�

m GahXd`_X`f>�PXdh\VX�m BZd\Vg^fgdX�m FadXefdj�m F\e[Xdj�m K\`\`Z�m KT`gYTVfgd\`Z�m Ca`efdgVf\a`�m STefXe>QdXTf_X`f@>�KT`TZX_X`f�

Service/A

pplication-oriented R

obotics R

obotics Foundations

JST, CRDS, 2009, Modified by Kosuge, June, 2012�

����

PXdh\VX@Bbb^\VTf\a`?�

ad\X`fXW>OaUaf\Ve�

OaUaf\Ve�

Fag`WTf\a

`e�

R`\f>QXV[`a^aZ\Xe�

OaUaf\Ve>�Fag`WTf\a`e�

Bbb^\VTf\a`e@PXdh\VXe�Da_T\`>o�E^WXd^j>CTdX�

Da_T\`>p�BZd\Vg^fgdX�

Da_T\`>q�KXW\V\`X�

CgddX`f>OaUaf>Fg`Vf\a`�

OXcg\dXW>PXdh\VXe�N

ew S

ervices�

QXV[`\VT^>IeegXe�

JST, CRDS, 2009, Modified by Kosuge, June, 2012�

OaUaf\Ve>OXeXTdV[>�

Outline�

•  Overview of Robotics (Center for Research and Development Strategies, Japan Science and Technology Agency)

•  Multiple Robots Coordination •  Robot Helpers •  Dance Partner Robot •  PaDY •  Conclusions

Cooperation of Humans for Handling an Object

Coordination of both arms Coordination of two arms by humans

Single-Master Multi-Slaves System (1989)

K. Kosuge, J. Ishikawa, K. Furuta, M. Sakai, ”Control of Single-Master Multi-Slave Manipulator Using VIM”, [Proceedings of the 1990 IEEE International Conference on Robotics and Automation (1990) 1172-1177]�

Cooperation of Humans for Handling an Object

Coordination of multiple humans

Multiple Mobile Manipulator Coordination (2001)

Y. Hirata, Y. Kume, Z. D. Wang, K. Kosuge, ”Decentralized Control of Multiple Mobile Manipulators Based on Virtual 3-D Caster Motion for Handling an Object in Cooperation with a Human”, [Proceedings of the 2003 IEEE International Conference on Robotics and Automation (2003) 938-943].�

Y. Kume, Y. Hirata, Z. D. Wang, K. Kosuge, ”Decentralized Control of Multiple Mobile Manipulators Handling a Single Object in Coordination”, [Proceedings of the 2002 IEEE/RSJ International Conference on Intelligent Robots and Systems (2002) 2758-2763].�

Cooperation of Mobile Dual Manipulators (2003)

Y. Hirata, Y. Kume, T. Sawada, Z. D. Wang, K. Kosuge, ”Handling of an Object by Multiple Mobile Manipulators in Coordination based on Caster-like Dynamics”, [Proceedings of the 2004 IEEE International Conference on Robotics and Automation (2004) 807-812].�

Outline�

•  Overview of Robotics (Center for Research and Development Strategies, Japan Science and Technology Agency)

•  Multiple Robots Coordination •  Robot Helpers •  Dance Partner Robot •  PaDY •  Conclusions

Human Power Augmentation

Human Power Augmentation (Kosuge, 1993)

K. Kosuge, Y. Fujisawa, T. Fukuda, ”Control of Robot Directly Maneuvered by Operator”, [Proceedings of the 1993 IEEE/RSJ International Conference on Intelligent Robots and Systems (1993) 49-54].�

Robot Helpers

Human-Robot Cooperation (Kosuge, 1993)

MR Helper (Mobile Robot Helper, 1997~)

Robot Helpers

K. Kosuge, M. Sato, ”Mobile Robot Helper”, [Proceedings of the 2000 IEEE International Conference on Robotics and Automation (2000) 583-588].�

�/��, <)(�, ="�*, �0�, 25�, ”�t�.�+��}yzkMR Helper”uwx'�v 3�6l, [�!$#��4�;(C-) 69 (685) (2003) 84-90].�

Robot Helpers

DR Helpers (Distributed Robot Helpers) Y. Hirata, K. Kosuge, ”Distributed Robot Helpers Handling a Single Object in Cooperation with a Human”, [Proceedings of the 2000 IEEE International Conference on Robotics and Automations (2000) 458-463].�

�)%�, :�8, �/��, &9�, ����, �,7 , ”�9t1�v���}yz|~{-tv 3uwx��'�v�6l, [�!$#��4�;(C-) 68 (668) (2002) 181-188].�

Lessons Learned

Lessons learned from Robot Helpers

•  Some simple tasks, which could not be done by a human/humans, could be done with a robot helper(s).

•  General tasks could not be done easily even with the assistive robot system(s), because the robot does not know how to collaborate with the human. �

Lessons Learned

•  In order to collaborate with the user, the robot has to know –  the task, –  its user’s intention, and – how the user wants to be assisted�

Outline�

•  Overview of Robotics (Center for Research and Development Strategies, Japan Science and Technology Agency)

•  Multiple Robots Coordination •  Robot Helpers •  Dance Partner Robot •  PaDY •  Conclusions

Dance Partner Robot

To develop a mechanism for closer human-robot coordination/interaction

Ballroom Dances

•  Consider a ball room dance performed by a dance couple, a male dancer and a female dancer.

•  A dance consists of a set of steps of the dance.

Ballroom Dances

•  The dance is lead by a male dancer.

Which step is coming next is controlled by the male dancer based on the rule of the dance and their surroundings.

Ballroom Dances

•  The female dancer estimates the following step at each step transition through the interaction with the male dancer based on the knowledge of the dance.

Ballroom Dances

•  The female dancer has to know the dance in order to dance with her partner:

–  The set of the dance steps. –  The dance transition rule.

–  How to be lead by the male dancer or how to estimate the partner’s step.

Dance Partner Robot

"  Consider a female dance robot. "  The female dance robot has to

know how to dance.

MS DanceR (Mobile Smart Dance Robot)

" Omni-directional mobile base " Body Force Sensor (6-axis force/torque sensor) " Body frame " Controller " Batteries (8kg)

Omni-directional mobile base

Height 162cm Weight 43kg Ms DanceR

Body Force Sensor

Height 162cm Weight 43kg

MS DanceR (Mobile Smart Dance Robot)

Ms DanceR

" Omni-directional mobile base " Body Force Sensor (6-axis force/torque sensor) " Body frame " Controller " Batteries (8kg)

Step of Waltz

•  Closed Change Left (CCL) –  Consists of three steps. CCL begins with male’s left foot and ends with male’s left

foot.

Start

o o

p p

Male’s Step Female’s Step

Start

q Stop

q Stop

Left foot Left foot Right foot Right foot

A Set of Steps

•  Closed Change Left (CCL) –  Consists of three steps. CCL begins with

male’s left foot, and ends with male’s left foot. •  Closed Change Right (CCR)

–  Consists of three steps. CCR begins with male’s right foot, and ends with male’s right foot.

•  Natural Turn (NT) –  Consists of six steps including turning to the

right by 270 degrees. NT begins with male’s right foot, and ends with male’s left foot.

•  Reverse Turn (RT) –  Consists of six steps including turning to the

left by 270 degrees. RT begins with male’s left foot, and ends with male’s right foot.

•  Square Turn (ST) –  Consists of three steps including turning to

the right by 90 degrees. ST begins with male’s right foot, and ends with male’s right foot.

Modeling of Step Motion� Modeling of Step Motion

•  We measure the motion of a dancing the waltz by using a motion capture system, and get the basic trajectories of these steps

Trajectories of NT in a plane (a) A trajectory along x-axis (b) A trajectory along y-axis (c) A trajectory about rotation

time for NT time for NT time for NT

disp

lace

men

t

disp

lace

men

t

disp

lace

men

t

Modeling of Step Transition Rule

“The Ballroom Technique”, Imperial Society of Teachers of Dancing�

A>Closed Change Left

A>Closed Impetus

A>Reverse Turn

A>Whisk

A>Hesitation Change A>Outside Change

A>Natural Spin Turn

A>Natural Turn

A>Weave from PP

A>Buck Whisk

A>Progressive Chasse to

A>Closed Change Right

A>Buck Lock

A>Reverse Corte

PCTR

CCR

BL

RC CFPP

BWH

WFPP

NT46 NT13

CI

RT13

WH

HC

OC

RT46

CCL

NST

CCL

CI

RT

WH

HC OC

NST

NT

WFPP

BWH

PCTR BL

CCR

RC

A>Chasse from PP CFPP

Right

Step Estimation�

•  Design of a step estimator based on resultant force/moment applied to the upper body of the robot and detected by the body force sensor.�

Design of Estimator

Primitive Methods: Instantaneous Data at Transitions of Steps

Time

Forc

e/M

omen

t Dance Step “A” Dance Step “B”

Reference Data

effectiveT

Transition

��

Motion Generation

•  Dancers continue to dance by coordinating their paces and rhythms each other.

–  Motion generation of a selected step trajectory based on the modeled step motion.

–  Motion coordination

Two functions are considered for the robot.

Motion Generator

+

+

Reference velocity based on Step Motion and

Corrective velocity generated based on force/moment acting on the robot

Desired velocity

where,

: force/moment applied by a human

Case1: Robot motion without motion adjustment

Case2: Robot motion with motion adjustment

: scaling factor

hx

hx

DsM +

1

α dx

fxsensorF

fx dx

α

sensorF

Control Architecture

Actuators

Step Estimator Motion Generator

Step Transition

Step Motion

Knowledge

Waltz Step Transition

Step Motion

Tango Step Transition

Step Motion

Blues

Sensors

"   CAST (Control Architecture based-on Step Transition)

Robot Hardware

Sensory information Desired motion

Knowledge of ballroom dances

Sensory information

Estimated step

Basic step trajectory

Experiment Aichi Expo (March 24 ~ September 25, 2005)

Design of Estimator

Time

Forc

e/M

omen

t

Dance Step “A” Dance Step “B”

Reference Data

effectiveT

Transition

Primitive Methods: Instantaneous Data at Transitions of Steps

��

Sensory Data Used for Estimation

Time series data include uncertainty such as time-lag and variation because a dancer cannot always apply the same force/moment for each step transition.

Time Fo

rce/

Mom

ent

Dance Step “A” Dance Step “B”

Reference Data

effectiveT

Transition

Sensory Data Used for Estimation

Time

Forc

e/M

omen

t

Dance Step “A” Dance Step “B”

effectiveT

Transition

HMM-based “Step Estimator”

Next Dance Step

Design of New Robot Body

Tatsuya Okonogi, a dress designer

Partner Ballroom Dance Robot Challenges for Dance Partner Robots

•  Stable physical interaction between a human and robot

•  Female Dance Partner Robot –  Human behavior/intention estimation

•  How to read the its partner’s lead

•  Male Dance Partner Robot –  How to convey robot’s intention to its human partner

•  Motion Entrainment –  Based on Human modeling

•  Emotional Behavior

Outline�

•  Overview of Robotics (Center for Research and Development Strategies, Japan Science and Technology Agency)

•  Multiple Robots Coordination •  Robot Helpers •  Dance Partner Robot •  PaDY •  Conclusions

Automobile Assembly Line

•  If a robot could provide the worker with necessary parts and tools when he/she needs them, the worker could concentrate on the assembly tasks.

•  What the worker has to do is to assemble delivered parts using delivered tools to accomplish the work.

•  PaDY is a robot which delivers necessary parts and tools to a worker when he/she needs them.

–  to reduce the worker’s load –  to improve efficiency of the work�–  to prevent mistakes of the work –  etc.

Working Partner Robot “PaDY”�

“in-time Parts/tools Delivery to You” robot

PaDY

Evaluation Experiment�

Measured data�

��LRF1 �LRF2

2010/10/21�Estimated result�

Effect of PaDY�

The worker’s motion necessary for picking parts/tools has been reduced. The worker could finish his tasks earlier than the work schedule.

Outline�

•  Overview of Robotics (Center for Research and Development Strategies, Japan Science and Technology Agency)

•  Multiple Robots Coordination •  Robot Helpers •  Dance Partner Robot •  PaDY •  Conclusions

Conclusions�

•  We introduced our research history of robot helpers and a dance partner robot as a research plat form for human-robot cooperation.

•  PaDY has been proposed as a real world application of the human-robot cooperation.

����

PXdh\VX@Bbb^\VTf\a`?�

ad\X`fXW>OaUaf\Ve�

OaUaf\Ve�

Fag`WTf\a

`e�

R`\f>QXV[`a^aZ\Xe�

OaUaf\Ve>�Fag`WTf\a`e�

Bbb^\VTf\a`e@PXdh\VXe�Da_T\`>o�E^WXd^j>CTdX�

Da_T\`>p�BZd\Vg^fgdX�

Da_T\`>q�KXW\V\`X�

CgddX`f>OaUaf>Fg`Vf\a`�

OXcg\dXW>PXdh\VXe�

New

Services

�

QXV[`\VT^>IeegXe�

JST, CRDS, 2009, Modified by Kosuge, June, 2012�

OaUaf\Ve>OXeXTdV[>�

Thank you very much for your attention !�