A Cloud based VR platform for Sharing Embodied

Experience in HRI

Tetsunari InamuraNational Institute of Informatics, Japan

The Graduate Univ. for Advanced Studies, Japan

http://researchmap.jp/inamura inamura@nii.ac.jp

Keynote talk at IROS2016, 11th Oct 2016

Interested in• Human-Robot Interaction systems• Natural HRI through similar cognitive

mechanism on robot systems

T. Inamura, M. Inaba and H. Inoue: "PEXIS : Probabilistic Experience Representation Based Adaptive Interaction System for Personal Robots," Systems and Computers in Japan, 2004.

T. Iamura et al: "HRP-2W: A Humanoid Platform for Research on Support Behavior in Daily Life Environments," Robotics and Autonomous Systems, 2009.

T. Inamura, M. Inaba and H. Inoue. "Dialogue Control for Task Achievement based on Evaluation of Situational Vagueness and Stochastic Representation of Experiences,“, IROS2004.

T. Inamura, Y. Nakamura, I. Toshima: "Embodied Symbol Emergence based on Mimesis Theory“, IJRR, 2004.

Bottlenecks in Research onHuman-Robot Interaction

• Budget & Time

NAO: > 20000 USDHRP2W: > 100,000 USDmore than 1 year

ASIMO: (rental)15000 USD / day

Bottlenecks in Research onHuman-Robot Interaction

• Test subjects / ExamineeA3$x?

Y5%!

A3$x?

Y5%!

A3$x?

Y5%!

A3$x?

Y5%!

ATR: Robovie Osaka Univ.: CB2

Bottlenecks in Research onHuman-Robot Interaction

• Storage of experience data for learningA3$x?

Y5%!

ExperienceDatabase

Sensor data(vision, distance)

Motor commands

UtteranceWho & WhenEye movementExpressionGesture

Ideal ‘experience’ for future robots

ExperienceDatabase

A3$x?Y5%!

A3$x?Y5%!

A3$x?Y5%!

A3$x?Y5%! A3$x?Y5%!

A3$x?Y5%!

A3$x?Y5%!

Cloud＋

Virtual Reality＋

Human-Robot Interaction

↓

SIGVerse

SIGVerse: the simulator forembodied social interaction

Simulator platform for interactive social robots

PerceptionSimulation

-Simulating sensorpatterns for visualaudio and tactile sensory patterns

- Abstract perception

Multi-agent Communication

-Control of communication range-Direction of eyes-Control of information propagation

Physics /DynamicsSimulation

Embodiment :Actuation and It’s reaction

Bridging between Cyber / Real world

Weak point of previous system

T. Inamura et al: “Simulator platform that enables social interaction simulation–SIGVerse: SocioIntelliGenesis simulator –”, Proc. of SII 2010. Best Paper Finalist

What is the difference between SIGVerse and usual simulator?

Multi-users can login to the VR environment as an avatar

Robot(behavior program

is embedded)

A typical task based on the SIGVerse

Humanoid Version Wheeled humanoid version

• Cooperative cooking task– Evaluates how robots can collaborate with humans

Towards Long-term and large scale HRI• Online development of social intelligence of robots• Modeling phase: Task model of robot behavior• Evaluation phase : testing with multiple users from

all over the world

Graphical model of robot behavior/ Task model

• Sensor recognition• Prediction model• manipulation• Interaction strategy

Human‐robot Interaction

Modification of robot behavior based on 

evaluation / experience through the 

interaction

J.T.C.Tan and T.Inamura:"Robot Learning Framework via Crowd-sourcing of Human-Robot Interaction for Collaborative Strategy Learning," Proc. of the 24th IEEE Int'l Symp. on Robot and Human Interactive Communication, 2015.

Interfaces for the SIGVerse

1. Head mount display for immersive visualization

2. Kinect for gesture communication / motion demonstration

Cheap and simple system is important to establish large-scale HRI by enormous amount of subjects

- Framework of RoboCup@Homesimulation league

- Competition for Human-Robot Interaction application in daily life environment

- Evaluation of robot functions with real/virtual robots

Challenge to the RoboCup@Home

Tasks in RoboCup@Home

Best of RoboCup@Home 2008[http://www.youtube.com/watch?v=-oGeU-58wrQ]

What’s your name?

T.Inamura, J.T.C.Tan, K.Sugiura, T.Nagai and H.Okada: "Development of RoboCup@Home Simulationtowards Long-term Large Scale HRI," LNCS: RoboCup 2013: Robot World Cup XVII, Vol.8371,pp.672-680,2013.

Dilemma of RoboCup@Home• Too high entry-level

for Human-Robot Interaction

Hardware implementation / maintenance

Sensor signal processingRecognition of real env.

NativationGrasping / Planning

HRI

Leve

l of i

ntel

ligen

ce

??

A typical embodied social interaction task for RoboCup@Home

Object identification and picking up

Ideal competition in HRI

• To start the high-level HRI intelligence

• Aim to reuse the intelligence in real robots

HRI

Dialogue management

Learning by demonstration

Embodied experience

Natural language proc.

Emotional behavior

Socialintelligence

RoboCup@Home Sim from 2013

• 6 teams participated to virtual CleanUp task in RoboCup JapanOpen 2016

• Even high school students attended

• Not one-shot, but statistical evaluation by repetition• Fair game with unbiased human behavior• Concentrate on high-level cognitive communication

Ongoing projects• Symbol emergence robotics ・ Word acquisition

T. Taniguchi et at “Symbol Emergence in Robotics: A Survey, Advanced Robotics, 2016.

• Neurorehabilitation platform • World Robot Summit

A. Taniguchi et al: “Spatial Concept Acquisition for a Mobile Robot that Integrates Self-Localization and Unsupervised Word Discovery from Spoken Sentences”, Trans. on Cognitive and Developmental System, 2016.

• 2015.10.1〜2021.3.31 （5.5‐year）• Project members

• Takayuki Nagai：UEC (Project leader)

• Tadahiro Taniguchi：Ritsumeikan Univ. (Concept group)

• Tetsuya Ogata：Waseda Univ. (Concept group)

• Naoto Iwahashi：Okayama Prefecture Univ. (Belief group)

• Komei Sugiura：NICT (Belief group)

• Tetsunari Inamura：NII (Application group)

• Hiroyuki Okada: Tamagawa Univ. (Application group)

Symbol Emergence in Robotics for Future Human‐Machine Collaboration

https://www.jst.go.jp/kisoken/crest/en/project/1111083/15656632.html

Symbol Emergence Robotics①Symbol Emergence

in Robotics

Mechanism of understanding

Concepts formation

Local knowledge

③Cloud Robotics（rospeex）

Global knowledge

④Augmentation of Knowledge

Cyber-physical application

（ SIGVerse ）

②Mutual Belief（ L-Core ）Model of Communication

⑤ImplementationEvaluation in real environmentRoboCup@Home（field of evaluation）

PI: Prof. Takayuki Nagai

T. Taniguchi, T. Nagai, T. Nakamura, N.Iwahashi, T.Ogata, and H. Asoh, Symbol Emergence in Robotics: A Survey, Advanced Robotics, .(2016)DOI:10.1080/01691864.2016.1164622

Neurorehabilitation Platform by Cloud VR

Physical therapistperforms a target motion

A patient follows the target

motion

• Analysis the gap between the target motion and the reaction of patients is supported by the comparison/search function of the motion database.

• The cloud-based motion database records both of target motion by P.T. and reaction by patients.

• The database support analysis of rehabilitation strategy

T. Inamura et al: “Development of VR Platform for Cloud-based Neurorehabilitation and itsapplication to research on sense of agency and ownership”, Advanced Robotics, 2017, (I hope it is almost accepted)

New version (developing)

Rich VR environment by UnityCommon developing environment by ROS

towards World Robot Summit in 2020

Summary: VR platform• To collect experience of cognitive HRI• To distribute the experience for all robots• To evaluate learning results of robot behavior• To influence on human’s cognition• To provide mental simulation for robot systems…..

Information on the Internet

• If you want to use SIGVerse, please access the following URL or email to inamura@nii.ac.jp

• It’s free & open source.

• Web: http://www.sigverse.org/• Facebook: www.facebook.com/SIGVerse• Github: https://github.com/SIGVerse

SearchSIGVerse

Statistics• About 4000 downloads from 21 countries

• A total of 22 teams attend RoboCup@Home(including high school students)

• Collaborating with 14 universities/institutes(11 in Japan, 3 in international)

• 1 TV program, 5 articles in newspaper.

Take home message

VR クラウド(Cloud Computing)＋ (Crowd Sourcing)accelerates robot intelligence

through SIGVerse

How to use it?• Free open source software

1. Download from www.sigverse.org 2. Install server on Linux, client on Win

3. Write program of robot 4. Execute and login to the VR

SIGServer as serverSIGViewer as client

Lastest ver. is v2.2.2Several clicks to install

C++ API is available. ROS coming soon