Embed Size (px)
Citation preview
EMPATHIC TELE-EXISTENCE: COLLABORATION ACROSS TIME AND SPACE
Mark BillinghurstUniversity of South Australia
January 16th 2018
Boston
Travelled 10,855 miles in 34.5 hours for a 20 minute talk …
Adelaide - 35o C/95o F
Boston - 1o C/35o F
“Communication is not only the essence of being human, but a vital property of life.”
- John A. Piece
A wide variety of communication cues used.
SpeechParalinguisticPara-verbalsProsodicsIntonation
AudioGazeGestureFace ExpressionBody Position
Visual
Object ManipulationWriting/DrawingSpatial RelationshipObject Presence
Environmental
Face to Face Communication
Teleconferencing Today
Limitations with Current Technology
• Lack of spatial cues• Person blends with background
• Poor communication cues• Limited gaze, gesture, non-verbal communication
• Introduction of artificial seams between physical/digital• Separation of task/communication space
Augmented Reality Teleconferencing
• Natural spatial cues• Blend physical and virtual• Rich communication cues
To Make This Science Fiction Fact..• Display
• Unobtrusive AR display• Tracking
• Robust AR tracking• Capture
• Real time volumetric capture of people• Networking
• High bandwidth, wireless networking
Nearly 20 Years of Research ...
1999Live video texture
2003Volumetric Video
2008Live Broadcast
2016Live Volumetric Capture
2016
https://www.youtube.com/watch?v=7d59O6cfaM0
Are We There Yet?• Display
• Vuzix, Lumus• Tracking
• SLAM, ARKit, ARCore• Capture
• Microsoft, Mimesys, DoubleMe• Networking
• Gb/s, 5G
Vuzix Blade
Mimesys
Going Beyond Being There • Better than Face to Face conferencing
• Changing body scale • Moving inside another’s body• Interacting with virtual content• Capture/replay of conferencing• Making multiple copies of yourself• Sharing enhanced communication cues
Mixed Space Collaboration (2017)
• Make 3D copy of real space• AR user in real world, VR user in 3D copy of real space• Share virtual body cues (head, hands, gaze information)
Real World Virtual World
Virtual Communication Cues
• AR/VR displays• Gesture input (Leap Motion)• Room scale tracking
Multi-Scale Collaboration
• Changing the user’s virtual body scale
From Tele-Presence to Tele-Existence• Tele-Presence
• Enabling you to feel that remote people are part of your space
• Tele-Existence• Enabling you to feel like you are in a remote place
Handheld Tele-Existence Examples
Head Worn Camera Views
• Camera + Processing + Display + Connectivity• Ego-Vision Collaboration (But with Fixed View)
Shared Sphere – 360 Video SharingShared
Live 360 Video
Host User Guest User
Theta S360 Camera
Hi-res Camera
Epson BT-200See-through HMD
Oculus RiftHMD
Leap Motion
Empathic Computing
Systems that allow us to share what we are seeing, hearingand feeling with others..
Empathy Glasses
• Combine together eye-tracking, display, face expression• Implicit cues – eye gaze, face expression
++
Pupil Labs Epson BT-200 AffectiveWear
AffectiveWear – Emotion Glasses
• Photo sensors to recognize expression• User calibration• Machine learning• Recognizing 8 face expressions
Empathy Glasses in Use
• Eye gaze pointer and remote pointing• Face expression display
https://www.youtube.com/watch?v=CdgWVDbMwp4
Technology Trends• Advanced displays
• Wide FOV, high resolution
• Real time space capture• 3D scanning, stitching, segmentation
• Natural gesture interaction• Hand tracking, pose recognition
• Robust eye-tracking• Gaze points, focus depth
• Emotion sensing/sharing• Physiological sensing, emotion mapping
Emotion Sensing HMDs are Coming• Looxid
• EEG/Emotion sensing• Eye tracking
• Emteq• EMG sensing
• Neurable• EEG/BCI for VR
• PhysioHMD (MIT Media Lab)• GSR, PPG, Emotion Sensing
Technology Trends
• Advanced displays• Real time space capture• Natural gesture interaction• Robust eye-tracking• Emotion sensing/sharing
Empathic Tele-Existence
Empathic Tele-Existence
• Move from Observer to Participant• Explicit to Implicit communication• Experiential collaboration – doing together
Future Research• Many directions for future research
• User evaluation• Communication models • Novel interactions• Scaling up• Emotion capture/sharing• Space/body capture• Possible applications
Conclusions• AR Reality Conferencing
• New types of seamless remote collaboration• From Tele-Presence to Tele-Existence
• Trend toward Empathic Computing• Sharing what you see, hear and feel
• Many directions for future research• Capture/sharing emotion, interface design, etc..
