Let's be there - Sharing Immersive Live Telepresence Experiences based … · 2020-01-23 · SLAMCast: Large -Scale, Real-Time 3D Reconstruction and Streaming for Immersive Multi

05.12.2019

Let's be there - Sharing Immersive Live Telepresence Experiences based on Efficient Real-time 3D Reconstruction and Streaming

Michael Weinmann University of Bonn, Germany

05.12.2019 1

Being There = Telepresence

Telepresence: „Subjective experience of being in an environment that may differ from the user’s actual local physical surrounding” Ideally multi-modal immersive experience

05.12.2019 2



Beyond standard displays

Images partially taken from presentermedia.com

05.12.2019 3



Beyond standard displays

https://www.syfy.com/syfywire/check-out-this-real-life-star-trek-holodeck-being-used -to-train-soldiers

Example: Holodeck


05.12.2019 4 https://medicalview.org/canadas-first-vr-medical-training-centre/

Applications

https://www.virtualrealityhire.com/wp-content/uploads/2017/02/samsungvrevent.jpg

https://visualise.com/2017/11/education-vr-5-examples-bending-reality-enhance-learning

https://geekologie.com/2014/02/count-me-in-playing-skyrim-in-virtual-re.php

Exploring virtual environments

Design

Education Entertain- ment

05.12.2019 5

Applications

Education

Colla- boration Experiences

https://visualise.com/2017/11/education-vr-5-examples -bending-reality-enhance-learning

http://news.mit.edu/2017/mit-csail-new-system -teleoperating-robots-virtual-reality-1009

Robotics

Exploring (captured) real-

world environments

Therapy

Tourism

Design

https://scooterise.com/modern-way -exploring-ancient-monuments/

https://medium.com/frulix/virtual- reality-emerging-backbone-of- tourism-industry-7b6000a526c0

Telecon- ferencing

https://www.theguardian.com/science/blog/2014/oct/ 16/virtual-reality-phobias-public-speaking-flying

Holoportation [Orts-Escolano et al. 2016]

https://xrsweek.com/2017/08/vrs-focus-vr-ar-design/

https://grapee.jp/en/60459

05.12.2019 6

Progress in Virtual Reality Technology

https://augmentedrealitymarketing.pressbooks.com/chapter/definition-and-history-of-augmented-and-virtual-reality/

05.12.2019 7


https://augmentedrealitymarketing.pressbooks.com/chapter/definition-and-history-of-augmented-and-virtual-reality/

What’s next

05.12.2019 8



05.12.2019 9

Live Telepresence Approaches

Video-based Live Telepresence

Model-based Live Telepresence

VS Situation

Awareness

Resolution

Latency

Handling Network Interruptions


05.12.2019 10

Outline Scalable Live Telepresence Immersive Teleoperation

P. Stotko, S. Krumpen, M. B. Hullin, M. Weinmann, and R. Klein. SLAMCast: Large-Scale, Real-Time 3D Reconstruction and Streaming for Immersive Multi-Client Live Telepresence. TVCG, 2019

P. Stotko, S. Krumpen, M. Weinmann, and R. Klein. Efficient 3D Reconstruction and Streaming for Group-Scale Multi-Client Live Telepresence. ISMAR, 2019

P. Stotko, S. Krumpen, M. Schwarz, C. Lenz, S. Behnke, R. Klein, and M. Weinmann. A VR System for Immersive Teleoperation and Live Exploration with a Mobile Robot. IROS, 2019

05.12.2019 11





05.12.2019 12

Motivation

Typical maintenance/consulting/education scenarios: Require on-site presence of experts/users Time-consuming traveling


Local User Expert

05.12.2019 13

Motivation


Goal: Live telepresence

system to gain efficiency

?


Local User Remote Expert Remote System

05.12.2019 14


Goal: Live telepresence

system to gain efficiency

Scalability of sharing live experiences with many remote users

Local User Remote Experts

Motivation

Remote System


?

05.12.2019 15

Telepresence of Users

Related Work

Efficient acquisition Small-scale only

Holoportation [Orts-Escolano et al. 2016]

05.12.2019 16


Related Work Telepresence of Places


Flexible scanning High bandwidth requirements Single-client only

Holoportation [Orts-Escolano et al. 2016] Incremental Streaming [Mossel and Kröter 2016]

05.12.2019 17


Related Work Telepresence of Places


Flexible scanning High bandwidth requirements Single-client only

Holoportation [Orts-Escolano et al. 2016] Incremental Streaming [Mossel and Kröter 2016]

Scalable Live Telepresence

Beyond Room-scale

05.12.2019 18

Overview

Local User

Acquisition of the scene of interest

Communication with the remote expert

Remote Experts

Independent scene exploration

Communication with the local user

Task

s Ke

y En

able

rs

1 5


05.12.2019 19

Overview

Scene reconstruction Streaming of the

reconstructed model and camera poses

Reconstruction Component Local User



Remote Experts



Efficient real-time reconstruction

Outlier filtering Real-time data

compression

Task

s Ke

y En

able

rs

1 2 5


05.12.2019 20

Overview



Reconstruction Component Server Local User



Remote Experts



Management of the reconstructed model

Streaming of data Handling of requests



compression

Guaranteed concurrent hash operations

Bandwidth-optimized scene representation

Empty voxel block pruning

Task

s Ke

y En

able

rs

1 2 3 5


05.12.2019 21 Images partially taken from presentermedia.com

Overview



Reconstruction Component Server Exploration

Components Local User



Remote Experts



Management of the reconstructed model

Streaming of data Handling of requests

Real-time rendering Data requests Interactions with the

scene



compression

Guaranteed concurrent hash operations

Bandwidth-optimized scene representation

Empty voxel block pruning

Level-of-Detail mesh rendering

Adaptive streaming according to client-side hardware/ requests

Task

s Ke

y En

able

rs

1 2 3 4 5


Overview

Sparse voxel-based scene representation managed via spatial hashing


Components Local User Remote Experts 1 2 3 4 5

Image Processing

Camera Pose Estimation

CPU-GPU Streaming

Data Fusion

Model Visualization

RGB-D

05.12.2019 23

3D Reconstruction Process

Depth Discontinuity Filter (DDF) Filter out potential outliers in depth image

Object boundaries

𝑫𝑜 = 𝑑 ∃𝑖 ∈ 𝑵 𝑑 : 𝑑 − 𝑑𝑖 > 𝑐𝑑 ∨ 𝑵𝑜 𝑑 > 𝑐ℎ ⋅ |𝑵 𝑑 |}

05.12.2019 24


Depth Discontinuity Filter (DDF) Filter out potential outliers in depth image

𝑫𝑜 = 𝑑 ∃𝑖 ∈ 𝑵 𝑑 : 𝑑 − 𝑑𝑖 > 𝑐𝑑 ∨ 𝑵𝑜 𝑑 > 𝑐ℎ ⋅ |𝑵 𝑑 |}

Object boundaries

Unreliable neighboorhood

05.12.2019 25


Voxel Block Allocation Downsampling (VBAD) Compact global truncation band

around surface for noise-free data

05.12.2019 26



around surface for noise-free data Sensor noise + remaining outliers

enlarge global band

05.12.2019 27




enlarge global band in every new frame „maximum function“

05.12.2019 28




enlarge global band in every new frame „maximum function“

Use downsampled depth image during

allocation step More efficient than garbage collection

05.12.2019 29


MC Voxel Block Pruning (MCVBP) Fusion cannot fully remove sensor noise + remaining outliers

3D model Fusion weights

red = low weight blue = high weight

05.12.2019 30


MC Voxel Block Pruning (MCVBP) Fusion cannot fully remove sensor noise + remaining outliers Discard voxels with low confidence (red) from raycasting & triangle generation

𝑉𝑜𝑇𝑇𝑇𝑇 = {(𝐷,𝑊,𝐶)|𝑊 < 𝑐𝑤}


Overview



05.12.2019 32

Central Server Process

Data Management via Hashing Problem: Missing guarantees in current

GPU hash map/set data structures Key uniqueness No concurrent insertion, removal, retrieval

Data loss during transmission due to insertion failures

Solution: First thread-safe GPU hash map/set data structure on thread-level Insertion, removal, retrieval and key

uniqueness Invariant: At any time, entry positions and links to colliding values are preserved!

P. Stotko, stdgpu: Efficient STL-like Data Structures on the GPU, 2019

05.12.2019 33


MC Voxel Block Pruning (MCVBP) Exploit SLAMCast’s bandwidth-efficient

scene representation for streaming [Stotko et al. 2019]

Compress/cut off voxels

𝑉𝑜𝑀𝑀 = {(𝐼,𝐶)|𝐼 = 0 ∨ 𝐼 = 255 ∨𝑊 < 𝑐𝑤}

No surface information

05.12.2019 34


MC Voxel Block Pruning (MCVBP) Exploit SLAMCast’s bandwidth-efficient

scene representation for streaming [Stotko et al. 2019]

Compress/cut off voxels

Intelligently prune empty voxel blocks from streaming

𝑉𝑜𝑀𝑀 = {(𝐼,𝐶)|𝐼 = 0 ∨ 𝐼 = 255 ∨𝑊 < 𝑐𝑤}

No surface information

Low confidence

∨

05.12.2019 35

Local mesh + levels of detail generated from MC voxel data

Local detail enhancement via projective texture mapping


Overview



05.12.2019 36

Evaluation

Bandwidth and Latency Analysis Latency between Reconstruction Client (RC) and Server (S) depends on scene

Does not affect immersion at exploration client

Latency between Server (S) and Exploration Client (EC) adjustable via package size 512 blocks/package at 100Hz best tradeoff

05.12.2019 37

Evaluation of Telepresence System

0 5 10 15 20 25 30

B

B+DDF

B+VBAD

B+MCVBP

Ours

Max ECs without introducing further latency

lounge copyroom heating_room pool lr kt2

SLAMCast v1 +

SLAMCast v1 +

SLAMCast v1 +

SLAMCast v1

System Scalability SLAMCast v1 [TVCG 2019] SLAMCast v1 +

{DDF,VBAD,MCVBP} Ours = SLAMCast v1 + DDF +

VBAD + MCVBP

Results Increased client scalability from

around 4 to more than 24 Total streaming delay < 1 second

05.12.2019 38

Evaluation of Reconstruction Process

Visual Quality Significantly reduced

amount of artifacts Improved model

compactness and performance

05.12.2019 39

The story so far …

First practical telepresence system for real-time capture and many-user exploration of static 3D scenes beyond room-scale (also with mobile devices) Efficient data representation for streaming (> 90% bandwidth savings) First thread-safe GPU hash map/set on thread-level for data management Limitations and future extensions:

Limited camera tracking accuracy integration of Bundle Adjustment Only single reconstruction client extension to multi-client acquisition [Golodetz et al. 2018] RGB textured models models with more detailed reflectance characteristics


05.12.2019 40

The story so far …

First practical telepresence system for real-time capture and many-user exploration of static 3D scenes beyond room-scale (also with mobile devices) Limitations and future extensions:

RGB textured models models with more detailed reflectance characteristics


L. Bode, S. Merzbach, P. Stotko, M. Weinmann and R. Klein. Real-time Multi-material Reflectance Reconstruction for Large-scale Scenes under Uncontrolled Illumination from RGB-D Image Sequences, Proc. of International Conference on 3D Vision (3DV), 2019

P. Stotko, M. Weinmann and R. Klein. Albedo estimation for real-time 3D reconstruction using RGB-D and IR data, ISPRS Journal of Photogrammetry and Remote Sensing, 2019

05.12.2019 41





05.12.2019 42

Motivation

Applications like disaster management, industrial inspection Require on-site presence of experts/users Unsafe or contaminated places


Expert Local Expert

05.12.2019 43

Motivation

Applications like disaster management, industrial inspection Require on-site presence of experts/users Unsafe or contaminated places

Goal Live teleoperation

system to reduce risk of human damage

?


Teleoperation System

Remote Expert Local Robot

05.12.2019 44

Related Work

Immersive Point Cloud Virtual Environments [Bruder et al. 2014]

Robot scene capture Sparse scene reconstruction

05.12.2019 45

Related Work

Immersive Point Cloud Virtual Environments [Bruder et al. 2014]

Robot scene capture Sparse scene reconstruction

Telepresence Robot [Kurup and Liu 2016]

Lightweight occupancy grid maps via SLAM High-latency video-based VR

05.12.2019 46

Our Teleoperation System

Self-contained mobile robot Mario [Schwarz et al. 2018]

05.12.2019 47



Live scene acquisition with RGB-D camera

05.12.2019 48




Real-time dense scene reconstruction and streaming

[Stotko et al. 2019]

05.12.2019 49




Real-time dense scene reconstruction and streaming

[Stotko et al. 2019]

Immersive VR teleoperation experience of operator

05.12.2019 50

Teleoperation in VR

Robot-Operator Interface Robot Client as modular extension to

SLAMCast system Exchange of poses of all robot links: posture

of 6 DoF robot arm + wheel orientations Exchange of additional data possible,

e.g. LiDAR (out of this works scope)

Robot operation via wireless gamepad

interface Control of omnidirectional velocity:

2D translation + rotation around vertical axis

05.12.2019 51

Evaluation of User Experience

User study Elaborate course with challenges of

different difficulty Task: Maneuver robot through course

05.12.2019 52


User study Elaborate course with challenges of

different difficulty Task: Maneuver robot through course Two different modes for robot

navigation

Video mode VR mode

05.12.2019 53


VR vs. Video Terrain assessment Maneuvering around corners

Obstancle avoidance Localization in the scene

Perceived latency Movement speed

++

++

++

++

– ○

05.12.2019 54


VR vs. Video Less collisions/higher navigation

accuracy Slower movement and completion

time due to repositioning Higher situation awareness

05.12.2019 55

Evaluation of System Performance

Scene Transmission: 14 Mbit/s (mean), 25 Mbit/s (max)

05.12.2019 56

Evaluation of System Performance

Scene Transmission: 14 Mbit/s (mean), 25 Mbit/s (max)

Virtual Scene Interaction 3D distance measurement tool Accuracy depends on voxel resolution

(sensor noise) + local tracking accuracy Observed errors of < 1cm

05.12.2019 57

Summary

First practical group-scale telepresence system for real-time capture and many-user exploration of static 3D scenes beyond room-scale (also with mobile devices) Improves performance of standalone volumetric

3D reconstruction techniques Allows immersing more than 24 remote users

into the live-captured scenario

Practical immersive teleoperation system in VR for live exploration of inaccessible places

05.12.2019 58

Summary

First practical group-scale telepresence system for real-time capture and many-user exploration of static 3D scenes beyond room-scale (also with mobile devices) Improves performance of standalone volumetric

3D reconstruction techniques Allows immersing more than 24 remote users

into the live-captured scenario

Practical immersive teleoperation system in VR for live exploration of inaccessible places

Future work: Multi-client/robot capture and collaboration, more general reflectance, etc.

05.12.2019 59

Acknowledgments

Reinhard Klein

Matthias B. Hullin

Stefan Krumpen

Patrick Stotko

Sven Behnke

Max Schwarz

05.12.2019 60

https://www.researchgate.net/project/Scalable-Live-Telepresence

Let's be there - Sharing Immersive Live Telepresence Experiences based on Efficient Real-time 3D Reconstruction and Streaming

Michael Weinmann University of Bonn, Germany

[email protected]

05.12.2019 61

References Patrick Stotko, Stefan Krumpen, Matthias B. Hullin, Michael Weinmann, and Reinhard Klein. SLAMCast: Large-Scale, Real-Time 3D Reconstruction and Streaming for Immersive Multi-Client Live Telepresence. In: IEEE Transactions on Visualization and Computer Graphics (TVCG), 25:5, pp. 2102-2112, 2019

Patrick Stotko, Stefan Krumpen, Michael Weinmann, and Reinhard Klein. Efficient 3D Reconstruction and Streaming for Group-Scale Multi-Client Live Telepresence. In: Proc. of IEEE International Symposium for Mixed and Augmented Reality (ISMAR), 2019

Patrick Stotko, Stefan Krumpen, Reinhard Klein, and Michael Weinmann. Towards Scalable Sharing of Immersive Live Telepresence Experiences Beyond Room-scale based on Efficient Real-time 3D Reconstruction and Streaming. CVPR Workshop on Computer Vision for AR/VR, 2019

Patrick Stotko, Stefan Krumpen, Max Schwarz, Christian Lenz, Sven Behnke, Reinhard Klein, and Michael Weinmann. A VR System for Immersive Teleoperation and Live Exploration with a Mobile Robot. In: Proc. of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2019

Documents

Let's be there - Sharing Immersive Live Telepresence Experiences based … · 2020-01-23 · SLAMCast: Large -Scale, Real-Time 3D Reconstruction and Streaming for Immersive Multi