VR Project Orientation

Ching-Chi LinInstitute of Information Science, Academia Sinica

Department of Computer Science and Information Engineering, National Taiwan University

Agenda

Introduction Research Issues Current Project and Working Items◦ Real-time Room fusion◦ Perspective Changing

What is VR

Virtual Reality(VR) replicates an environment that simulates a physical presence in places in the real world or an imagined world, allowing the user to interact in that world.[1]◦ Artificially create sensory experiences, which

can include sight, touch, hearing, and smell.

[1] https://en.wikipedia.org/wiki/Virtual_reality

Side-by-side Image

Augmented Reality(AR)

Mixed Reality

VR: virtual objects + virtual environment AV: real objects + virtual environment AR: virtual objects in real environment

VR Devices

Head Mount Device(HMD):◦ Oculus Rift◦ Samsung GearVR◦ Sony PlayStation VR◦ HTC Vive◦ Google Cardboard◦ ...

Position Tracker, controller, joystick, …etc.

VR Applications

Games◦ FPS, adventure, music, … etc.

Media◦ 3D movie, 360 ° videos◦ VR movies

Art Architecture …

Research Issues

Improve user experiences◦ Make the VR/AR effect more realistic.◦ Reduce the dizziness ◦ Reduce random wobble of the picture.

Research Issues(Cont.)

Mobile VR/AR◦ Find a good balance between QoE and

battery life of a mobile device.◦ Dynamically scale resolution and refresh rate

to reduce mobile GPU power consumption.◦ Use user head movement information to

decide the appropriate QoE metrics.

Research Issues(Cont.)

Cluster-Assisted VR◦ Move the computations from local device to a

server cluster, such that users without high-end hardware equipments can alsoexperience VR fluently.◦ How to allocate sufficient computing

resources to different VR applications in the cluster?

Research Issues(Cont.)

Develop new VR/AR applications.

Current Project

Live Reality Fusion◦ Combine the real-time images from two

different locations into one.◦ ex: fuse two seminar rooms for oversea joint

meeting; “wall removing” for Interior design.

◦ Cooperate with Dr. Wang’s group.

Two Approaches

One-360’-camera

Multiple-camera

One-360’-Camera Approach

Only one 360’ camera in the remote room.

A server captures and process the stream from camera, then fuse the processed stream into the video stream captured by user HMD.

One-360’-Camera Approach(Cont.)

Remote room

Observation Room

Multiple-Camera Approach

Remote room

Observation Room

Steps

Break the video captured by 360’ camera into frames.◦ Panoramic image

Construct a cuboid model of the remote room.◦ Transform the panoramic frame into cube.◦ Transform the cube into cuboid.

Steps(Cont.)

Compute what an user should see from the current angle.◦ By projection or other methods.

Replace the wall in observation room with the projected image of the remote room.

Short-Term Goal

Fuse R106 and R107 in IIS.◦ The wall between the two rooms can be

removed to achieve real “room fusion”.◦ Take picture/video as ground-truth.

Current Progress

Dr. Wang’s RA, 祖詒, has completed the construction of the cuboid model of the two rooms.

Working Plans Speedup the model construction process.◦ The current time required to process one

image is a few seconds. Less than 1/33 seconds in order to achieve real-

time.◦ Study the Matlab functions 祖詒 used in his

code.

Continue 禎佑‘s work on multi-camerainterpolation.

Finish

Thank you! Any Questions?