Interactive Video ToursInteractive Video Tours

MSR Interactive Visual Media GroupMSR Interactive Visual Media Group//msrweb/vision/IBR//msrweb/vision/IBR

Rick Szeliski, Sing Bing Kang, Matt Uyttendaele, Rick Szeliski, Sing Bing Kang, Matt Uyttendaele, Simon Winder, Antonio CriminisiSimon Winder, Antonio Criminisi

Being There:Being There:Video-Based Reality ProjectVideo-Based Reality Project

A new computer-mediated interactive mediumA new computer-mediated interactive medium

Goal:Goal: Combine theCombine theinteractivityinteractivity of 3D games and virtual worlds of 3D games and virtual worlds with the with therealismrealism and richness of film and video and richness of film and video

Being There:Being There:Video-Based Reality ProjectVideo-Based Reality Project

How?How?

Capture real-world video footage with an omni-Capture real-world video footage with an omni-directional camera and add interactive directional camera and add interactive video/graphic elementsvideo/graphic elements

Application: Application: Video-Based WalkthroughsVideo-Based Walkthroughs[IEEE CG&A May/June 2004][IEEE CG&A May/June 2004]

Virtual ToursVirtual Tours

Move camera along a rail (“dolly track”) and Move camera along a rail (“dolly track”) and play back a 360play back a 360 video video

Applications:Applications:– Homes and architectureHomes and architecture– Outdoor locationsOutdoor locations

(tourist destinations)(tourist destinations)

Surround video acquisition system Surround video acquisition system

OmniCam (six-camera head)

OmniCamOmniCam

Built by Point Grey Research (Built by Point Grey Research (LadybugLadybug))

Six camera headSix camera head

Portable hard drives, fiber-optic linkPortable hard drives, fiber-optic link

Resolution per image: 1024 x 768Resolution per image: 1024 x 768

FOV: ~100FOV: ~100oo x ~80 x ~80oo

Acquisition speed: 15 fps uncompressedAcquisition speed: 15 fps uncompressed

Acquisition platformsAcquisition platforms

Robotic cartRobotic cart

WearableWearable

Calibration: Lens distortionCalibration: Lens distortion

Original (distorted) image Output (corrected, perspective) image

Calibration: VignettingCalibration: Vignetting

Before correction After correction

StitchingStitching

(Only 4 of 6 images shown here)

Problem with simple featheringProblem with simple feathering

Multiperspective plane sweepMultiperspective plane sweep

With MPPSWithout MPPS

Feature tracking for calibration and Feature tracking for calibration and ego-motionego-motion

Points tracked in all frames in all 6 cameras

Edges tracked in all frames in all 6 cameras

StabilizationStabilization

Before motion stabilization After motion stabilization

Align frame-

to-frameand

distributeheading

Augmented surround videoAugmented surround video

A movie clip from the original surround data.

The augmented movie clip.Surround augmented with video

DemoDemoA Virtual Home TourA Virtual Home Tour

Head-mounted outdoor scene acquisitionHead-mounted outdoor scene acquisition Map controlMap control Localized audio for a richer experienceLocalized audio for a richer experience Complex path navigationComplex path navigation

(greater freedom of motion)(greater freedom of motion)

A second scenario: A second scenario: Bellevue Botanical GardensBellevue Botanical Gardens

Data acquisitionData acquisition

Walk through the garden with the Ladybug Walk through the garden with the Ladybug camera mounted on helmetcamera mounted on helmet

Acquisition done in about 15 minutesAcquisition done in about 15 minutes 44 Gb worth of video and audio data44 Gb worth of video and audio data

Map controlMap control

• The garden map

• Drawing the acquisition path

• Mapping video frames positions on map

• Placing sound sources (background and dynamic)

• Output is a descriptive XML file

Audio processing and streamingAudio processing and streaming

All audio sources are:1. Attenuated (based on

distance from current position),

2. Mixed together,3. Streamed to sound

card(DirectX Audio).

Two types of sound: background and dynamic

• Some audio sources

• Current ego-position

Bifurcation handlingBifurcation handling

Hypothesize, align, Hypothesize, align, choose best pair as choose best pair as bifurcationbifurcation

Choice of path Choice of path depends on current depends on current heading and heading and bifurcation point bifurcation point orientationorientation

Current heading

Bifurcation handlingBifurcation handling

Hypothesize, align, Hypothesize, align, choose best pair as choose best pair as bifurcationbifurcation

Choice of path Choice of path depends on current depends on current heading and heading and bifurcation point bifurcation point orientationorientation

Current heading

Demo 2Demo 2A walk in Bellevue Botanical A walk in Bellevue Botanical

GardenGarden

High-Dynamic range (HDR) imagingHigh-Dynamic range (HDR) imaging Complex indoor path navigationComplex indoor path navigation

33rdrd scenario: High-End Home scenario: High-End Home(architectural walkthrough)(architectural walkthrough)

Combine multiple exposures to obtain a high dynamic range (HDR) image.Combine multiple exposures to obtain a high dynamic range (HDR) image.

High dynamic range imagingHigh dynamic range imaging

OverexposedOverexposedCombinedCombinedUnderexposedUnderexposed

High dynamic range imagingHigh dynamic range imaging

All frames Underexposed frames Overexposed frames

HDR imaging is challenging for videos.HDR imaging is challenging for videos. Image registration is necessary.Image registration is necessary.

Demo 3Demo 3A High-End HomeA High-End Home

Features and statisticsFeatures and statistics

Rapid scene acquisitionRapid scene acquisition Map controlMap control Localized audio for a richer experienceLocalized audio for a richer experience Complex path navigationComplex path navigation High data rates:High data rates:

– 44 Gb worth of video and audio data in 15 44 Gb worth of video and audio data in 15 minutesminutes

Current/future workCurrent/future work

Scene ModelingScene Modeling– Motion estimation and map generationMotion estimation and map generation– More interactive 3D elementsMore interactive 3D elements

Compression & selective decodingCompression & selective decoding– Tiled video representationTiled video representation

RenderingRendering– Better use of GPUBetter use of GPU

Interactive DVD deploymentInteractive DVD deployment