Evaluation of bandwidth performance for interactive spherical video

COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.

Evaluation of Bandwidth Performance for Interactive Spherical VideoPatrice Rondao Alface, Jean-François Macq, Nico Verzijp

Bell Labs, Alcatel-LucentICME’s WoMAN’11, Barcelona, July 2011


2

AcknowledgmentFP7 FascinatE project


3

AGENDA

1. Introduction

2. Interactive Spherical Video Transmission

3. Transmission Optimization

4. Experimental Results

5. Conclusions


4

IntroductionInteractive Spherical Video

Heymann et al., “Representation, Coding & Interactive Rendering of High-Resolution Panoramic Images and Video using MPEG-4”, in Proc. Panoramic Photogrammetry Workshop (PPW), Berlin, Germany, Feb. 2005.

• Capture devices

6 to 12 HD views

• Panorama/Texture mapping on mesh sphere


5

IntroductionInteractive Spherical Video - Applications

• Immersive Media Cultural heritage…

• Entertainment – live events 360º Media Coverage

Music concerts

Sports events…

• Video-surveillance & Security

• Geographical Information Systems (GIS) Real Estate

Google Street View…

Image source: www.immersivemedia.com


6

Interactive Spherical Video TransmissionState of the Art

• Compression of the panorama as a video and implicit mapping on a sphere at the client’s renderer side

Panorama coding with uniform quality

MPEG-4 Visual (Smolic et al. 2003)

Flash/PaperVision3D

A. Smolic et al.,”Efficient Representation & Coding of Omni-directional Video using MPEG-4”, Proc. WIAMIS 2003, 4th European Workshop on Image Analysis for Multimedia Interactive Services, London, UK, April 9.-11. 2003.


7

Interactive Spherical Video Transmission

• Observations:

Field-of-View (fov) for a given user-selected orientation is limited to 45 - 60 degrees (vertical axis, aspect ratio gives the horizontal fov width)

More than the half of the bandwidth is wasted on parts of the panorama or views that are not viewed.

• Contribution:

Leveraging from the 2D case (Mavlankar et al. 2009), namely a tiling approach,

we propose to adapt the encoding for Interactive Random Access we propose to adapt the encoding for Interactive Random Access Viewing in live video transmission/adaptation scenariosViewing in live video transmission/adaptation scenariosA. Mavlankar, B. Girod, Background extraction and long-term memory motion-compensated

prediction for spatial-random-access-enabled video coding, Picture Coding Symposium 2009, pp.1-4, 6-8 May 2009


8

Interactive Spherical Video TransmissionSpherical Video Random Access

• Tiling on panorama:

the requested ROI does not map as in 2D to an equal number of tiles.

Visibility mask


9

Interactive Spherical Video TransmissionTiling

• Tile compression with adaptive quality

Tile Intra compression using JPEG2000 (quality constant on a tile)

Tiles coded independently at different rates (qualities) depending on their estimated visibility

Fixed bandwidth budget for the panorama

Whole panorama Visible tiles only (RoI) Possible impairment (interaction delay)


10

Transmission OptimizationTile-Based Quality Adaptation

• Study of four tile-based adaptation techniques with respect to transmission delay

• The delay induces an uncertainty about the actual requested ROI

• For a given bandwidth budget, bitrate is adapted on tiles based on following approaches:

• Motivation: when delay increases, a probabilistic approach should behave better

1. Constant rate2. Direct ROI encoding3. ROI with uniform quality background encoding4. ROI with probability-based adaptive quality background


11

Transmission OptimizationProposed probability-based approach

• Problem of optimizing the quality of the tiles in function of the user requests is modeled as an integer linear program

Given an offered bandwidth BW, the objective is to maximize, for each Given an offered bandwidth BW, the objective is to maximize, for each frame, the total utility of the selected tiles, hence minimizing their frame, the total utility of the selected tiles, hence minimizing their global MSE.global MSE.

Tile marginal utility

Tile at position i, quality qTile bitrate

Tile MSE

Tile marginal cost

Tile inserted at quality ≥ q

Probability of tile visibility

,


12

Transmission OptimizationProposed probability-based approach

• Relaxing the problem by considering the MSE of each tile as a convex function of its cost-bitrate ratio, and dropping the integrality constraint, this fractional knapsack can be solved by a greedy heuristic:


13

Transmission OptimizationProposed probabilistic Model of the Tile Visibility

Optimization of the tile selection under 2 classes of test scenarios

1. Ideal case:

• No delay

• Probability of visibility is known

• Knapsack optimization can directly use the visibility information

2. User interaction delay case


14

Results (case 1: no delay)

Tiling: client viewport quality vs. bandwidthRate-distortion for client viewport

30

35

40

45

50

55

0 200 400 600 800 1000 1200 1400

Client - Bandwidth budget (kB/frame)

Client viewport - PSNR (dB)

1x12x24x48x816x1632x32


15

Transmission OptimizationProbabilistic Model of the Tile Visibility

Optimization of the tile selection under 2 classes of test scenarios

1. Ideal case

2. User interaction delay case

• Delay assumed to be constant in the system

• If delay increases, the uncertainty around requested viewport position also increases

• The uncertainty around the visibility mask for the panorama tiles is characterized by a Gaussian with a standard deviation set proportional to the product of the navigation speed and the delay: σ v . d


16

Low delay: 40ms

Quality is higher close to the estimated viewport

High delay: 1280ms

Low probability for any viewport position; quality tends to be constant on the whole panorama

15dBbetterthanconstantbitrate

7dBbetterthanconstantbitrate

Results (case 2: user interaction delay)

75kB for 4Kx2K panorama


17


PSNR vs. Delay for a fixed bandwidth 75KBs


18


Discrepancy at the client side - delay of 1280ms

Discrepancy for a delay of 1280 ms

15,00

20,00

25,00

30,00

35,00

40,00

45,00

50,00

75000 150000 300000 600000 1200000

bandwidth (Bps)

PS

NR

DirectRoI

RoIandconstantqualitybackground

RoIwithadaptivequalitybackground


19

Conclusions

• Analysis of interactive spherical video delivery techniques.

• Received quality optimization under bw constraints and interaction delay.

• Proposed probability-based tile quality optimization method behaves better than state-of-the-art.

• Future work: enabling this adaptation for different live delivery architectures.


20

http://belllabs.be/internships


21

Transmission OptimizationExperimental Setting

• Point Grey Ladybug3 (6 HD cameras)

• Panorama: 4096x2048

• Viewport: 1280x960

• FOV: 45 degrees vertically

• Pan-Tilt interactions only

• Static camera setting

• Live Constant bitrate transmission with smooth user interactivity

• Live coding: JPEG2000 intra-picture coding-only for low delay

• Panorama partitioned into JPEG2000 independently coded tiles

• Analysis in terms of Interaction speed v and transmission delay d


22

Interactive Spherical Video TransmissionState of the Art

• Dodecahedron-based spherical patches optimized sampling for view coding using MPEG-2 (multi-texture approach)

Chi-Wing Fu et al., "The Rhombic Dodecahedron Map: An Efficient Scheme for Encoding Panoramic Video," Multimedia, IEEE Transactions on , vol.11, no.4, pp.634-644, June 2009


23

Interactive Spherical Video Transmission2D Random Access

• Example: Random Access for H.264

Interactive Regions of Interest (ROIs) are usually set at the encoder side by applying a regular grid of tiles on the video frames.

Content-based approaches (e.g. background/foreground) with FMO slices

Content-basedArbitrary


24


Temporal analysis


25


Delay 1280ms

Delay 40ms


26

Technology

Evaluation of bandwidth performance for interactive spherical video