MPEG Developments in Multi-View Video Coding and 3D Video

8/4/2019 MPEG Developments in Multi-View Video Coding and 3D Video

http://slidepdf.com/reader/full/mpeg-developments-in-multi-view-video-coding-and-3d-video 1/21

MPEG Developments in Multi-view

Video Coding and 3D Video

Jens-Rainer Ohm

RWTH Aachen University

Lehrstuhl und Institut für Nachrichtentechnik

[email protected]

htt ://www.ient.rwth-aachen.de

RWTH Aachen University Jens-Rainer Ohm Multiview & 3D Video Coding – EBU Workshop April 2009



Outline

1. Introduction – Purpose and Applications2. tereo an u t -v ew eo o ng stan ar zat on

in MPEG and JVT

3. 2D Video plus depth (MPEG-C part 3)

4. 3D Video / Free-view oint Video

5. Conclusions




Stereo and Multi-view Video Coding

"Classic" Stereo requires only two views which are" "

a en as s – .e. e cap ure mus a rea y a e sp ayproperties into account

Simulcast Combination of two views into one Exploitation of inter-view redundancy

s oes no suppor N-view displays (autostereoscopic, holographic) ona unc ona y: ase ne a ap a on

For these purposes, either coding of multiple views (if-




Multi-view Video Coding (MVC)

Multi-view and 3D video representations requiremu p yn r n z v n a s ow esame scenery from different viewpoints

efficiently -

statistical dependencies than stereo


providing the Breakdancers and Ballroom data sets.



Video-plus-Depth / 3D Video

Support for N-view displays (various types) or stereo with- and associated depth map(s) is encoded

At minimum: One video, one depth map

Technolo ies re uired: Depth estimation (non-normative) De th encodin normative View synthesis (non-normative or with minimumnormative quality requirements)

Depth Transmission

sender receiver

Search /Storage

N MKx video




Video-plus-Depth / 3D Video

Examples:

ereo wbaseline adaptation,or view shift e. .after head tracking)

expected to be around 20 degrees – also for the upcominggenerations of N-view autosteroscopic displays




Stereo Video

MPEG-2 Multi-view profile is essentially stereo withtemporal L/R interleaving

- MPEG-4 part 2 video (L/R packing, for handhelds)

-Packing Arrangement SEI message (the latter in 14496-10/5e Amd.1 to be finalized b Jul 2009 allow variousmethods of L/R packing

- - -

and-bottom, checkerboard (quincunx)

14496-10/5e Amd.1, to be finalized by July 2009)


progressive and interlaced stereo



Multi-view Video Coding (MVC)

VIEW-1

TV/HDTV

VIEW-1

TV/HDTV-

VIEW-3

-

VIEW-3

Stereo systemChannel

MultiMult i --viewview

videovideo

MultiMult i--viewview

videovideo

Stereo systemChannel

MultiMult i --viewview

videovideo

MultiMult i--viewview

videovideo

-

-

-

-

-

-

-

Multi-view-

-

-

Multi-view

-

3DTV

-

3DTV




MVC Standard – Status and Overview

Standard was approved in July 2008 Specified as an amendment of H.264/MPEG-4 AVC Integrated into 5th Edition of ISO/IEC 14496-10 (Annex H)

Syntax

No changes to lower-level AVC syntax (slice and lower), socompa e an eas y n egra e w ex s ng ar ware

Small backward compatible changes to high-level syntax, e.g., to

specify view dependency, random access points Base layer required and easily extracted from video bitstream

(identified by NAL unit type syntax)

-

Enabled through flexible reference picture management Allow decoded pictures from other views to be inserted and

Core decoding modules not aware of whether reference picture is a

time reference or view reference




MVC Standard – Inter-view Prediction

-

Dependencies flexible for multiview, much simpler for stereo

Limitations: a inter-view rediction onl from same time instance

(b) cannot exceed maximum number of stored reference pictures

TimeTime

Base viewinde endent ofother views andAVC compatible

V i e

V i e




MVC Standard – Coding Efficiency

Sample comparison of simulcast vs inter-view prediction(majority of gains due to inter-view prediction at I-picture locations)

Ballroom38

36

R [ d B ]

34

g e Y - P S N ~

32 A v g e r

Simulcast (IBBP)

Simulcast (Hierarchical B)

30

100 300 500 700 900

MVC Inter-view Prediction


Avgerage Rate [kb/s]



MVC Standard – Profiles and Levels

MVC Profiles

Profiles determine the subset of coding tools u t v e w g – na ze , part o or g na amen ment

Supports same subset of coding tools for inter-view prediction as

Stereo High – draft spec in 14496-10/5e Amd.1, expect to finalizeby July/October 2009

Includes support for interlaced and limits number of views tostereo only

Level limits

Levels im ose constraints on resources/com lexit

MVC repurposes fixed decoder resources of single-view AVCdecoders for decoding stereo/multiview video bitstreams

n a g ven eve , ra eo spa o- empora reso u on wnumber of views (e.g., specify max MBs/sec)

Additional constraints to enable multi le arallel decoder


implementations of MVC



MVC Standard – Summary

MVC standard has recently been finalized Follow up work on conformance and transport specs

un erway Necessary for testing interoperability and for delivery

encoding/decoding processes unchanged

Offers the option to extract a compatible 2Dre resentation from the 3D version




MVC Standard – Limitations/Issues

Acquisition/production with large camera arrays is not

Although more efficient than simulcast, rate of MVC isstill ro ortional to the number of views Varies with scene, camera arrangement, etc.

Rate Increase for Multiview System40

30

t i p l e

Simulcast

Multiview

10

20

R a t e

M u l

0

2 4 8 16 32


Number of Cameras



MVC Standard – Limitations/Issues

MVC is about encoding a discrete set of multiple views

Goal: Highest pixel fidelity

-

compressed representation and technologies allowing to generate alarge number of views from a sparse view set

Requires depth/disparity maps representation/compression andinterpolation/rendering method

Hi her distortion ma be ex ected in terms of ixel fidelit not necessarily visual quality)

First phase is 3D Video with expected synthesis baseline up to ±10

MPEG has already defined MPEG-C part 3 (23002-3) standard in 2006

codecs

Allows one video plus depth from which a second view is generated


Rate not significantly increased compared to monoscopic video



Anticipated 3D Video Format

LeftStereoscopic displays

• Variable stereo baseline

Limited

Right• Adjust depth perception

Data Data

ameraInputs

Format Format

Constrained Rate(based on distribution)

• Wide viewing angle•

Auto-stereoscopic

output views-




Bit Rate vs 3D Rendering Capablities

SimulcastSimulcast

MVCMVC

3DV should be compatible with: • existing standards

• mono and stereo devices

R

a t e

• existing or planned infrastructure

B i t

3DV3DV

2D+Depth2D+Depth2




3D Video Framework

Limited Video Inputse. . 2 or 3 views

DepthEstimation

Video/DepthCodec

ViewSynthesis

Larger #Output Views

1010001010001

+

Binary Representation

& Reconstruction Process


3D Vid Ob i d R k



3D Video – Observations and Remarks

coding video and depth has been achieved

ua y qu c y e er ora es w ep enco ng Fine quantization of depth causes notable artifacts,

Blurring artifacts with sub-sampling, might be reduced

with better decimation/interpolation scheme (simple

Better compression algorithms needed uture a or roposa s p anne

Subjective evaluation necessary PSNR results not indicative of artifacts

New metrics could be considered


3D Vid S



3D Video – Summary

Main Objectives

-number of input views and also variable baseline for

Inclusion of depth: decouple number of transmitted

exp ora on un erway

In the process of establishing suitable reference

Expecting to issue Call for Proposals later this year


O ll S



Overall Summary

MPEG has actively contributed compressiontechnolo for stereo and multi-view video and is

considering to take the next steps towards 3D andfree-view oint video

We are trying to define generic formats that are as far

,display processes (not easy!)

bodies concerned with these matters appears

happened in stereo)


Documents

MPEG Developments in Multi-View Video Coding and 3D Video