ICME 2016 - High Efficiency Video Coding - Coding Tools and Specification: HEVC V3 and Coming Developments

High Efficiency Video Coding – Coding Tools andSpecification: HEVC V3 and Coming Developments

Mathias Wien

Institut für NachrichtentechnikRWTH Aachen University

ICME2016

Mathias Wien (RWTH) HEVC V3 and Beyond ICME2016 1 / 235

Outline Part I

1 Video Coding Systems

2 Structure of a Video Sequence

3 Specification


Outline Part II

4 Coding Structures

5 Reference Pictures

6 High-Level Syntax


Outline Part III

7 Intra Prediction

8 Inter Prediction

9 Residual Coding

10 Loop Filtering

11 Entropy Coding

12 Profiles in HEVC Edition 1


Outline Part IV

13 Range Extensions

14 Annex F: Common specifications for multi-layer extensions

15 Annex G: Multiview High Efficiency Video Coding

16 Annex H: Scalable High Efficiency Video Coding

17 Annex I: 3D High Efficiency Video Coding


Outline Part V

18 Screen Content Coding

19 Free Viewpoint Television

20 Wide Color Gamut / High Dynamic Range Coding

21 Future Video Coding


Outline Part VI

22 Summary and Outlook

23 Books and Tools

24 Resources

25 Acronyms

26 References


Part I

Introduction


Outline



3 Specification


Video Coding Systems

source pre-processing encoding

transmission

decodingpost-processingdisplay

Generalized overview of the processing chainVarious realizations of the chain

Communication (e. g. video conferencing)Broadcast (e. g. TV, streaming)Storage (DVD, Blu-Ray, . . . )

Transcoding may be part of transmission


Hybrid Coding Scheme: Encoder

Decoder

CTB

input picture+−

TR+Q

TB

iTR+iQ TB

+

Intra

PB

EntropyCod-ing

bitstream

Deblk. Slice

LoopFilter

Slicerec. picture

Inter

PB

Buffer n pics

ME

PB

CTB – Coding Tree BlockME – Motion EstimationPB – Prediction BlockQ – QuantizationTB – Transform BlockTR – Transform


Outline


2 Structure of a Video SequenceRepresentation of Color

3 Specification


Structure of a Video Sequence

Sequence of pictures successively captured or rendered

Progressive and interlaced formats

Picture rate measured in pictures per second, unit Hertz (Hz)Minimum picture rate at 24 Hz for fluent motion [1]

Standard Definition TV at 50/60 Hz interlacedHigh Definition (HD) video at 50/60 Hz progressiveUltra HD (UHD) video up to 120 HzUp to 300 Hz considered


Chroma Sub-Sampling

Human visual system less sensitive to color than to structure and texture⇒ full resolution luma, lower resolution chroma

Chroma sub-sampling types commonly specified by relation betweennumber of luma an chroma samples

YCbCr Y :X1:X2

With Y : number of luma pixels

Sub-sampling format of chroma components specified by X1 and X2

X1: horizontal sub-sampling

X2 = 0: vertical sub-sampling identical to horizontal sub-sampling

X2 = X1: no vertical sub-sampling


Chroma Sub-Sampling

Example:Assumed location of chroma samples relative to luma samples in HEVC

luma

chroma

(a) YCbCr 4:2:0 (b) YCbCr 4:2:2


Outline



3 SpecificationSpecification Scope


Standardization

Driving factor for standardization: InteroperabilityDesire to enable devices and applications from different manufacturersand sources to interoperateDefinition of requirements

Application needsExtensibility options


Specification Scope

source pre-processing encoding

transmission

decodingpost-processingdisplay

Parsing process

Bitstream syntax

Decoding process


Part II

HEVC Coding Structures and High-LevelSyntax


Outline

4 Coding StructuresRandom Access PointsLeading and Trailing PicturesTemporal Sub-Layer AccessBlocks, Units, and SlicesTilesBlock Types


6 High-Level Syntax


Picture Types: Intra Random Access PointsInstantaneous Decoder Refresh (IDR)

Start of new CVS, reset of decoding process, DPB emptied

Only reference to pictures following IDR in coding order

No reference ’across’ IDR picture

Clean Random Access (CRA)

DPB left intact if within coded video sequence

Pictures after CRA in coding and output order without reference topictures before CRA

If used as starting point: Conformingly decodable pictures following CRAin output order

Broken Link Access (BLA)

Renaming of CRA after splicing operation

Indicates removal of pictures containing broken / unavailable references

CVS = Coded Video Sequence, DPB = Decoded Picture Buffer


Picture Types: Leading and Trailing Pictures

Leading Pictures

Precede associated IRAP in output order

Follow associated IRAP in coding order

Random Access Decodable Leading Picture (RADL)

Correctly decodable if decoding starts at corresponding IRAP

No reference to pictures prior to IRAP in coding and output order

Random Access Skipped Leading Picture (RASL)

May contain references to unavailable pictures if decoding starts at IRAP

Shall not be output if associated IRAP is a BLA

Trailing Pictures

Follow the associated IRAP in coding and output order

Follow all leading pictures associated to the IRAP in coding order


Picture Types: Leading and Trailing Pictures – Example

2 70 1 3 4 5 6 8 9output ord.

coding ord. 01 2 34 56 7 9 8

pic. type RADL RADL IDR TRAIL TRAIL RADL RADL CRA TRAIL TRAIL

NUT 7 6 19 0 1 7 6 21 0 1

(a)

2 70 1 3 4 5 6 8 9 10output ord.

coding ord. 01 2 34 56 7 89

pic. type RADL RADL IDR TRAIL TRAIL RASL RASL CRA TRAIL TRAIL

NUT 7 6 19 0 1 9 8 21 0 1

(b)

NUT: NAL unit type, NAL = Network Abstraction Layer

Coding structure for demonstration purpose only


Picture Types: Leading and Trailing Pictures – Splicing

SEQUENCE A

2 0 1 4 3 7 5 6 9 8 12 10 11 14 13output ord.

coding ord. 0 1 2 3 4 5 6 7 8 9

pic. type IDR RADL RADL TRAIL TRAIL CRA RASL RASL TRAIL TRAIL

NUT 19 7 6 1 0 21 9 8 1 0

SEQUENCE B

0 1 2 3 × × 5 4 8 6 7 10 9 11output ord.

coding ord. 0 1 2 3 4 5 6 7 8 9

pic. type IDR TRAIL TRAIL BLA RASL RASL TRAIL TRAIL CRA RASL

NUT 19 1 0 16 9 8 1 0 21 9


Picture Types: Temporal Sub-Layer Access

Switching temporal layers

Temporal nesting: at any picture to higher or lower tidGeneral: switching of temporal layer only at tid = 0

Temporal sub-layer access: additional option

Temporal Sub-Layer Access (TSA)

Switch to any higher tid at TSA picture

No reference to higher tid by TSA picture

Stepwise Temporal Sub-Layer Access (STSA)

Switch to tid of STSA picture possible

Switch to higher temporal layers not possible


Spatial Coding StructuresBlocks and Units

Block: Square or rectangular area in a color component arrayUnit: Collocated blocks of the (three) color components, associatedsyntax elements and prediction data (e. g. motion vectors)

Picture partitioningCoding Tree Blocks / Coding Tree Units (CTBs / CTUs)Each CTU in exactly one slice segmentIndependent slice segment: Full header, independently decodableDependent slice segment: very short header, relies on correspondingindependent slice, inherits CABAC state

Slice typesI-slice: Intra prediction onlyP-slice: Intra prediction and motion compensation with one referencepicture listB-slice: Intra prediction and motion compensation with one or tworeference picture lists

CABAC: Context-based Adaptive Binary Arithmetic Coding


Wavefront Parallel Processing (WPP)

Storage of CABAC states for synchronization

Two CTUs offset per row (availability of top-left CTU)

Entry points coded in the slice segment header

CTU CTU CTU CTU CTU CTU0 1 2 3 4 5

CTU CTU CTU CTUNc Nc+1 Nc+2 Nc+3

CTU CTU2Nc 2Nc+1

decoder1

decoder2

decoder3

slice seg.header

CTU CTU CTU CTU CTU CTU0 1 Nc Nc+1 2Nc 2Nc+1

· · · · · · · · ·

ep0 ep1 ep2

Bitstream


Tiles

Change scanning order of CTBs in picture

Slices in tiles, or tiles in slices

Reset of prediction and entropy coding→ parallel processing(entry points like WPP)

Slice 1

Slice 2

Slice 3

Slice 4

Slice 5

Slice 6


Tiles




Slice 1

Slice 2

Slice 3

Slice 4

Slice 5

Slice 6

Tile 1 Tile 2 Tile 3


Tiles




Slice 1

Slice 2 Slice 3 Slice 4

Slice 5





Coding Tree Blocks and Coding Blocks (CBs)

Quadtree partitioning of CTB into CBs

If picture size not integer multiple of CTB size:Implicit CTB partitioning to meet picture size (must be multiple of 8×8pixels)

0

1 2

3 4

5 67

8 9

10 11

12 13

14 1516

17 18

19 2021

0

1 2

3 4 5 6

7 8 9 10 11

12 13 14 15

16

17 18 19 20

21

(a) (b)


Prediction Blocks (PBs) and Transform Blocks (TBs)

Prediction block partitioning of a 2N×2N CB

INTER

2N×2N 2N×N N×2N N×N

2N×nU 2N×nD nL×2N nR×2N

INTRA

2N×2N

4×4

Transform block partitioning of a CB

Quadtree partitioning of CB→ Residual Quad Tree (RQT)

Transform size 4×4 to 32×32

TB size 4×4 to 64×64

PB boundaries inside TBs allowed


Outline

4 Coding Structures

5 Reference PicturesReference Picture SetReference Picture List

6 High-Level Syntax


Reference Picture Sets (RPS)Reference Picture Set

Set of previously decoded pictures

To be used as reference for inter prediction

Identified by POC value

Picture marking

Use in current or following pictures

Unused for reference (can be removed from DPB)

Construction

Short-term before (B)

Short-term after (A)

Long-term

spec further reading: [4]


Short-Term RPS – Example

POC

RPS

0 8B0 *B1B2B3

0 842 61 3 5 7-1-2-3-4-5-6-7-8

1 9B0 A0*B1

2 10B0 A1A0*B1

3B0 A2A1A0*

4B1 A1A0B0 *

5B2 A0B0B1 *

6B1 A1B0 A0*

7B2 A0B1 B0 *

0 842 61 3 5 7-1-2-3-4-5-6-7-8

RPS of random access configuration from the JCT-VCcommon testing conditions JCTVC-K1100 [8]


Reference Picture List (RPL)

Reference picture lists constructed from available RPSSize of RPL signaled in PPS or slice segment header

One list in P-slices (List0)

Two lists in B-slices (List0, List1)

ConstructionList0

Short-term beforeShort-term afterLong-term

List1Short-term afterShort-term beforeLong-term


Outline

4 Coding Structures


6 High-Level SyntaxNetwork Abstraction LayerParameter SetsPicture Order CountHypothetical Reference DecoderSupplemental Enhancement InformationVideo Usability Information


Network Abstraction Layer (NAL)

Coded Video Sequence (CVS)

Starts with a ’new’ IRAP (associated RASLs to be discarded)

One or more CVSs in a bitstream→ Coded Video Sequence Group (CVSG)

Network Abstraction Layer

Encapsulation of coded video sequence for transport and storageVideo coding layer (VCL) NAL units

All video data, i. e.Slices with CTUs, PUs, TUs

Non-VCL NAL unitsParameter setsSupplemental enhancement information. . .

ScHaWaWe12 [9]


NAL Unit Structure

| | | | | | | | |

. . .

byte 0 1 2 . . . . . . NU−1

1 0 0 0 0 0

NALU header RBSP

SODBRBSP stop bitMSB LSB

RBSP: Raw byte sequence payloadSequence of bytes comprising the coded NAL unit payloadRBSP stop bit (=’1’) plus zero bytes for byte alignment

SODB: String of data bitsConcatenation of bits in the RBSP bytes from MSB to LSBAll bits needed for the decoding processOnly the bits needed for the decoding process

MSB: Most significant bitLSB: Least significant bit


NAL Unit Header (2 bytes)

“0” NAL unit type NUH layer id temporal id

byte 1 byte 2

NAL unit type: characterize content of VCL/ non-VCL NAL unit

NAL unit header (NUH) layer id: equal to 0 for HEVC Version 1, for use inscalable extension (spatial layers, quality layers), multi-view extension(view id)

temporal id ( tid): temporal sub-layer identifier


Access Units

startAUD

(prefix)SEI

CSSsuffixSEI

EOS EOBend

Access Unit (AU)Set of all NAL units associated to exactly one pictureAll NAL units of AU share output time of included pictureNot depicted

Parameter setsOther NAL units types

Decoding UnitsDecoder operation on sub-picture levelOperation of the HRD on sub-picture basis(HRD = Hypothetical Reference Decoder, see below)Enables sub-picture output before complete decoding of full picture

AUD Access unit delimiterSEI Supplemental enhancement informationEOS End of sequenceEOB End of bitstream


Parameter Sets

Hierarchical structure

Separation of information for different hierarchy levels

Highest-priority information

In-band or out-of-band transmission

Available parameter setsVideo parameter set (VPS)Sequence parameter set (SPS)Picture parameter set (PPS)


Parameter Sets

VPS1

VPS2

SPS1

SPS2

SPS3

PPS1

PPS2

PPS3

PPS4

PPS5

slice headers in coded video seq. A

slice headers in coded video seq. B

slice headers in coded video seq. M

slice headers in coded video seq. N

...

...

bitstream I

bitstream II


Coded Video Sequence Group

CVS1

CVS2

CVSG

VPS1

SPS1 PPS2 SSH lid = 1





Video Parameter Set

Introduced for handling of multi-layer bitstreamsGeneral information, activated for all layers

Coded video sequence(s), included layers, available operation points

Highest layer parameter set: sets global constraints

HEVC Version 1: not needed, copy of SPS information


Sequence Parameter Set

Activated for the coded video sequence

Profile, tier, and level

Usage of tools

Video usability information

. . .


Picture Parameter Set

Activated per picture

May change from picture to picture (only one per picture)Tool configuration

CABACQuantizers,Loop filtersTiles. . .


Slice Segment Header

Activates the chain of parameter sets (once per picture)

Slice segment header must refer PPS with identical content (but notnecessarily identical PPS id)

Includes picture order count

Includes all information needed to decode the independent slice segmentand associated dependent slice segments

Entry points for wavefront parallel decoding / tiles


Picture Order Count (POC)

POC-2

POC-1

POC

POC+1

Identifier of picture in the DPB

Indicates output order of the pictures, strictly increasing

POC of IDR pictures always 0: start of new coded video sequence

Used for derivation of picture distance in inter prediction, scaling ofmotion vectors

Constant POC delta not necessary

VPS: indication of POC relation to picture time difference possible


Hypothetical Reference Decoder (HRD)

hypotheticalstream

scheduler(HSS)

codedpicturebuffer(CPB)

decodingprocess(instan-

taneous)

decodedpicturebuffer(DPB)

outputcropping

Encoder: control of buffer states

Introduction of timing

Conformance testing

Parameter signalling in VPS or SPS

Sub-picture operation possibleAU operation→ DU operation (new in HEVC)

Specified in HEVC Annex C spec , further reading: [10]


HRD: Decoded Picture Buffer

hypotheticalstream

scheduler(HSS)

codedpicturebuffer(CPB)

decodingprocess(instan-

taneous)

decodedpicturebuffer(DPB)

outputcropping

Reconstructed picturesReference pictures (short-term / long-term)Pictures to be displayed

Picture output timing: Delay relative to CPB removal timeNote: Prevention of picture output possible

Signalled in the bitstreamIndicated by BLA picture type


Supplemental Enhancement Information (SEI)

SEI message content

Not required for the decoding process

Potentially useful for the decoding process

Attached and integrated into bitstream, sent with coded video sequence

Error recovery, testing for integrity, . . .

Prefix SEI: Cannot occur after the NAL units of the coded picture

Suffix SEI: Cannot occur before the NAL units of the coded pictures

Concept of SEI from H.264 |AVC, payload types aligned for inherited SEImessages

Specified in Annex D spec


Video Usability Information VUI

Not required by the decoding process

Information on the interpretation of the decoded video sequence

Signaling in-band (as part of SPS) or out-of-band

Parameters

Geometric relations→ sample aspect ratio (SAR), overscan indication

Video signal type and color information

Frame / field indication

Default display window (suggested area within the cropping window)

HRD/ Timing information (including clock tick for picture rate)

Restrictions: Tiles, motion vectors, RPL, spatial segmentation, maximumbyte cost (picture / CTU)

Specified in Annex E spec


Part III

HEVC Coding Tools


Outline

7 Intra PredictionIntra Prediction ModesIntra Coding Example

8 Inter Prediction

9 Residual Coding

10 Loop Filtering

11 Entropy Coding



Intra Prediction

Decoder

CTB

input picture+−

TR+Q

TB

iTR+iQ TB

+

Intra

PB

EntropyCod-ing

bitstream

Deblk. Slice

LoopFilter

Slicerec. picture

Inter

PB

Buffer n pics

ME

PB


Intra Prediction Modes

0 : Planar1 : DC

1817

16

15

14

13

12

11109

8

7

6

5

4

32

3433

3231

3029 28 27 26 25 24 23

2221

2019

Planar prediction: mode 0DC intra prediction: mode 1Numbering from diagonal-up to diagonal-downHorizontal: mode 10, vertical: mode 26


Intra Prediction Block size

Intra prediction mode coded per CU

Prediction block size derived from residual quadtree

Boundary samples of neighboring block used for prediction

Efficient representation

Local update of prediction source


Intra Coding Example

DC

DC

DC

DC

DC

P

P DC

P P

DC

P

P

DC

P

P

P

DC

P

P P

DC

original



DC

DC

DC

DC

DC

P

P DC

P P

DC

P

P

DC

P

P

P

DC

P

P P

DC

prediction



DC

DC

DC

DC

DC

P

P DC

P P

DC

P

P

DC

P

P

P

DC

P

P P

DC

residual


Outline

7 Intra Prediction

8 Inter PredictionMotion Compensated PredictionMotion Vector RepresentationInter Coding Examples

9 Residual Coding

10 Loop Filtering

11 Entropy Coding



Inter Prediction

Decoder

CTB

input picture+−

TR+Q

TB

iTR+iQ TB

+

Intra

PB

EntropyCod-ing

bitstream

Deblk. Slice

LoopFilter

Slicerec. picture

Inter

PB

Buffer n pics

ME

PB


Motion Compensated Prediction

POC-2

POC-1

POC

POC+1

Prediction from reference picture listsUni-prediction

P-slices only with List0, B-slices with List0 or List1Minimum PB size 8×4 or 4×8

Bi-prediction, only in B-slicesOne predictor from List0, one predictor from List1Minimum PB size 8×8


Motion Vector Representation

Merge mode

Motion vector (MV) derived from candidate set (spatial and temporalneighborhood)

Merge mode candidate index coded

No motion vector difference encoded

Advanced motion vector prediction

Predictor derived from candidate set (spatial and temporal neighborhood)

Predictor index coded

Motion vector difference encoded

HeOuBrMaBi12 [12]


Merging Motion Vectors

Merge candidate list

List with up to five different motion merge candidates (list length indicatedin the slice segment header)

Unavailable candidates are ignored

List filled to specified length→ Improved loss robustness:Available list length independent of derivation process

Additional combination of reference picture list 0 and 1 candidates forB-slices

Merge mode granularity

PU grid size configured in the PPS

PUs below grid size share merge candidate list

Adjustable computational complexity


Merging Motion Vectors – Spatial Candidates

A0

A1

B0B1

B2

(xP,yP)

PU

(xP,yP) (xP,yP)

(a) Merge candidates (b) 4×8 PUs

Processing order: A1,B1,B0,A0,B2

Candidates A1,B1 only if not right or bottom PU in CU with two partitions,respectively

Maximum four spatial merge candidates

Reduced line buffer storage requirement: shifted neighbor locations forblocks of width 4

spec


Merging Motion Vectors – Temporal Candidates

collocated region inreference picture

C0

C1n−5 n−4 n−3 n−2 n−1 n

current picture

POC

td tb

Location C1 if C0 unavailable, or PU at bottom right CTU boundary(complexity)

Applicable reference picture selected on slice basisB-slices: Reference picture list 0 or 1 indicated by flag

Reference index of candidate always set to 0 (error resilience, complexity)

MV scaling according to POC distance

mvPU =tbtd·mvcol


Predictive Motion Vector Coding

A0

A1

B0B1B2

(xP,yP)

PU

collocated region in

reference picture

C0

C1

Advanced motion vector prediction

Reference index and motion vector difference coded!

Selection of predictor by flag (only two options)Derivation process for each reference picture list

Locations from spatial neighborhood as shown for merge modeCandidate MVs scaled based on POC differenceCandidate A first of A0,A1, candidate B first of B0,B1,B2

Optional: Additional temporal candidate C if A or B not available

spec


Motion Vector Signaling: Skip spec

CU Skip

Indication via flag at beginning of CU syntax

Based on Merge mode

Only merge index signalled

No further syntax elements for CU→ Residual inferred to be zero→ Very cheap coding mode


Inter Coding Examples

Legend: List 0 = diagonal-down hatch, List 1 = diagonal-up hatch, ref.idx=0: gray, ref.idx=1, dark gray



(a) mvL0 (b) mvL1






Outline

7 Intra Prediction

8 Inter Prediction

9 Residual CodingCore TransformsQuantizationCoded Representation of Transform BlocksSpecial Modes

10 Loop Filtering

11 Entropy Coding



Residual Coding

Decoder

CTB

input picture+−

TR+Q

TB

iTR+iQ TB

+

Intra

PB

EntropyCod-ing

bitstream

Deblk. Slice

LoopFilter

Slicerec. picture

Inter

PB

Buffer n pics

ME

PB


Core Transforms

Transform block sizes 4×4, 8×8, 16×16, and 32×32

Integer approximations of the DCT-II transform matrix

Additionally, integer approximation of the DST-VI transform matrix

’Single-norm’ design per transform block size→ simple quantizerimplementation

Not all perfectly orthogonal, leakage below normalization threshold

BuFuBjSzSa13 [13]


Quantizer Step Size. . . . . .

nq

x-4 -3 -2 -1 1 2 3 40

∆q

Quantizer step size ∆q derived from quantization parameter QP

Logarithmic relation of quantizer step sizes

Double step size every 6 QP

∆q( QP + 1) =6√

2 ·∆q( QP)

Definition: ∆q = 1 for QP = 4, thereby

∆q,0 ∈{

2−46 ,2−

36 ,2−

26 ,2−

16 ,1,2

16

}Quantizer step sizes for QP > 5

∆q( QP) = ∆q,0( QPmod6) ·2bQP6 c


Quantizer Step Size

Quantizer rangeQP = 0, . . . ,51

Resulting quantizer step sizes

0.630≤∆q ≤ 228.1

Covering value range of an 8 bit input signal

Higher input bit depth:Extension towards finer quantizationRange extended by 6 QP steps per additional bit


Transform Sub-Blocks (TSBs)

last significant coefficient (xlsc,ylsc)

16

16

4

4transform sub-block

Scan of transform sub-blocks

Last significant coefficient position

Determination of transform sub-blocks with non-zero coefficients

Coded sub-block flag

spec


Transform Sub-Block Scanning

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

0 1 2 3

4 5 6 7

8 9 10 11

12 13 14 15

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

0

1

2

3

0

2

1

3

diag.,vert.

horiz.

(a) 4×4 diagonal (b) 4×4 horizontal (c) 4×4 vertical (d) 8×8 TSB scans

Partitioning of transform block into 4×4 transform sub-blocks (TSBs)

Scan direction selectable for 4×4 and 8×8 blocks, diagonal otherwise

Scan direction in TSB depending on (intra) prediction modeLevel distribution: ’trailing ones’ expected towards higher frequencies

Scan used in inverse directionStart with expected ’1’ values

Level: quantized transform coefficient level nq


Special Modes

Transform skip, applicable to 4×4 TUs

Omit transform step

Inverse quantization operation

Additional bit shift to compensate missing transform

Transform and quantization bypass

No inverse transform, no quantizer scaling

Perfect reconstruction of residual (useful e. g. for graphics content)

PCM mode

Direct encoding of pixel levels, lossless representation option for CU

Configurable bit depth (SPS)

Maximum PCM block size 32×32


Outline

7 Intra Prediction

8 Inter Prediction

9 Residual Coding

10 Loop FilteringDeblocking FilterSample Adaptive Offset

11 Entropy Coding



Loop Filtering

Decoder

CTB

input picture+−

TR+Q

TB

iTR+iQ TB

+

Intra

PB

EntropyCod-ing

bitstream

Deblk. Slice

LoopFilter

Slicerec. picture

Inter

PB

Buffer n pics

ME

PB


Deblocking Filter

Design considerations

Reduction of visible blocking artifacts induced from block-wise processing

Computational complexity

Parallel processing

Deblocking processing

Operation on slice / tile basis

Slice-based control of deblocking filter configuration possiblePicture-based or block-based implementations possible

Line buffer for block-based processing

[14]


Deblocking Filter Neighborhood

vertical edges horizontal edges

q0,A1p0,A1

q0,A2p0,A2

q0,A3p0,A3

q0,A4p0,A4

q0,B1p0,B1 q0,B2p0,B2

q0,C1p0,C1

q0,C2p0,C2

q0,C3p0,C3

q0,A1

p0,A1

q0,A2

p0,A2

q0,A3

p0,A3

q0,A4

p0,A4

q0,B1

p0,B1

q0,B2

p0,B2

q0,B3

p0,B3

q0,B4

p0,B4

q0,C

p0,C

(a) (b)

Neighborhood determination

q0,p0: reference samples in current block and neighboring block

Filtering only on 8×8 sample grid!

Edge sections of 4 sample length


Boundary Strength

Boundary strength parameter bS

Neighboring blocks determined by q0,p0

Possible boundary parameter strength values: 0,1,2

Determination of bS

At least one intra coded block: bS = 2

Transform block boundary and non-zero coefficients: bS = 1Motion information conditions: bS = 1

Different reference pictures or different number of motion vectorsSame number of MVs and |mvx ,p−mvx ,q| ≥ 1 or |mvy ,p−mvy ,q| ≥ 1(full-sample resolution)

Otherwise: bS = 0

spec


Deblocking Filter Operation

Deblocking filter operationOperation on a 4-sample edge basis

Luma: Deblocking if bs ≥ 1Chroma: Deblocking if bs = 2

First vertical filtering, then horizontal filtering

Independent operation on 8×8 block grid→ parallel processing!

spec


Deblocking Filter Example

(a) Original (b) Reconstruction with deblocking



(c) Structure, deblocked samples (d) Reconstruction without deblocking



(e) Normal deblocking (f) Strong deblocking


Sample Adaptive Offset (SAO)

New filter type in ITU-T / MPEG video coding specificationsLocal processing of samples

Depending on local neighborhood (edge offset), orDepending on sample value (band offset)

Operation independent of processed samples→ parallel processing

Local filter parameter adaptation

Four different offset values available (plus SAO off)

Dedicated SAO parameters for each Y, Cb, Cr component

Common SAO mode for chroma components

further reading: [15]spec


SAO: Edge Offset

pcp0 p1 pc

p1

p0

pc

p1

p0

pc

p1

p0

(a) horizontal (b) vertical (c) diagonal-down (d) diagonal-up

Neighborhood

Two samples from 8-connected neighborhood considered

Direction of neighborhood signaled on CTU basis

Relation of sample values determines edge offset index ie


SAO: Edge Offset

p0 pc p1 p0 pc p1 p0 pc p1 p0 pc p1 p0 pc p1

(a) ie = 0 (b) ie = 1 (c) ie = 2

p0 pc p1 p0 pc p1 p0 pc p1

(d) ie = 4 (e) ie = 3

Offset categories

Edge offset index identifies category

Smoothing only, direction of offset predefined

Sign of SAO offsets not signaled


SAO: Band Offset

low band transition high band

transition band position io

0 8 16 24 32 40 48 56 64 72 80 88 96 104 112 120 128 136 144 152 160 168 176 184 192 200 208 216 224 232 240 248

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

255

intensity value (example: bit depth 8bit)

Correction of sample intensity values for four transition bands

Signaling of transition band position

Offset values for transition bands freely configurable


SAO Filter Example

(a) Reconstructed region (b) Samples with band offset active


SAO Filter Example

(c) Reconstructed region (d) Samples with edge offset active


SAO Filter Example

CTBs with SAO edge (lines) and band offset (gray squares)


Outline

7 Intra Prediction

8 Inter Prediction

9 Residual Coding

10 Loop Filtering

11 Entropy CodingFixed Length and Variable Length CodingContext-Based Adaptive Binary Arithmetic Coding



Entropy Coding

Decoder

CTB

input picture+−

TR+Q

TB

iTR+iQ TB

+

Intra

PB

EntropyCod-ing

bitstream

Deblk. Slice

LoopFilter

Slicerec. picture

Inter

PB

Buffer n pics

ME

PB


Entropy Coding

Fixed length and variable length codes (FLC, VLC)

High-level syntax

Parameter sets, slice segment header

SEI messages

Fixed-length codes, Exp-Golomb codes

Arithmetic coding

Slice level, CTUs

Context-based adaptive coding

Bypass coding (complexity, throughput)


Variable Length and Arithmetic Coding

slice seg.header

NALUheader

CTU CTU CTU CTU0 NC−1 NC NS−1

· · · · · ·

FLC FLC,VLC CABAC CABACba

ep0

VCL NAL Unit

FLC, VLC for header information

CABAC for CTUs

Byte alignment in case of multiple tiles, or with wavefront parallelprocessing (not present otherwise)

CABAC = Context-based Adaptive Binary Arithmetic Codingba = byte alignment


Context-Based Adaptive Binary Arithmetic Coding – CABAC

Binary Arithmetic Coder

syntaxele-

ment

BinarizerContextModeler

AdaptiveEngine

BypassEngine

bitstream

binstring

binary value

binvalue coded bits

bin value coded bits

context update

BinarizationContext model selectionBinary arithmetic codingOptimized binarization design, reduced number of non-bypass binscompared to H.264 |AVC

MaScWi03 [16]SzBu12 [17]


Outline

7 Intra Prediction

8 Inter Prediction

9 Residual Coding

10 Loop Filtering

11 Entropy Coding

12 Profiles in HEVC Edition 1ProfilesTiers and Levels


Profiles, Tiers, and Levels in HEVC Edition 1

Main Still Picture

Main

Main 10

Annex A


HEVC Edition 1 ProfilesMain Profile

YCbCr 4:2:0 8 bit video only

CTB block sizes 16×16, 32×32, and 64×64

Either tiles or wavefront parallel processing

Minimum tile size 256×64 pixels

Maximum number of CABAC coded bits in N×N CTU: 53 braw

with braw = (N2 + N/2) ·Bd for YCbCr 4:2:0, bit depth Bd

Main 10 Profile

Additional support of bit depths of 9 and 10 bit

Main Still Picture Profile

Like Main, but only one picture in the bitstream

No timing constraints for the decoder


Tiers and Levels

Profiles

Defined tool (sub-)set of specification

Tools determined from application space

Tiers and levels

Levels: Restrictions on parameters which determine decoding andbuffering capabilities (13 levels defined)

Tiers: Grouping of level limits for different application spaces (currently:Main Tier for consumer, High Tier for professional applications)

Decoder capable of decoding Profile@Tier/Level must be able to decodeall lower levels of same and lower tier


Extensions of HEVC

Range extensions (HEVC V.2/Ed. 2, 10/2014) V.x=ITU-Version, Ed.y=ISO/ICE Edition

Extended color formats (4:2:2, 4:4:4)

Extended bit depth

Scalable extensions(HEVC V.2/Ed. 2, 10/2014)

Simple, multiloop approach, no modifications on tool-level

Supports spatial, SNR scalability

Multi-view (HEVC V.2/Ed. 2, 10/2014)3D extensions (HEVC V.3/Ed. 3, 04/2015)

Stereo / multi-view coding

Multi-view with depth coding

Screen Content Coding (V.4/Ed. 3, 2016)

Dedicated tools for this content type


Part IV

Range and Multi-Layer Extensions


Outline

13 Range ExtensionsExtended Color FormatsRExt ToolsRExt ProfilesRExt Levels






Range Extensions Applications

Professional Requirements

Highest quality, very high bitrates

Increased chroma resolution

Increased precision: higher bit depth

Applications

Professional: contribution, distribution

Medical applications

. . .


4:2:2 Residual Quadtree

Y Cb Cr

YCbCr 4:2:2

Chroma shares vertical luma resolution, half horizontal resolution

Residual quadtree kept for the three components

Chroma includes two stacked N2×

N2 transform blocks with N×N luma

transform block


4:4:4 Residual Quadtree

Y Cb Cr

YCbCr or GBR or YZX 4:4:4

Chroma shares vertical and horizontal luma resolutionTwo option, depending on separate_colour_plane_flag

0: Residual quadtree kept for the three components1: Each color component treated as independent monochrome plane


RExt Tools: Transform Skip Rotation

0 1 2 3

4 5 6 7

8 9 A B

C D E F

=⇒180◦

F E D C

B A 9 8

7 6 5 4

3 2 1 0

Rotation of residual in transform block

Signaled in SPS

Applicable to intra 4×4 transform blocks using transform skip

Only if transform and quantization bypass enabled


RExt Tools: Implicit/Explicit Residual DPCM Coding

Implicit Residual DPCM Coding

Usage signaled in SPS

Activated only for blocks using horizontal and vertical intra prediction

Transform skip or transform and quantization bypass to be enabled

Explicit Residual DPCM Coding

Inter blocks with transform bypass

Usage and direction explicitly signaled

Example: horizontal DPCM

r(x ,y)+ = r(x−1,y); for x = 1 . . .bS−1, y = 0 . . .bS−1


RExt Tools: Inter-Component Residual Prediction

Tool for 4:4:4 content

Prediction of chroma component residual block from luma residual block

RC = R′C + α ·RY

with α ∈ {0,±1,±2,±4,±8}Additional decorrelation of residual signal

Gains on RGB content reported [18]

Activated in PPS


RExt Profiles

Main

Intra

Main4:4:4Intra

Main4:4:4

10 Intra

Main4:4:4

12 Intra

Main4:4:4

16 Intra

High Throughput4:4:4

16 Intra

Monochrome

Monochrome16

Monochrome 12Main

4:2:2 Intra

Main4:2:2

10 Intra

Main10 Intra

Main10

Main12 Intra

Main4:2:2

12 Intra

Main12

MainStill Picture

Main4:4:4

Still Picture

Main4:4:416

Still Picture

Main4:4:4

Main4:4:410

Main4:4:412

4:2:2Main4:2:210

Main4:2:212

Range extensions profiles: general profile idc = 4High throughput profiles: general profile idc = 5


RExt Levels

Level limits specified in Version 1 unsufficient for RExt profilesIntroduction of scaling factors

HbrFactor: High bitrate factorControlled by low bit rate flag spec

Values can be 1, 2, or 12, 24 for High Throughput profiles

CpbVclFactor / CpbNalFactorScaled by HbrFactor to apply for table A.4 spec

FormatCapabilityFactorScaling factor for number of bytes in an Access Unit

MinCrScaleFactorScaling factor for minimum compression ratio


RExt Performance

Comparison of HEVC RExt in HM 16.2 relative to AVC FRExt in JM 18.6

Lossy coding [BD-rate %] Lossless coding, rate saving [%]

Figure from FlMaNaNgRoShSoXu16 [19]


Outline

13 Range Extensions

14 Annex F: Common specifications for multi-layer extensionsCommon Specification StructureParameter SetsBitstream SubsetsSEI messages





Common Specification Structure for Multilayer Video

Scalable and multiview coding share concept of layersScalable layers: different picture resolutions, different reconstruction qualityMultiview layers: different views on the same scene

Specification approach in HEVC extensionsUnify common parts of extensions in a joint specification annexOnly separate tools in separate annexes

Chosen extension structureAnnex F: “Common syntax, semantics and decoding processes formultilayer video coding extensions”Annex G: “Multiview coding”Annex H: “Syntax, semantics and decoding processes for scalableextension”Annex I: “3D High Efficiency Video Coding”


Annex F: Main Features

High level syntax

Introduction of layer identification and layer reference mechanims

Specification of layer structure and layer dependencies

Usability and scalability information

Extended ConceptsIntroduction of auxiliary layers (not for output)

DepthAlpha-plane

Option for non-HEVC baselayer (H.264 |AVC)


Video Parameter Set

VPS1



Specification of available layers in the bitstreamScalable layersMultiple viewsAuxiliary pictures (e. g. depth or alpha planes)

Specification of layer dependenciesPrediction relationsLayer sets

Available operation points, output layer setsAvailable profiles and levels


NAL Unit Header: Layer ID

“0” NAL unit type NUH layer id temporal id

byte 1 byte 2

Each layer has unique layer identifier lidScalability types defined in VPS

Indication of texture or depth layerView order index for multi-view coding voidx

Spatial / quality scalability→ dependency identifier didAuxiliary pictures (not used for coding of primary picture)

Auxiliary picture identifier aidNo prediction between pictures with different value of aid

Meaning of the NUH bits in identifier configurableCombination of different scalability types possible→ splitting flagAdditional identifier for each layer encoded in VPSOutput-flags: control of presentation of layers


VPS Extension: Layer Dependencies

target layer

direct reference layer

indirect reference layer

poc-1

lid = 2

lid = 1

lid = 0

lid = 1

lid = 0

poc

lid = 2

lid = 1

lid = 0

poc+1

lid = 2


Layer Dependencies – Adaptation in Network

Strict lid hierarchy

Layer hierarchy→ enable scalability

Simple MANEs: achieve sub-bistream extraction by cutting at some lid, tidcombination

Advanced MANES: analyze dependencies and make use of advancedknowledge

MANE = Media Aware Network Element


VPS Extension: Layer Sets

Layer set

set of lid values which can be extracted from the bitstream for decoding(e. g. different spatial resolutions)

Output layer set (OLS)

Set of layers which are output if flag indicates OLS to be a target OLS

output_layer_flag: indicates if layer is to be output or notExample: Depth layer not output layer, but associated texture layer isouput layer

alt_output_layer_flag: top non-output layer is output if outputlayer not presentExample: Quality scalability, dropping of highest quality layer


VPS: Layer Video Usability Information

Information on characteristics of layer (optional)

Cross layer picture type alignment

Cross layer IRAP alignment

Cross layer IDR alignment

Information on average bitrate, maximum bitrate

Indication of constant or variable picture rate

Hints for use of tiles, loop filters, WPP, cross layer tile alignment

Indication of restrictions on inter-layer prediction

. . .spec


Sequence Parameter Set Extensions

Profile/Tier/Level information conveyed in VPS for non-baselayer SPS

Reference to layer representation format in VPS, index in SPSMultilayer extension

Constraint: Vertical component of inter-view MVs ≤ 56 luma samples⇒ Reduced memory access (within CTB-row above collocated CTB ininter-view reference picture)

spec


Picture Parameter Set Extension

Specification of spatial relation of layersScaled reference layer offset

Offset of luma pixels collocated to corner luma pixels of reference layerregion

Reference region offsetOffset of reference layer region corner luma pixels to corner luma pixels ofreference layer picture

Resample phase informationSpatial phase shift between reference layer and current layer

spec


PPS Extension: Color mapping table

Support of different color spaces, different layer bitdepths

Look-up table: map reference layer YCbCr values to target layer

Only increasing bitdepth between layers allowed

Figure from BoYeChRa15 [20]


Random Access Pictures and POC

IRAP pictures not necessarily aligned over layers

Different temporal resolution

Dependencies / independencies of layers

POC resetPOC reset indication

0: No reset1: Only MSB reset2: MSB and LSB reset3: More information signaled

RASL, RADL, pictures with tid > 0, discardable pictures:Identical POC reset handling for all pictures in AU requiredClean handling after potential bitstream extraction process

If at least IDR picture in AU: only reset types 1 or 2

If lid = 0 non-IDR: no reset types 1 or 2


Slice Segment Header Extension

Extension at beginning of slice segment headerReserved bits before slice type syntax element (number from PPS)Version 1: number of reserved bits equal to 0 spec

Version 2/3: Number is set to max 2 spec

Decoders shall accept any value

Activation of inter-layer predictionNumber of inter-layer reference picturesIdentifiers of inter-layer pred ref pictures ( lid)

Slice segment header extension spec

POC modification information (if applicable)


Independent Non-Baselayer Rewriting

Capability of bitstream rewriting spec

Indication of independent non-baselayer decodability of layer lid attemporal sub-layer tidProfile-level syntax structureFormal processing steps

Removal of all NAL units with lid 6=target lid and not SPS, PPS, EOBRemoval of SPS, PPS NAL units if not lid =target lid or lid = 0Removal of VPS NAL unitsRemoval of NAL units with tid >target tidRewriting of lid to lid = 0 for all NAL units→ Decodable HEVC V1 bitstream


Annex F SEI messages

New SEI messages

plt Name Summary

160 Layers not present Indicate layers of VPS missing in bitstream [persists from SEI to changeSEI]

161 Inter-layer constrained tile sets Indicate decodability of tiles/tile sets[associated CLVS]162 Bitstream partition nesting Carries SEI messages applicable to defined OLS [persistence depending

on nested SEIs]163 Bitstream partition initial arrival time Initial arrival times for CPB operation [remainder of bitstream partition]164 Sub-bitstream property Bit rate information for a sub-bitstream [CVS containing SEI]165 Alpha channel information Control Alpha channel picture persistence [persistence specified by SEI]166 Overlay information Identify alpha and content of overlay pictures [persistence specified by

SEI]167 Temporal MV prediction constraints Indicate storage needs for MVs [persistence specified by SEI]168 Frame-field information How to display pictures [associated AU]

Annex D SEI messages

Syntax of previous SEI messages unchanged

Semantics partly adapted to meet requirements of application withmulti-layer bitstreams


Outline

13 Range Extensions


15 Annex G: Multiview High Efficiency Video CodingMultiview Coding ConceptMultiview ProfilesMultiview SEI MEssagesMultiview Performance




Multiview Scenario

P1

picture plane C0 C2 cameras


Multiview Scenario

P1

picture plane C0 C2 camerasC1


Multiview Coding: View Arrangement

poc-2poc-1

poc

C1

C2

C0


Multiview Concept: View-Id and View Order Index

vid = 1

vid = 2

vid = 0

voidx = 0

voidx = 1

voidx = 2

C1

C2

C0

Identification of views

Relative arrangement of views for presentation: view ID vid

Ordering of views for coding: view order index voidx


MV-HEVC Layer Terminology

Base View, Independend ViewView order index voidx = 0, lid = 0Conforming to HEVC version 1Independent from other views

Dependent ViewsPredict from reference views


Multiview Main Profile

Multiview Main Profile: general profile idc = 6

Multiview Profile based on Main profile

Scalability ID: only view ID or AUX pictures allowed

Aux picture may carry depth information(not used in decoding process for texture view)

Inter view MV vertical constraint flag active→ vertical MV component restricted

All view pictures identical size; no ref local offset allowed(all views aligned)

Number of ref layers (direct or indirect) ≤ 4

Number of pictures in DPB ≤ 8


Annex G SEI Messages

plt Name Summary

176 3D reference displays information Information on recommended display [persistence specified by SEI]177 Depth representation information Information on depth and disparity ranges for auxiliary pictures178 Multiview scene information Minimum and maximum disparity in AU [associated CVS]179 Multiview acquisition information Information on aquisition environment [associated CVS]180 Multiview view position SEI message View position from left to right [associated CVS]


Multiview Performance

Results of subjective evaluation from MV-HEVC verification testJCT3V-N1001 [21]: Comparison of

MVC: AVC-based multi-view video coding (in which the non-base view iscoded using inter-view prediction)

Simulcast HEVC, in which each view is coded independently

MV-HEVC: HEVC-based multi-view video coding (in which the non-baseview is coded using inter-view prediction)

⇒ Gain of approx. 33 % compared to HEVC simulcast⇒ Gain of approx. 50 % compared to H.264 |AVC MVC


Multiview Performance

Figure from JCT3V-N1001 [21]


Outline

13 Range Extensions



16 Annex H: Scalable High Efficiency Video CodingSHVC: Coding ConceptSHVC ToolsScalable ProfilesSHVC Coding Performance



SHVC Layer Terminology

Base layer (BL)Conforming to HEVC version 1Lowest available layer in bitstreamAlternative option: external base layer (H.264 |AVC)

Enhancement layers (EL)Hierarchical organizationDependent or independent from lower layersLower layers used for prediction in enhancement layer: reference layers


SHVC Scalable Layers

target layer (EL2)

direct reference layer (EL1)

indirect reference layer (BL)

poc-1

lid = 2

lid = 1

lid = 0

lid = 1

lid = 0

poc

lid = 2

lid = 1

lid = 0

poc+1

lid = 2


SHVC SpecificationApproach

Simple high-level syntax approach (“simple solution”)

No changes on CTU level

Inter-layer prediction via reference picture list

Multi-loop coding: Each layer with motion compensation, loop filter, . . .

H.264 |AVC- Scalable Video Coding (SVC) [22]

Single loop design

Motion compensation, loop filter only on target layer

Key picture concept: Limit drift if not all layers decoded

Claim: Performance only 10% below single layer codingIssues

High description complexityPerformance claim only fulfilled byvery well optimized encoders


SHVC Tools: Resampling for Inter-Layer Prediction

Size and offset of reference layer picture to target layer picture definescaling for interpolation

Reference layer sample location derivation (16th-sample precision)

Interpolation operation

16 dedicated interpolation filters

DCTIF design: integrate with motion compensation interpolation filters

Phase alignment specified in VPS

(a) Center alignment (b) Top left alignment


SHVC Tools: Resampling for Inter-Layer Prediction

−2 0 2 4−16

0

16

32

48

64Phase 0

−2 0 2 4−16

0

16

32

48

64Phase 1

−2 0 2 4−16

0

16

32

48

64Phase 2

−2 0 2 4−16

0

16

32

48

64Phase 3

−2 0 2 4−16

0

16

32

48

64Phase 4

−2 0 2 4−16

0

16

32

48

64Phase 5

−2 0 2 4−16

0

16

32

48

64Phase 6

−2 0 2 4−16

0

16

32

48

64Phase 7

−2 0 2 4−16

0

16

32

48

64Phase 8


SHVC Tools: Coding Standard Scalability

Conventional scalable scheme

Base layer HEVC version 1

Enhancement layer with different spatial resolution, different quality

Coding standard scalability

Base layer coded with different standard (H.264 |AVC)

Enhancement layer using HEVC toolsApproach

Only use access to reconstructed samples (standard agnostic)Possible alternative: Specify means to access information regardingpartitioning, prediction modes, motion vectors (not further pursued)


SHVC Tools: Coding Standard Scalability

poc-2

lid = 1

lid = 0

poc-1

lid = 1

lid = 0

poc

lid = 1

lid = 0

AVC BL

AVC BL

AVC BL

HEVC EL

HEVC EL

HEVC EL


Scalable Main Profiles

Scalable Main Profile and Scalable Main 10 Profile: general profile idc = 7

Based on Main and Main 10 profiles, respectively.

Chroma format 4:2:0 for all layers

Maximum of 16 different representation formats in VPSs

Aux pictures allowed (not used in decoding process)

Layer ID lid → Dependency ID did



SHVC Coding Performance

Results of subjective evaluation from SHVC verification testJCTVC-W1004 [23]: Comparison of SHVC vs. HEVC simulcast

Figure from JCTVC-W1004 [23]


Outline

13 Range Extensions




17 Annex I: 3D High Efficiency Video Coding3D Video Coding Concept3D Coding Tools3D SEI Messages3D Profiles3D-HEVC Coding Performance


3D Video Coding

Multiple views with depth information

Video→ “texture view”

Depth information: represent object distance from camera

Coding of texture and depth

Ability to render additional virtual viewsExploit depth information for improved compression efficiency

Auxiliary picture carrying depthDecoding process changed for lid > 0 (not compatible to HEVC Version 1)Dedicated depth coding toolsTools exploiting the relation between texture and depth


3D Coding Scenario with Depth

0s

z

s∆

P1

P2

f

u d1 d2

xd

zP1

zmax

zmin

d1

xd=

fzP1

;d2

s∆− xd=

fzP1

intercept theorem

d = d1 + d2 =f · xd

zP1

+f · (s∆− xd)

zP1

⇒ d =f · s∆zP1


3D Coding Scenario with Depth

s∆1 s∆2

0s

z

P1

P2

zP1

zmax

zmin


Depth Coding

World coordinate distance zP1 between camera system and object

Range between znear and zfar

Representation by depth value vd

Reciprocal relation between depth and distanceRepresentation by vd = 0 . . .vmax, e. g. vmax = 255 for 8 bit representationdepth array with same resolution as associated picture→ depth value for each pixel


Depth Representation

0s

z

s∆

P1

f

u d1 d2

xd

zP1

zmax

zmin vd = 255

vd = 0

vd

Depth value vd:

zP1 =1

vd255 ·

(1

znear− 1

zfar

)+ 1

zfar


3D Video Coding Concept

Extension of multiview approach by depth informationEnabling view synthesis at the receiver side

Generation of additional views as needed by the (N-view) displayAdaptation of the depth impression(e. g. ’correct’ stereo on large/small screens)

Figure by Fabian Jäger, RWTH internal report


3D Video Coding Concept

Inter-component / -view dependencies→ improved predictionDisparity compensated predictionInter-view prediction of motionInter-view prediction of residualView synthesis prediction

Figure by Fabian Jäger, RWTH internal report


Coding of Depth Information

Characteristics of depth map differ from texture

Homogeneous areas

Depth edges at (or close to) object boundaries


Coding of Depth Information

Characteristics of depth map differ from texture

Homogeneous areas

Depth edges at (or close to) object boundaries


Coding of Dependent Views

Coding tools

Disparity compensated prediction

Inter-view motion prediction

Advanced residual prediction

Illumination compensation

View synthesis prediction

Depth-based block partitioning


Virtual Depth Estimate

Depth of reference view is availableDetermine ’virtual depth’ for current view

PU grid for depth view8×8 grid for texture viewMaximum depth value to derive disparity estimate

Use disparity estimate for prediction tools

Figure from JCT3V-K1003 [24]


Advanced Residual Prediction (ARP)

Determination of collocated block by estimated depth

Application of weighted residual as additional predictor for residual ofcurrent block

Prediction from temporal reference or reference view

Figure from JCT3V-K1003,JCT3V-C0049 [24, 25]


Illumination Compensation

Callibration of cameras my vary over views

Compensated for changed by linear model



View Synthesis Prediction (VSP)

Use estimated depth of reference view

Warp texture of reference view to current view



Depth-Based Block Partitioning

Two motion partitions

Derive texture segmentation from depth information

Integration based on conventional motion information

Ja13 [26]Mathias Wien (RWTH) HEVC V3 and Beyond ICME2016 164 / 235

Coding of Depth Maps

Modifications to basic coding tools

Depth modelling

Segment-wise DC coding

Depth Loop-Up Table


Depth Modelling:Wedgelets

Explicit coding of non-rectangular partitioning



Depth Modelling: Contour Partitions

Shape of partitions estimated from corresponding luma component



Depth: Segment-Wise DC Coding

DC-like structure often observed in depth componentCoding of Constant Partition Values (CPV)

Single value for conventional HEVC intra prediction modestwo values for segmented blocksPrediction from neighborhoodCoding using depth look-up table


Depth Intra Coding

Horizontal or vertical prediction only

Prediction value from center depth pixel at boundary

No residual coded



Annex I SEI Messages

SEI messages of Annexes D, F, and G applyAdditional SEI message: Alternative depth information

Global View and Depth (GVD)Warp map


3D Main Profile

3D Main Profile: general profile idc = 8

3D Profile based on Main profile

Aux picture to carry depth information

Scalability ID: only view ID or depth layer allowed

Inter view MV vertical constraint flag active→ vertical MV component restricted

All view pictures identical size; no ref local offset allowed(all views aligned)


Number of pictures in DPB ≤ 8


3D-HEVC Coding Performance

Results of subjective evaluation from 3D-HEVC verification testJCT3V-M1001 [27]: Comparison of

MV-HEVC: anchor codec for multiview and depth without block-levelchanges to decoding process

3D-HEVC: enhanced compression of multiview and depth withmodifications to block-level decoding process for dependent texture views

⇒ Gain of approx. 20 % for 3D-HEVC compared to MV-HEVC



Figure from JCT3V-M1001 [27]



Figure from JCT3V-M1001 [27]


Part V

Recent Extensions and Developments


Outline

18 Screen Content CodingApplication ScenarioScreen Content Coding ToolsSCC ProfilesSCC Performance





Screen Content Coding (SCC)

Characteristics of screen content sequences differfrom camera captured video

Applications

Gaming

Remote desktops

. . .


Screen Content Coding Tools

Additional set of tools on top of RExt profile

Intra block copy

Palette mode coding

Adaptive cross-component transformation

Adaptive motion vector resolution


Intra Block Copy

Displaced prediction within the same picture

Displaced prediction of current PB/CB from available parts of picture

Harmonized with inter prediction under study, JCTVC-T1000 [28]

Figure from JCTVC-S1014 [29]


Palette mode

Pixel value prediction instead of ’regular’ prediction and transform

Palette predictor, size configurable in SPS

Horizontal or vertical traversal scan of block

Differential coding of new palette entries



Adaptive cross-component transformation

SCC content in 4:4:4 color format – redundancy among components

Selectable cross-component transform for decorrelation



Adaptive motion vector resolution

Motion in screen content sequences

Integer pixel displacements not unlikely (e. g. when moving windows on adesktop)

Quarter-sample motion vectors not needed in such casesSolution

Analysis of the observed motion in the coded picture at encoderIndication of quarter-sample or integer-sample motion vector precision


SCC Development

Amendment for Screen Content Coding finalized, publication to follow

Tool specification integrated into main body of HEVC specification text⇒ Closer integration, usage with other extensions easier, like RExt

Seven dedicated profiles specified


SCC Profiles

Screen Content Coding Profiles: general profile idc = 9

Screen-Extended Main and Screen-Extended Main 10 profiles

Screen-Extended Main 4:4:4 and Screen-Extended Main 4:4:4 10 profiles

Screen-Extended High Throughput 4:4:4, Screen-Extended HighThroughput 4:4:4 10, and Screen-Extended High Throughput 14 profiles


SCC Performance

Comparison of HM-16.4 vs. HM-16.4+SCM-4.0 for SCC CTC sequences,lossy coding

Figure from JCTVC-U0051 [30]


SCC Performance




SCC Performance




Outline






Free Viewpoint TelevisionExtension to 3D video coding

Restriction to co-planar camera arrangement relaxed

New application scenarios

Super-Multiview

Figure from N16128 [31]


Free Viewpoint TelevisionExtension to 3D video coding

Restriction to co-planar camera arrangement relaxed

New application scenarios

Free Navigation



Status of FTV Development

MPEG Ad-Hoc Group on FTV

June 2015: Call for Evidence on Free-Viewpoint Television:Super-Multiview and Free Navigation, N15348 [32]

February 2016: Results of the Call for Evidence on Free-ViewpointTelevision: Super-Multiview and Free Navigation, N16128 [31]

Outcome

1 proposal for Super-Multiview, 3 proposals for Free Navigation

Evaluation of results not yet conclusive


Outline






Wide Color Gamut / High Dynamic Range Coding

Colour spaceStandard Dynamic Range (SDR) video

Contrast approx. 1000 : 0ITU-R BT.709 colour space, [33]

High Dynamic Range (HDR) videoContrast approx. 1000000 : 0ITU-R BT.2020 colour space, [34]



Wide Color Gamut / High Dynamic Range Coding

Dynamic Range



Standardization Activities on HDR/WCG

ISO/IEC JTC 1/SC 29/WG 11 (MPEG)

02/2015: Requirements and Use Cases for HDR and WCG ContentCoding N15083 [36]

02/2015: Call for Evidence for HDR and WCG Video Coding N15084 [35]

06/2015: Test Results of Call for Evidence (CfE) for HDR and WCG VideoCoding N15350 [37]

10/2015: Assignment of HDR/WCG as work item to JCT-VC

JCT-VC

02/2016: Decision: no new profile, no new coding tools

02/2016: Technical report: Conversion and coding practices forHDR/WCG video (ISO/IEC TR 23008-14), N16063 [38]

06/2016: Conversion and Coding Practices for HDR/WCG Y’CbCr 4:2:0Video with PQ Transfer Characteristics, JCTVC-X1017 [39]


HDR/WCG Conversion Practices: Scope

Figure from JCTVC-X1017 [39]


Outline






Future Video Coding Development

Standardization Requirements, e.g. MPEG N16359 [40]

50 % bitrate saving (30 % might suffice on important usecases)

Tentative target timeline: new standard by 2020

Variety of content (captured, computer generated, mixed, 3D, variousprojections (→ 360VR), multispectral, . . . )

Licensing model(s)?


Joint Video Exploration Team (JVET)

Exploration activitiesFuture Video Coding AhG in MPEG

Exploration and requirements discussions, workshops

Coding Efficiency Improvements AhG in VCEGKey Technical Areas software, VCEG-AZ01 [41]

Foundation of JVET of MPEG and VCEG, similar to JCT-VC, Oct. 2015

Publicly available document sitehttp://phenix.int-evry.fr/jvet/

Publicly available software repository https://jvet.hhi.fraunhofer.de/svn/svn_HMJEMSoftware/


http://phenix.int-evry.fr/jvet/

https://jvet.hhi.fraunhofer.de/svn/svn_HMJEMSoftware/


JVET Status

Joint Exploration Model (JEM)JVET-A1001,JVET-B1001,JVET-C1001 [42, 43, 44]

Definition of Common Testing Conditions, JVET-B1010 [45]

Definition of Ad-Hoc Groups, JVET-B1000 [46]

Definition of Exploration Experiments, JVET-B1011 [47]

Collection of new test materialJVET-A1002,JVET-B1002,JVET-C1002 [48, 49, 50]


JEM: New Coding Tools – Block Partitioning

QTBT - Quadtree plus binary tree partitioning

Replacement of HEVC CU/PU/TU structure→ Adaptive block size transforms, Wi03 [51]

Maximum CTU size 256×256

Figure from JVET-C1001 [44]


JEM: New Coding Tools – Intra Coding

67 intra prediction modes, 65 directional modesIntra prediction mode coding: 6 MPM candidates4-tap interpolation filters for directional predictorsEnhanced boundary prediction, . . .



JEM: New Coding Tools – Inter Coding

Sub-PU level motion vector prediction→ Advanced temporal motion vector prediction (ATMVP)

Adaptive motion vector resolution (AMVR)

Overlapped block motion compensation (OBMC) at left and top blockboundary possible

Affine motion predition (MVs for 4×4 sub-blocks)

Local illumination compensation (LIC), Pattern-matched motion vectorderivation, bi-directional optical flow (BIO, pixel dense)



JEM: New Coding Tools – Transform Coding

More core transforms: additionalDST-VII, DCT-VIII, DST-I and DCT-V

Mode dependent non-separablesecondary transforms

Signal dependent transform (SDT)


JEM: New Coding Tools – Loop Filters

Re-introduction of Adaptive Loop Filters (ALF)


JEM: New Coding Tools – Entropy Coding

Modified context modeling for transform coefficients

Multi-hypothesis probability estimation with context-dependent updatingspeed

Adaptive initialization for context models


JEM: Performance



Part VI

Summary and Outlook


Outline

22 Summary and OutlookSummaryVideo Coding Development: Outlook

23 Books and Tools

24 Resources

25 Acronyms

26 References


Summary

Coding tools in HEVC version 1

Range extensions

Multilayer extensions- Multiview- Scalability- 3D

Screen Content coding

Free Viewpoint Television

High Dynamic Range and Wide Color Gamut

Future Video Coding - Status of JVET


Video Coding Development: Outlook

Towards Immersive Media360 Virtual Reality

JVET AhG8: 360 video test conditions, JVET-C1000d [53]MPEG FTV AhG

LightfieldsMPEG AhG on lightfield compressionJoint AhG (JPEG and MPEG) for digital representations of light/soundfields for immersive media applications, WG1N72033 [54]

Left: 1st frame of BearAttacks sequence, Nokia. Right:https://s3.amazonaws.com/lytro-corp-assets/blog/camera_array.png


https://s3.amazonaws.com/lytro-corp-assets/blog/camera_array.png

Outline


23 Books and Tools

24 Resources

25 Acronyms

26 References


Further Reading

Books published

Wien, Mathias. High Efficiency Video Coding – Coding Tools andSpecification, Springer Berlin Heidelberg, 2015.

Sze, Vivienne, Budagavi, Madhukar, Sullivan, Gary J. (Eds.). HighEfficiency Video Coding (HEVC) – Algorithms and Architectures, SpringerInternational Publishing, 2014.


YUView Player

YUV player supporting display of statistics informationhttps://github.com/IENT/YUView

Play back of raw YUV files, playlists

Comparison / side-by-side views of two files

Visualization of statistics in xml-format

HEVC bitstream decoding and mode visualization


https://github.com/IENT/YUView

Outline


23 Books and Tools

24 Resources

25 Acronyms

26 References


HEVC Resources (1/2)

JCT-VC Mailing List

http://mailman.rwth-aachen.de/mailman/listinfo/jct-vc

JCT-VC Document Repository

http://phenix.it-sudparis.eu/jct/index.php

http://ftp3.itu.int/av-arch/jctvc-site/

JCT-VC HM Software Repository

SVN:https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware

Trac:https://hevc.hhi.fraunhofer.de/trac/hevc


http://mailman.rwth-aachen.de/mailman/listinfo/jct-vc

http://phenix.it-sudparis.eu/jct/index.php

http://ftp3.itu.int/av-arch/jctvc-site/

https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware

https://hevc.hhi.fraunhofer.de/trac/hevc

HEVC Resources (2/2)

JCT-3V Mailing List

http://mailman.rwth-aachen.de/mailman/listinfo/jct-3v

JCT-3V Document Repository

http://phenix.int-evry.fr/jct-3v/

http://wftp3.itu.int/av-arch/jct3v-site/

JCT-3V HM Software Repository

SVN:https://hevc.hhi.fraunhofer.de/svn/svn_3DVCSoftware/

Trac:https://hevc.hhi.fraunhofer.de/trac/3d-hevc


http://mailman.rwth-aachen.de/mailman/listinfo/jct-3v

http://phenix.int-evry.fr/jct-3v/

http://wftp3.itu.int/av-arch/jct3v-site/

https://hevc.hhi.fraunhofer.de/svn/svn_3DVCSoftware/

https://hevc.hhi.fraunhofer.de/trac/3d-hevc

JVET Resources

JVET Mailing List

http://mailman.rwth-aachen.de/mailman/listinfo/jvet

JVET Document Repository


http://wftp3.itu.int/av-arch/jvet-site/

JVET JEM Software Repository

SVN:https://jvet.hhi.fraunhofer.de/svn/svn_HMJEMSoftware/

Trac:https://hevc.hhi.fraunhofer.de/trac/jem/


http://mailman.rwth-aachen.de/mailman/listinfo/jvet


http://wftp3.itu.int/av-arch/jvet-site/


https://hevc.hhi.fraunhofer.de/trac/jem/

Outline


23 Books and Tools

24 Resources

25 Acronyms

26 References


Acronyms3D-HEVC – 3D high efficiency video coding

AAC – Advanced audio codingAAP – Alternative Approval ProcessAHG – Ad-hoc group

AI – All intra (JCT-VC CTC)AIF∗ – Adaptive interpolation filterALF∗ – Adaptive loop filterAMD – Amendment (for ISO standard)AMP – Asymmetric motion partitioning

AMVP – Advanced motion vector predictionANG∗ – Angular intra predictionASO – Arbitrary slice ordering (H.264 |AVC)

AP – Aggregation packetAU – Access unit

AUD – Access unit delimiterAVC – Advanced video coding

BDSNR – Bjøntegaard Delta PSNRBD-rate – Bjøntegaard Delta rate

BL – Base layerBLA – Broken link accessBO – SAO band offset


Acronyms IIBoG – Break-out group

BP – Bandpass [filter]BPB – Bitstream partition buffer

CABAC – Context adaptive binary arithmetic codingCAVLC – Context adaptive variable length coding

CB – Coding blockCBF∗ – Coded block flagCBP – Coded block pattern (H.264 |AVC)CBR – Constant bitrate

CD – Committee draftCD – Compact diskCE – Core experimentCfE – Call for evidenceCfP – Call for proposalsCIE – Commission internationale de l’éclairageCIF – Common interchange format 352×288

CPB – Coded picture bufferCRA – Clean random access

CRFB∗ – Compressed reference frame bufferCRT – Cathode ray tube

CS – Constraint set (in CfP)Mathias Wien (RWTH) HEVC V3 and Beyond ICME2016 218 / 235

Acronyms IIICSS – Coded slice segment

CT – Collaborative teamCTB – Coding tree blockCTC – Common testing conditionsCTU – Coding tree unitCTX – CABAC context

CU – Coding unitCVS – Coded Video Sequence

CVSG – Coded Video Sequence GroupDAM – Draft amendment

DC – Direct currentDCT – Discrete cosine transformDST – Discrete sine transformDON – Decoding order numberDPB – Decoded picture buffer

DU – Decoding unitDCT – Discrete cosine transform

DCTIF – DCT interpolation filterDIF∗ – DCT interpolation filter, syn. DCTIFDIF∗ – Directional interpolation filterDIS – Draft international standard


Acronyms IVDLP∗ – Decodable leading picture, syn. RADL

DMVD – Decoder side motion vector derivationDPB – Decoded picture buffer

DPCM – Differential Pulse Code ModulationDST – Discrete sine transform

DU – Decoding unitDUT∗ – Directional unified transformDVD – Digital Versatile Disk

EG – Exp-Golomb codeEL – Enhancement layerEO – SAO edge offset

EOB – End of bitstreamEOS – End of sequence

EOTF – Electro-optical transfer functionFCD – Final committee draft

FD – Filler dataFDAM – Final draft amendmentFDIS – Final draft international standardFLC – Fixed length code

FMO – Flexible macroblock ordering (H.264 |AVC)fps – Frames per second


Acronyms VFRExt – Fidelity range extensions

FU – Fragmentation unitGBR – Green red blue, color format (see also RGB)GOP – Group of picturesGRD – Gradual Decoder Refresh (H.264 |AVC)

HBPS – Hypothetical bitstream partition schedulerHRD – Hypothetical reference decoder

HD – High definitionHDR – High dynamic range

HDTV – High definition televisionHEVC – High efficiency video coding

HM – HEVC test modelHP – Highpass [filter]

HRD – Hypothetical reference decoderHSS – Hypothetical stream schedulerHTM – 3D-HEVC test model

HVC∗ – High performance video coding (name of HEVC pre-project in MPEG)HVS – Human visual system

IBDI∗ – Internal bit-depth increaseIDCT – Inverse discrete cosine transform

IDR – Instantaneous decoder refreshMathias Wien (RWTH) HEVC V3 and Beyond ICME2016 221 / 235

Acronyms VIIEC – International Electrotechnical Commission

IEEE – Institute of electrical and electronics engineersIRAP – Intra random access point, see also RAP

IS – International standardISDN – Integrated services digital network

ISO – International Standardization OrganizationITU – International telecommunication unionJCT – Joint collaborative team (of ISO and ITU)

JCT-VC – Joint collaborative team on video codingJCT-3V – Joint collaborative team on 3D video coding extension development

JEM – Joint exploration model (JVET test model)JM – Joint model (AVC test model)

JPEG – Joint photographic experts groupJTC – Joint technical committeeJVT – Joint video team

JVET – Joint video exploration teamKLT – Karhunen-Loève transformKTA – Key Technical Areas (H.264 based exploration software of VCEG)

LCTB∗ – Largest coded tree block, syn. CTBLCTB∗ – Largest coded tree unit, syn. CTU

LCU∗ – Larges coding unit, syn. CTUMathias Wien (RWTH) HEVC V3 and Beyond ICME2016 222 / 235

Acronyms VIILD – Low delay (JCT-VC CTC)LP – Lowpass [filter]

LPS – Least probably symbolLSB – Least Significant Bit

MAC – Multiplexed analog componentsMBAFF – Macroblock adaptive frame/field coding (H.264 |AVC)

MC – Motion compensationMDDT∗ – Mode dependent directional transform

ME – Motion estimationMerge – Merge Mode (MV prediction)

MMCO – Memory management control operation (H.264 |AVC)MP3 – MPEG-2 audio layer III

MPEG – Moving picture experts groupMPM – Most probable modeMPS – Most probably symbol

MRST – Multiple RTP streams over a single media transportMRMT – Multiple RTP streams over multiple media transports

MSB – Most Significant BitMSE – Mean squared error


Acronyms VIIIMV – Motion vector

MVC – Multiview video coding (H.264 |AVC)MVD – Motion vector difference

MV-HEVC – Multiview high efficiency video codingNAL – Network abstraction layer

MANE – Media aware network elementMB – Macroblock (H.264 |AVC)

NALU – NAL unitNB – National body (in ISO)

NGVC∗ – Next generation video coding (name of HEVC pre-project in VCEG)NTSC – National television systems committeeNUH – NAL unit headerNUT – NAL unit typePACI – Payload Content Information (packet)OLS – Output layer set

PAFF – Picture adaptive frame/field coding (H.264 |AVC)PAL – Phase alternating linePB – Prediction block

PCM – Pulse code modulationPDAM – Proposed draft amendment

POC – Picture order countMathias Wien (RWTH) HEVC V3 and Beyond ICME2016 224 / 235

Acronyms IXPPS – Picture parameter set

PSNR – Peak signal to noise ratioPU – Prediction unit

QCIF – Quarter common intermediate format 176×144QHD – Quarter High Definition 960×540

QP – Quantization parameterRA – Random access (JCT-VC CTC)

RADL – Random access decodable leading pictureRAP – Random access point

RASL – Random access skipped leading pictureRBSP – Raw byte sequence payload

RD – Rate-distortionRDO – Rate-distortion optimization

RDOQ – Rate-distortion optimized quantizationRExt – HEVC Range extensionsRGB – Red green blue, color formatRPL – Reference picture listRPS – Reference picture setRQT – Residual quad-treeRTP – Real-time transport protocolSAD – Sum of absolute differences


Acronyms XSAO – Sample adaptive offsetSAR – Sample aspect ratio

SATD – Sum of absolute transformed differencesSC – Sub-committee

SCC – Screen Content CodingSD – Standard definition (TV)

SDH – Sign data hidingSDP – Session description protocol

SECAM – Séquentiel couleur à mémoireSEI – Supplemental enhancement informationSG – Study group

SODB – String of data bitsSHVC – Scalable high efficiency video codingSODB – String of data bits

SOP – Structure of picturesSPS – Sequence parameter set

SRST – Single RTP stream over a single media transportSSD – Sum of squared differencesSSE – Sum of squared error

STSA – Stepwise temporal sub-layer accessSVC – Scalable video coding (H.264 |AVC)


Acronyms XITB – Transform blockTE – Tool experiment

TFD∗ – Tagged for discard [picture], syn. RASLTSA – Temporal sub-layer access

TMuC – Test model under considerationTMVP – Temporal motion vector predictor

TR – Truncated Rice [binarization]TSA – Temporal sub-layer accessTSB – Transform sub-block

TSCI – Temporal scalability control informationTU – Transform unitTV – Television

UHD – Ultra High DefinitionVBR – Variable bitrate

VCEG – Visual coding experts groupVCL – Video coding layerVGA – Video Graphics Array 640×480VLC – Variable length codeVPS – Video parameter setVUI – Video usability information

WCG – Wide colour gamutWD – Working draftWG – Working group

WPP – Wavefront parallel processing


Acronyms XIIXGA – Extended Graphics Array 1024×768XYZ – XYZ color space, also color format

YCbCr – Color format with luma and two chroma componentsYUV – XYZ color format

(Acronyms marked with ∗ are deprecated)


Outline


23 Books and Tools

24 Resources

25 Acronyms

26 References


References I

[1] Charles Poynton. Digital Video and HD: Algorithms and Interfaces. Waltham, MA, USA: Morgan Kaufman Publishers, 2012.

[2] High efficiency video coding. ITU-T, Apr. 2013. URL: http://www.itu.int/rec/T-REC-H.265/en.

[3] Gary J. Sullivan et al. “Overview of the High Efficiency Video Coding (HEVC) Standard”. In: IEEE Transactions on Circuits andSystems for Video Technology 22.12 (Dec. 2012), pp. 1649–1668. DOI: 10.1109/TCSVT.2012.2221191.

[4] Rickard Sjöberg et al. “Overview of HEVC high-level syntax and reference picture management”. In: IEEE Transactions on Circuits andSystems for Video Technology 22.12 (Dec. 2012), pp. 1858–1870. DOI: 10.1109/TCSVT.2012.2223052.

[5] Advanced video coding for generic audiovisual services. ITU-T, Jan. 2012. URL:http://www.itu.int/rec/T-REC-H.264/en.

[6] Stephan Wenger. “H.264/AVC over IP”. In: IEEE Transactions on Circuits and Systems for Video Technology 13.7 (July 2003),pp. 645–656. DOI: 10.1109/TCSVT.2003.814966.

[7] Thomas Wiegand et al. “Overview of the H.264/AVC Video Coding Standard”. In: IEEE Transactions on Circuits and Systems for VideoTechnology 13.7 (July 2003), pp. 560–576.

[8] Frank Bossen. Common test conditions and software reference configurations. Doc. JCTVC-K1100. Shanghai, CN, 11th meeting: JointCollaborative Team on Video Coding (JCT-VC) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, Oct. 2012.

[9] Thomas Schierl et al. “System Layer Integration of HEVC”. In: IEEE Transactions on Circuits and Systems for Video Technology 22.12(Dec. 2012), pp. 1871–1884. DOI: 10.1109/TCSVT.2012.2223054.

[10] Jordi Ribas-Corbera, Philip A. Chou, and Shankar Regunathan. “A Generalized Hypothetical Reference Decoder for H.264/AVC”. In:IEEE Transactions on Circuits and Systems for Video Technology 13.7 (July 2003), pp. 674–687. DOI:10.1109/TCSVT.2003.814965.

[11] Jani Lainema et al. “Intra Coding of the HEVC Standard”. In: IEEE Transactions on Circuits and Systems for Video Technology 22.12(Dec. 2012), pp. 1792–1801. DOI: 10.1109/TCSVT.2012.2221525.

[12] Philipp Helle et al. “Block Merging for Quadtree-based Partitioning in HEVC”. In: IEEE Transactions on Circuits and Systems for VideoTechnology 22.12 (Dec. 2012), pp. 1720–1731. DOI: 10.1109/TCSVT.2012.2223051.


http://www.itu.int/rec/T-REC-H.265/en

http://dx.doi.org/10.1109/TCSVT.2012.2221191


http://www.itu.int/rec/T-REC-H.264/en






References II

[13] Madhukar Budagavi et al. “Core Transform Design in the High Efficiency Video Coding (HEVC) Standard”. In: IEEE Journal of SelectedTopics in Signal Processing tbd (2013), tbd. DOI: 10.1109/JSTSP.2013.2270429.

[14] Andrey Norkin et al. “HEVC Deblocking Filter”. In: IEEE Transactions on Circuits and Systems for Video Technology 22.12 (Dec. 2012),pp. 1746–1754. DOI: 10.1109/TCSVT.2012.2223053.

[15] Chih-Ming Fu et al. “Sample Adaptive Offset in the HEVC Standard”. In: IEEE Transactions on Circuits and Systems for VideoTechnology 22.12 (Dec. 2012), pp. 1755–1764. DOI: 10.1109/TCSVT.2012.2221529.

[16] Detlev Marpe, Heiko Schwarz, and Thomas Wiegand. “Context-Based Adaptive Binary Arithmetic Coding in the H.264/AVC VideoCompression Standard”. In: IEEE Transactions on Circuits and Systems for Video Technology 13.7 (July 2003), pp. 620–637. DOI:10.1109/TCSVT.2003.815173.

[17] Vivienne Sze and Madhukar Budagavi. “High Throughput CABAC Entropy Coding in HEVC”. In: IEEE Transactions on Circuits andSystems for Video Technology 22.12 (Dec. 2012), pp. 1778–1791. DOI: 10.1109/TCSVT.2012.2221526.

[18] Wei Pu et al. RCE1: Descriptions and Results for Experiments 1, 2, 3, and 4. Doc. JCTVC-O0202. Geneva, CH, 15th meeting: JointCollaborative Team on Video Coding (JCT-VC) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, Oct. 2013.

[19] David Flynn et al. “Overview of the Range Extensions for the HEVC Standard: Tools, Profiles, and Performance”. In: IEEE Transactionson Circuits and Systems for Video Technology 26.1 (Jan. 2016), pp. 4–19. DOI: 10.1109/TCSVT.2015.2478707.

[20] Jill Boyce et al. “Overview of SHVC: Scalable Extensions of the High Efficiency Video Coding (HEVC) Standard”. In: IEEE Transactionson Circuits and Systems for Video Technology PP.99 (2015). DOI: 10.1109/TCSVT.2015.2461951.

[21] Vittrorio Baroncini, Karsten Mueller, and Shinya Shimizu. MV-HEVC Verification Test Report. Tech. rep. JCT3V-N1001. San Diego, CA,USA, 14th meeting, Feb. 2016. URL: http://phenix.int-evry.fr/jct-3v/doc_end_user/documents/14_San%20Diego/wg11/JCT3V-N1001-v2.zip.

[22] Heiko Schwarz, Detlev Marpe, and Thomas Wiegand. “Overview of the Scalable Video Coding Extension of the H.264/AVC Standard”.In: IEEE Transactions on Circuits and Systems for Video Technology 17.9 (Sept. 2007), pp. 1103–1120. DOI:10.1109/TCSVT.2007.905532.


http://dx.doi.org/10.1109/JSTSP.2013.2270429







http://phenix.int-evry.fr/jct-3v/doc_end_user/documents/14_San%20Diego/wg11/JCT3V-N1001-v2.zip

http://phenix.int-evry.fr/jct-3v/doc_end_user/documents/14_San%20Diego/wg11/JCT3V-N1001-v2.zip


References III[23] Yan Ye, Vittrorio Baroncini, and Ye-Kui Wang. SHVC verification test report. Doc. JCTVC-W1004. San Diego, CA, USA, 23rd meeting:

Joint Collaborative Team on Video Coding (JCT-VC) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, Feb. 2016. URL: http://phenix.it-sudparis.eu/jct/doc_end_user/documents/23_San%20Diego/wg11/JCTVC-W1004-v1.zip(visited on 07/03/2017).

[24] Ying Chen et al. Test Model 11 of 3D-HEVC and MV-HEC. Tech. rep. JCT3V-K1003. Geneva, CH, 11th meeting, Feb. 2015.

[25] Li Zhang et al. 3D-CE4: Advanced residual prediction for multiview coding. Tech. rep. JCT3V-C0049. Geneva, CH, 3rd meeting, Jan.2013. URL:http://phenix.int-evry.fr/jct-3v/doc_end_user/documents/3_Geneva/wg11/JCT3V-C0049-v2.zip.

[26] Fabian Jäger. “Depth-based Block Partitioning for 3D Video Coding”. In: Proc. of International Picture Coding Symposium PCS ’13.San Jose, USA: IEEE, Piscataway, Dec. 2013.

[27] Vittrorio Baroncini, Karsten Mueller, and Shinya Shimizu. 3D-HEVC Verification Test Report. Tech. rep. JCT3V-M1001. Geneva, CH,13th meeting, Oct. 2015. URL: http://phenix.int-evry.fr/jct-3v/doc_end_user/documents/13_Geneva/wg11/JCT3V-M1001-v2.zip (visited on 07/03/2016).

[28] Gary J. Sullivan and Jens-Rainer Ohm. Meeting report of the 20th meeting of the Joint Collaborative Team on Video Coding (JCT-VC),Geneva, CH, 10–1x Feb. 2015. Doc. JCTVC-T1000. 20th meeting, Geneva, CH: Joint Collaborative Team on Video Coding (JCT-VC)of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, Feb. 2015.

[29] Rajan Joshi et al. Screen content coding test model 3 (SCM 3). Doc. JCTVC-S1014. 19th meeting, Strasbourg, F: Joint CollaborativeTeam on Video Coding (JCT-VC) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, Oct. 2014.

[30] Bin Li, Jizheng Xu, and Gary Sullivan. Comparison of Compression Performance of HEVC Test Model 16.4 and HEVC Screen ContentCoding Extensions Test Model 4 with AVC High 4:4:4 Predictive profile. Doc. JCTVC-U0051. 21st meeting, Warsaw, PL: JointCollaborative Team on Video Coding (JCT-VC) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, June 2015.

[31] Vittorio Baroncini, Masayuki Tanimoto, and Olgierd Stankiewicz. Results of the Call for Evidence on Free-Viewpoint Television:Super-Multiview and Free Navigation. Tech. rep. N16128. San Diego, CA, USA, 114th meeting: ISO/IEC JTC1/SC29/WG11, Feb.2016. URL: http://phenix.it-sudparis.eu/mpeg/doc_end_user/documents/114_San%20Diego/wg11/w16128-v2-w16128.zip.


http://phenix.it-sudparis.eu/jct/doc_end_user/documents/23_San%20Diego/wg11/JCTVC-W1004-v1.zip

http://phenix.it-sudparis.eu/jct/doc_end_user/documents/23_San%20Diego/wg11/JCTVC-W1004-v1.zip

http://phenix.int-evry.fr/jct-3v/doc_end_user/documents/3_Geneva/wg11/JCT3V-C0049-v2.zip

http://phenix.int-evry.fr/jct-3v/doc_end_user/documents/13_Geneva/wg11/JCT3V-M1001-v2.zip

http://phenix.int-evry.fr/jct-3v/doc_end_user/documents/13_Geneva/wg11/JCT3V-M1001-v2.zip

http://phenix.it-sudparis.eu/mpeg/doc_end_user/documents/114_San%20Diego/wg11/w16128-v2-w16128.zip

http://phenix.it-sudparis.eu/mpeg/doc_end_user/documents/114_San%20Diego/wg11/w16128-v2-w16128.zip

References IV

[32] MPEG Requirements. Call for Evidence on Free-Viewpoint Television: Super-Multiview and Free Navigation. Tech. rep. N15348. 112thmeeting, Warsaw, Poland: ISO/IEC JTC1 SC29WG11 MPEG, June 2015. URL:http://mpeg.chiariglione.org/standards/exploration/free-viewpoint-television-ftv/call-evidence-free-viewpoint-television-super.

[33] BT.709: Parameter values for the HDTV standards for production and international programme exchange. ITU-R, Apr. 2004. URL:http://www.itu.int/rec/R-REC-BT.709/en.

[34] BT.2020: Parameter values for ultra-high definition television systems for production and international programme exchange. ITU-R,Oct. 2015. URL: http://www.itu.int/rec/R-REC-BT.2020/en.

[35] Ajay Luthra, Edouard Francois, and Walt (Eds.) Husak. Requirements and Use Cases for HDR and WCG Content Coding. Doc.N15084. Geneva, CH, 111th meeting: MPEG, Feb. 2015. URL:http://phenix.it-sudparis.eu/mpeg/doc_end_user/documents/111_Geneva/wg11/w15084.zip.

[36] Ajay Luthra, Edouard Francois, and Walt (Eds.) Husak. Call for Evidence (CfE) for HDR and WCG Video Coding. Doc. N15083.Geneva, CH, 111th meeting: MPEG, Feb. 2015. URL:http://phenix.it-sudparis.eu/mpeg/doc_end_user/documents/111_Geneva/wg11/w15083.zip.

[37] Requirements. Test Results of Call for Evidence (CfE) for HDR and WCG Video Coding. Doc. N15350. Warsaw, PL, 112th meeting:ISO/IEC JTC1/SC29/WG11, June 2015. URL:http://phenix.it-sudparis.eu/mpeg/doc_end_user/documents/112_Warsaw/wg11/w15350.zip (visitedon 07/03/2016).

[38] Jonatan Samuelsson. WD of ISO/IEC TR 23008-14 Conversion and coding practices for HDR/WCG video. Tech. rep. San Diego, CA,USA: ISO/IEC JTC1/SC29/WG11, Feb. 2016.

[39] Jonatan Samuelsson et al. Draft 2 of Conversion and Coding Practices for HDR/WCG YCbCr 4:2:0. Doc. JCTVC-X1017. 24th meeting,Geneva, CH: Joint Collaborative Team on Video Coding (JCT-VC) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, May 2016.URL: http://phenix.it-sudparis.eu/jct/doc_end_user/documents/24_Geneva/wg11/JCTVC-X1017-v1.zip.


http://mpeg.chiariglione.org/standards/exploration/free-viewpoint-television-ftv/call-evidence-free-viewpoint-television-super

http://mpeg.chiariglione.org/standards/exploration/free-viewpoint-television-ftv/call-evidence-free-viewpoint-television-super

http://www.itu.int/rec/R-REC-BT.709/en

http://www.itu.int/rec/R-REC-BT.2020/en

http://phenix.it-sudparis.eu/mpeg/doc_end_user/documents/111_Geneva/wg11/w15084.zip


http://phenix.it-sudparis.eu/mpeg/doc_end_user/documents/112_Warsaw/wg11/w15350.zip

http://phenix.it-sudparis.eu/jct/doc_end_user/documents/24_Geneva/wg11/JCTVC-X1017-v1.zip

http://phenix.it-sudparis.eu/jct/doc_end_user/documents/24_Geneva/wg11/JCTVC-X1017-v1.zip

References V[40] MPEG Requirements. Requirements for a Future Video Coding Standard v4. Tech. rep. N16359. Geneva, CH, 115th meeting: ISO/IEC

JTC1/SC29/WG11, June 2016. URL:http://phenix.it-sudparis.eu/mpeg/doc_end_user/documents/115_Geneva/wg11/w16359.zip.

[41] Marta Karaczewicz and Madhukar Budagavi. Report of AHG1 on Coding Efficiency Improvements. Doc. VCEG-AZ01. Warsaw, PL,52nd meeting: ITU-T SG16/Q6 VCEG, June 2015. URL:http://wftp3.itu.int/av-arch/video-site/1506_War/VCEG-AZ01.zip.

[42] Jianle Chen et al. Algorithm Description of Joint Exploration Test Model 1. Doc. JVET-A1001. Geneva, CH, 1st meeting: Joint VideoExploration Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, Oct. 2015. URL:http://phenix.int-evry.fr/jvet/doc_end_user/documents/1_Geneva/wg11/JVET-A1001-v1.zip.

[43] Jianle Chen et al. Algorithm Description of Joint Exploration Test Model 2. Doc. JVET-B1001. San Diego, CA, USA, 2nd meeting: JointVideo Exploration Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, Feb. 2016. URL: http://phenix.int-evry.fr/jvet/doc_end_user/documents/2_San%20Diego/wg11/JVET-B1001-v3.zip.

[44] Jianle Chen et al. Algorithm Description of Joint Exploration Test Model 3. Doc. JVET-C1001. Geneva, CH, 3rd meeting: Joint VideoExploration Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, May 2016. URL:http://phenix.int-evry.fr/jvet/doc_end_user/documents/3_Geneva/wg11/JVET-C1001-v3.zip.

[45] Xiang Li and Karsten Suehring. JVET common test conditions and software reference configurations. Doc. JVET-B1010. San Diego,CA, USA, 2nd meeting: Joint Video Exploration Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, Feb. 2016. URL:http://phenix.int-evry.fr/jvet/doc_end_user/documents/2_San%20Diego/wg11/JVET-B1010-v2.zip.

[46] Gary J. Sullivan and Jens-Rainer Ohm. Meeting report of the 2nd meeting of the Joint Video Exploration Team (JVET), San Diego, US,20–26 Feb. 2016. Doc. JVET-B1000. San Diego, CA, USA, 2nd meeting: Joint Video Exploration Team (JVET) of ITU-T SG 16 WP 3and ISO/IEC JTC 1/SC 29/WG 11, Feb. 2016. URL: http://phenix.int-evry.fr/jvet/doc_end_user/documents/2_San%20Diego/wg11/JVET-B1000-v1.zip.

[47] Elena Alshina et al. Description of Exploration Experiments on Coding Tools. Doc. JVET-B1011. San Diego, CA, USA, 2nd meeting:Joint Video Exploration Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, Feb. 2016. URL: http://phenix.int-evry.fr/jvet/doc_end_user/documents/2_San%20Diego/wg11/JVET-B1011-v2.zip.



http://wftp3.itu.int/av-arch/video-site/1506_War/VCEG-AZ01.zip

http://phenix.int-evry.fr/jvet/doc_end_user/documents/1_Geneva/wg11/JVET-A1001-v1.zip

http://phenix.int-evry.fr/jvet/doc_end_user/documents/2_San%20Diego/wg11/JVET-B1001-v3.zip


http://phenix.int-evry.fr/jvet/doc_end_user/documents/3_Geneva/wg11/JVET-C1001-v3.zip







References VI

[48] Teruhiko Suzuki and Jianle Chen. Work plan for investigation of test sequence encoding. Doc. JVET-A1002. Geneva, CH, 1st meeting:Joint Video Exploration Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, Oct. 2015. URL:http://phenix.int-evry.fr/jvet/doc_end_user/documents/1_Geneva/wg11/JVET-A1002-v1.zip.

[49] Andrey Norkin et al. Call for test material for future video coding standardization. Doc. JVET-B1002. San Diego, CA, USA, 2nd meeting:Joint Video Exploration Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, Feb. 2016. URL: http://phenix.int-evry.fr/jvet/doc_end_user/documents/2_San%20Diego/wg11/JVET-B1002-v1.zip.

[50] Teruhiko Suzuki. Work plan for assessment of test material. Doc. JVET-C1002. Geneva, CH, 3rd meeting: Joint Video ExplorationTeam (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, May 2016. URL:http://phenix.int-evry.fr/jvet/doc_end_user/documents/3_Geneva/wg11/JVET-C1002-v2.zip.

[51] Mathias Wien. “Variable Block-Size Transforms for H.264/AVC”. In: IEEE Transactions on Circuits and Systems for Video Technology13.7 (July 2003), pp. 604–613. DOI: 10.1109/TCSVT.2003.815380.

[52] Marta Karaczewicz and Elena Alshina. JVET AHG report: Tool evaluation (AHG1). Doc. JVET-C0001. Geneva, CH, 3rd meeting: JointVideo Exploration Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, June 1, 2016. URL:http://phenix.int-evry.fr/jvet/doc_end_user/documents/3_Geneva/wg11/JVET-C0001-v1.zip.

[53] Gary J. Sullivan and Jens-Rainer Ohm. Meeting report of the 3rd meeting of the Joint Video Exploration Team (JVET), Geneva, CH, 26May – 1 June 2016. Doc. JVET-C1000. 3rd meeting, Geneva, CH: Joint Video Exploration Team (JVET) of ITU-T SG 16 WP 3 andISO/IEC JTC 1/SC 29/WG 11, June 1, 2016. URL:http://wftp3.itu.int/av-arch/jvet-site/2016_05_C_Geneva/JVET-C_Notes_d7.doc.

[54] JAhG of MPEG and JPEG. Technical report of the joint ad hoc group for digital representations of light/sound fields for immersive mediaapplications. Doc. WG1N72033. Geneva, CH: ISO/IEC JTC1/SC29/WG1(JPEG) & WG11(MPEG), June 2016. URL: https://jpeg.org/downloads/jpegpleno/wg1n72033_20160603_report_jahg_light_sound_fields.pdf.


http://phenix.int-evry.fr/jvet/doc_end_user/documents/1_Geneva/wg11/JVET-A1002-v1.zip






http://wftp3.itu.int/av-arch/jvet-site/2016_05_C_Geneva/JVET-C_Notes_d7.doc

https://jpeg.org/downloads/jpegpleno/wg1n72033_20160603_report_jahg_light_sound_fields.pdf

https://jpeg.org/downloads/jpegpleno/wg1n72033_20160603_report_jahg_light_sound_fields.pdf

Presentations & Public Speaking

ICME 2016 - High Efficiency Video Coding - Coding Tools and Specification: HEVC V3 and Coming Developments