Error Resiliency & Concealment in H.264

Code from jm9.8/x264

http://getonebyone.com/H264.html

“To enable the compressed bit-stream to better resist channel errors”

OR

“Error resilience techniques enable the compressed bit-stream to resist channel errors so that the

impact on the reconstructed image quality is minimal”

What is Error Resiliency?

But…

some part of the information is inevitably lost and the goal of error concealment, at the receiving side, is to estimate the losses and conceal them in the displayed video.

Error Resiliency Compression

1

Because, generally, the error resiliency schemes introduce some redundancy in the data. On the other hand, compression schemes aim to remove various redundancies from the data

In H.264 we can find several parameters that can be tuned so that a trade-off between compression rate and error resilience can be made targeting different type of problems found in heterogeneous environments.

Tradeoff

H.264 Eco SystemHeterogeneous Environments

Transport

EntropyCoding

DataPartition

NAL

internet

Packets(nal-units or slices)

NALEntropy

DecodingDecode

Prediction Info, Quantized Transform Coeffs….

VLC or AC

VLC CodedData

Coded MBs

Coded Slice/Partition

Packets(nal-units or slices)

Encode

Abstract Level Flow Diagram

Resilience & Concealment in Heterogeneous Environment

Error Propagation A transmission error such as a bit error or packet loss may cause a

video decoder to lose synchronization with the sequence of decoded VLCs.

This can cause the decoder to decode incorrectly some or all of the information after the occurrence of the error and this means that part or all of the decoded VOP will be distorted or completely lost (i.e. the effect of the error spreads spatially through the VOP, ‘spatial error propagation’).

If subsequent VOPs are predicted from the damaged VOP, the distorted area may be used as a prediction reference, leading to temporal error propagation in subsequent VOPs.

Spatial and Temporal Error Propagation

Error Position

Damaged area

Forward Prediction

Error Resilience Employed w.r.t Profiles

Baseline profile includes some enhanced error resilience tools – Flexible Macroblock Ordering (FMO), – Arbitrary Slice Ordering (ASO), and – Redundant Slices (RS)

Extended profile adds further error resilience support in the form of data partitioning (DP).

Main profile does not include enhanced error resilience tools like FMO, ASO, RS, DP, SP or SI Slices.

Error Resiliency Schemes

Flexible Macroblock Ordering (FMO) Arbitrary Slice Ordering (ASO) Data Partitioning (DP) Redundant Slices (RS) SP/SI frame for bitstream switching Reference Frame Selection Intra-block refreshing by R-D control. Random Macroblock Intra Refresh

Flexible Macroblock Ordering (FMO)

Why FMO? Because it aims at avoiding error propagation from a corrupted

packet to subsequent packets.

Objective? The objective behind FMO is to scatter possible errors to the whole

frame as equally as possible, to avoid error accumulation in a limited region.

How it works? It randomizes the data prior to transmission, so that if a segment of

data is lost (e.g. a packet or several packets), the errors are distributed more randomly over the video pictures, rather than causing corruption of a complete regions, making it more likely that relevant neighboring data is available for concealment of lost content.

FMO cont..

Randomizing Data To achieve this, each MB is statically assigned to a slice

group using Macroblock Allocation Map (MBAmap).

How does a MBAmap look like? The MBA_Map is a data structure that maps the spatial

address of a MB to a slice group, in a raster-scan order.

FMO cont..

MBA_Map in H.264 supports 7 arbitrary shapes: 1. Interleaved2. Dispersed3. Foreground & Background4. Box-out5. Raster scan6. Wipe7. Explicit

The parameters that control the mapping functions are included in a Picture Parameter Set (PPS) which is sent to the receiver.

FMOInit

FmoGenerateMapUnitToSliceGroupMap

FmoGenerateMBAmap

FmoGenerateType0MapUnitMap

FmoGenerateType1MapUnitMap

FmoGenerateType2MapUnitMap

FmoGenerateType3MapUnitMap

FmoGenerateType4MapUnitMap

FmoGenerateType5MapUnitMap

FmoGenerateType6MapUnitMap

FMO cont..

Main Encode_one_frame()

Frame_picture()

Field_picture()

FMOInit

Arbitrary Slice OrderingWhy ASO? Since each slice of a coded

picture can be decoded independently of the other slices in the picture, the H.264 design enables sending and receiving the slices of the picture in any order relative to each other.

OR Slices in a coded frame may

follow any decoding order.

sender receiver

Slice # 0

Slice # 1

Slice # 2

0

20

40

Slice Size Criteria Encoder intelligently groups MBs into a slice whose size

is less than (or equal to) the size of MTU (Maximum Transportation Unit).

MTU represents the largest size of a packet that can be transported through networks without being split.

Arbitrary Slice Ordering

Data partitioning enables unequal error protection (UEP) according to syntax elements’ importance, since some syntax elements in the bitstream are more important than others.

The coded data that makes up a slice is placed in three separate Data Partitions (A, B and C), each containing a subset of the coded slice.

Each Partition can be placed in a separate NAL unit and may therefore be transported separately.

Information regarding the Data Partitioning is put into the slice header.

Data Partition (DP)

Data Partition (DP)DP A: Contains header information (MB types, quantization params,

and motion vectors). With the loss of data of DP A, data of the other two partitions

becomes useless.

DP B: Contains Intra CBPs and transform coefficients of I-blocks. The loss of this part will severely impair the recovery of successive

frames due to error propagation, because intra frame and intra-MBs are used as references.

DP C: Contains Inter CBPs and coefficients of P-blocks. Compared to the DPs A and B, the data contained in DP C is less

important. However, it is the biggest partition of a coded slice because a large number of frames are coded as p-frames.

Error concealment mechanism implemented by the decoder

*A = header info *B = Intra CBPs and I-block trans-coeffs*C = Inter CBPs and P-block trans-coeffs

Concealment in Data Partitioning

Error Concealment API Encoder & Decoder Both Have the Same DataPartition Struct

//! DataPartitiontypedef struct datapartition{

Bitstream *bitstream; EncodingEnvironment ee_cabac;

int (*writeSyntaxElement)(SyntaxElement *, struct datapartition *); /*!< virtual function;actual method depends on chosen data partition and entropy coding method */

} DataPartition;

//! Slicetypedef struct{DataPartition *partArr; …..} Slice

Code_a_picturre

Encode_one_Slice()Start_slice()

Generates Slice (Partiton) Header

Main Encode_one_frame()

Frame_picture()

Field_picture()

Data Partition (DP)

SliceGroupId, Picture *, TotalCodedMBs

Picture *

Picture *

Picture *

Picture *

#define MAXSLICEPERPICTURE 100typedef struct { int no_slices; Slice *slices[MAXSLICEPERPICTURE]; ……} Picture;

Redundant Slices

Why? In order to enhance robustness in the event of loss of slices

due to packet drops in network during transmission

Usage In normal operation, the decoder reconstructs the frame

from ‘primary’ pictures and discards any redundant pictures. However, if a primary coded picture is damaged, the decoder may replace the damaged area with decoded data from a redundant picture if available.

A redundant slice is another representation of one or more MBs in the same bitstream.

Encoder put redundant slice information in PPS

SP- /SI- Synchronization/ Switching Frame

SP-/SI mechanism is designed for the purpose of video bitstream switching

It is regarded as an important error resiliency feature in network environments with feedback channel

This scheme assumes that more than one video bitstreams are being sent through network.

In the case of synchronization-lost or presence of error in bitstream, the decoder can signal the presence of error in the frame(s) via feedback channel and the SP frame can be generated for error recovery.

SP/SI-Switching H.264/AVC uses SP- frames that enable random

switching and its info is put in ImageParameters struct

Same valueEn

cod

er

Dec

od

er

Feedback channel

Reference Frame Selection

The reference frame selection mode can be applied both with and without feedback channel (channel which gives lost-packet report)

The reference frame selection with feedback channel uses the reference frame based on the feedback acknowledgement

Reference Frame Selection

There are four possible modes defined in reference frame mode

1. Neither:no back-channel data is returned from the decoder to the encoder.

2. ACK: the decoder returns only acknowledgment messages. In this mode the encoder uses only an acknowledged segment as a reference for inter-frame encoding.

3. NACK: the decoder returns only non-acknowledgment messages. Whenever NACK is received for a frame, the new reference frame is selected from amongst the frames coded before the NACK frame.

4. ACK+NACK: the decoder returns both acknowledgment and non-acknowledgment messages. In the ACK+NACK mode, the encoder switches between two kinds of modes according to the upstream messages from the decoder.

The error propagation and drift due to predictive coding can be eliminated by periodically inserting intra-coded MBs in the

bitstream.

Adaptive Intra Refresh (AIR):H.264 uses intelligent intra-block refreshing by R-D (rate-distortion) control, such that an appropriate block coding option o* will be selected that minimizes the Lagrangian cost function, unlike the periodic or random intra refresh in earlier standards (e.g., MPEG-4, H.263)

Rate Distortion Optimization

Here,o = block coding mode (inter/intra mode and block size)

D(o) = distortion introduced by encoding with mode o and is computed by the SAD (sum of absolute difference) in low complexity mode and by SSD (sum of squared difference) in high complexity mode

R(o) = corresponding coding rate, whereas for inter-block mode it represents the block residual and corresponding motion vector(s).

λ = Lagrange parameter for appropriate weighting of rate and distortion.

Intra Block Refreshing by R-D Control

double RDCost_for_8x8blocks (

int* cnt_nonz, // --> number of nonzero coefficients int64* cbp_blk, // --> cbp blk double lambda, // <-- lagrange multiplier

int block, // <-- 8x8 block numberint mode, // <-- partitioning mode

short pdir, // <-- prediction direction short ref, // <-- reference frame short bwd_ref // <-- abp type)

Error Concealment API

Desc: Calculates Cost function to find the appropriate block coding option. Return: Get Rate-Distortion Cost

Intra Block Refreshing by R-D Control

In order to enhance robustness to data loss, the H.264/AVC design contains a new ability to allow an encoder to send redundant representations of regions of pictures.

void RandomIntraInit(int xsize, int ysize, int refresh)

Desc: Initializes Random Intra module.

xsize, ysize = size of picture in MBs

refresh= refresh rate in MBs per picture

void RandomIntraNewPicture()

Desc: Selects new set of MBs for forced Intra based on the RefreshPattern initialized in RandomIntraInit()

Random Intra MB Refresh

Main Init_img() RandomIntraInit()

int xsize, int ysize, int refresh

Code_a_picturreMain Encode_one_frame()

Frame_picture()

Field_picture()

Picture *

Picture *

Picture *

Picture *

RandomIntraNewPicture()

Called, only after initialization (or changes) of the picture size or the random intra refresh value, immediately after parsing config file

Allocates forced INTRA MBs

Random Intra MB Refresh

Error Concealment

Decoder Error Concealment

Decode ALL Frames(main)

Docode_One_Frame

( Flag, DecParams, DecOutputs )

Read_New_Slice

( DecParams, DecInputs, DecOutputs )

Flag is the state variable for error detector and concealor

( DecParams, DecInputs, DecOutputs )

The Endhttp://getonebyone.com/

H264.html