MPEG2 FGS Implementation

ECE 738

Advanced Digital Image Processing

Author: Deshan Yang05/01/2003

Introduction of FGS FGS = fine granularity scalability For MPEG2 / MPEG4 and others Comparing to SNR, temporal, spatial

scalability, FGS enhances video quality continuous instead of stepwise.

Using bitplane coding for enhancement layer, base layer is still using regular MPEGx coding

Enhancement layer bitstream can be truncated. The quality of video stream is proportional to the amount of enhancement bitstream data used.

Project outline Implement MPEG2 FGS enhancement into the current

MPEG2 codec project from MPEG MSSG (MPEG Software Simulation Group).

Implementation includes:• Updating the current MPEG2 codec source code to support FGS• Implementing bitplane VL Codec and bit streaming• Implementing a new FGS enabled MPEG2 player

Challenges• Understanding and using the MSSG MPEG2 codec source code

(much more difficult than it seems to be)• Bitplane coding / decoding• MPEG2 player on Windows platform• Media data to play with

Project outline (cont.)

DCT Q

Regular MPEG2encoder

Bit-plane codingNoise

channelsimulator

Bit-plane decoding

Regular MPEG2decoder

IQIDCT

Base layer

Enhancementlayer

Enhancementlayer

Base layer

VideoInput

VideoOutput

Eva

lua

tion

Source code

MPEG2 codec source code is from:MPEG Software Simulation Group (MSSG) at http://www.mpeg.org/MPEG/MSSG/

MPEG2 player source code: searching google.com for mpg2w11b.zip.

My MPEG2 FGS player is combined from the the MPEG2 player with my MPEG2 FGS decoder MPEG2 player source.

MPEG2 Encoder More configurable options to control FGS

encoding. Integrating FGS enhancement layer bitplane

encoder into the original MPEG2 encoder• Encoding base layer in very low bit rate• Calculate the DCT coefficients residues before and after

the quantization step.• Encoding the DCT coefficients residues with the bitplane

VLC• Encoded bitplane VLC output data are stored (as binary

files) as the enhancement layer bitstream Supporting I-frame and P-frame, not B-frame

MPEG2 FGS Encoder Architecture

DCT

Q-1

IDCT

ClippingMotion

Estimation

MotionCompensation

Base LayerMPEG VLC

FrameMemory

Q

-

+

Base

La

yer

Mo

tion V

ect

or

Base Layer

Input Video Stream Base LayerOutput

Bitplaneshift

FindMaximum

BitplaneVLC

EnhancementLayer Output

Enhancement Layer

1 2

-

+

MPEG2 Decoder More configurable options to control FGS decoding Integrating the FGS bitplane VLD into the original

MPEG2 decoder• Accessing the enhancement layer bit stream (from binary

files on disk)

• Decoding DCT residues from the enhancement bit stream

• Control the degree of enhancement

• Generating the enhanced DCT coefficients

• Decoding the enhanced DCT coefficients to get better video over base layer

Supporting I-frame and P-frame, not B-frame

MPEG2 FGS Decoder Architecture

Q-1 IDCT Clipping

MotionCompensation

FrameMemory

Base Layer Motion Vector

Base LayerMPEG VLD

Base LayerBit Stream

Base Layer

EnhancedVideo

Base LayerVideo

FindReference

EnhancementLayer

Bitplane VLD

BitplaneShift

IDCT Clipping

EnhancementLayer

MPEG2 Player Combination of original MPEG2 player with the new

MPEG2 FGS decoder New features

• Supporting FGS• Replay• Stop• Stretching to window• Supporting wide screen format• FGS scale change on the fly• Turning FGS on/off• Player window resizing• Supporting new command line options

DCT Coefficients Before quantization

88,-14,0,0,-3,0,0,0,-40,-2,-2,0,0,0,0,0,-7,8,3,0,0,0,0,0,-5,0,0,0,0,0,0,0,0,3,0,0,0,0,0,0,0,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4,0,0,0,0,2

After base layer quantization88,0,0,0,0,0,0,0,-32,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,-0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1

Difference0,-14,0,0,-3,0,0,0,-8,-2,-2,0,0,0,0,0,-7,8,3,0,0,0,0,0,-5,0,0,0,0,0,0,0,0,3,0,0,0,0,0,0,0,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4,0,0,0,0,1

Sign bitsx,1,x,x,1,x,x,x,1,1,1,x,x,x,x,x,1,0,0,x,x,x,x,x,1,x,x,x,x,x,x,x,x,0,x,x,x,x,x,x,x,0,x,x,x,x,x,x,x,x,x,x,x,x,x,x,x,x,0,x,x,x,x,0

Compressed sign bits1,1,1,1,1,1,0,0,1,0,0,0,0

Bitplane coding DCT coefficient differences:

0, -14, 0, 0, -3, 0, 0, 0, -8, -2, -2, 0, 0, 0, 0, 0, -7, 8, 3, 0, 0, 0, 0, 0, -5, 0, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0, 0

Bit planes:0: 00000000000000000000000000000000000000000000000000000000000000001: 00000000000000000000000000000000000000000000000000000000000000002: 00000000000000000000000000000000000000000000000000000000000000003: 00000000000000000000000000000000000000000000000000000000000000004: 01000000100000000100000000000000000000000000000000000000000000005: 01000000000000001000000010000000000000000000000000000000001000006: 01001000011000001010000000000000010000000100000000000000000000007: 00001000000000001010000010000000010000000100000000000000000000008: 0200200022200000211000002000000001000000010000000000000000100000 (sign bits)

Compressed sign bits: 1,1,1,1,1,1,0,0,1,0,0,0,0

Bitplane VLC encodingPlanes 0,1,2,3 are all zeroPlane 4: (1, 0, 0), (6, 0, 0), (8, 1, 0)Plane 5: (1, 0, 0), (14, 0, 0), (7, 0, 0), (33, 1, 0)Plane 6: (1, 0, 0), (2, 0, 0), (4, 0, 0), (0, 0, 0), (5, 0, 0), (1, 0, 0), (14, 0, 0), (7, 1, 0)Plane 7: (4, 0, 0), (11, 0, 0), (1, 0, 0), (5, 0, 0), (8, 0, 0), (7, 1, 1)

Bitplane coding (cont.)Bit plane VLC coding Binary code

0 All zero 3F

1 All zero 3F

2 All zero 3F

3 All zero 3F

4 (1, 0, 0), (6, 0, 0), (8, 1, 0) 01, 06, 48

5 (1, 0, 0), (14, 0, 0), (7, 0, 0), (33, 1, 0) 01, 0E, 07, 61

6 (1, 0, 0), (2, 0, 0), (4, 0, 0), (0, 0, 0), (5, 0, 0), (1, 0, 0), (14, 0, 0), (7, 1, 0)

01, 02, 04, 00, 05, 01, 0E, 47

7 (4, 0, 0), (11, 0, 0), (1, 0, 0), (5, 0, 0), (8, 0, 0), (7, 1, 1)

04, 0B, 01, 05, 08, C7

Code format: BPnnnnnn

B: End of block flag, P: End of plane flag, nnnnnn: number of 0 before 1

3F=00111111=“All zero plane”, BF=0x10111111=“All planes are zero)

Bitplane VL Codec VLC

• Splitting DCT coefficients to bitplanes

• Converting each bitplane to VL codes

• Converting VL codes to binary codes (Huffman codes)

• Storing binary codes as enhancement layer bitstream (binary file)

VLD• Loading binary codes from enhancement layer bitstream

(binary file)

• Converting binary codes (Huffman codes) to VL codes

• Converting VL codes to bitplanes

• Merging bitplanes to DCT coefficients

Enhancement Layer Bitstream

No FGS 15% FGS

30% FGS 30% FGS65% FGS

Demo

Run fgs.bat

Issues Enhancement layer bitstream size is much bigger

than the base layer bitstream size. Motion prediction can not be applied to enhancement

layer Current implementation only supports I-frame and P-

frame, not B-frame yet Current implementation does not support Huffman

VLC coding in the bitplane VLC. I don’t have a good Huffman VLC table, and Huffman VLC will only help to reduce the enhancement bitstream size ¼ anyway.