Image and Video Coding II - TU Berlin ... k Optimalbitallocation = dD k dR k = 2ln2"2 k«â„¢ 2 k 2 2R k

  • View
    0

  • Download
    0

Embed Size (px)

Text of Image and Video Coding II - TU Berlin ... k Optimalbitallocation = dD k dR k = 2ln2"2...

  • JPEG

  • Introduction

    Video Codec

    video

    encoder

    video

    decoder

    input

    sample

    arrays

    bitstream reconstr.

    sample

    arrays

    Video Encoder Maps raw sample arrays into a bitstream Determines coding efficiency for given bitstream syntax and decoding process

    Video Decoder Reconstructs sample arrays from the received bitstream

    Typical use case Encoder and decoder know bitstream syntax and decoding process

    T. Wiegand (TU Berlin) — Image and Video Coding: JPEG 2 / 49

  • Introduction

    Lossy Compression

    HD movie UHD broadcast Video chat on Blu-ray disc over DVB-S2 over the Internet

    raw video 1920×1080, 24 fps, 3840×2160, 60 fps, 1280×720, 50 fps, data format Y’CbCr 4:2:0, 8 bit Y’CbCr 4:2:0, 10 bit Y’CbCr 4:2:0, 8 bit

    raw data rate ca. 600 Mbit/s ca. 7.5 Gbit/s ca. 550 Mbit/s

    channel bit rate 36 Mbit/s (read speed) 58 Mbit/s (8PSK 2/3) depends on connection

    typical ca. 20 Mbit/s ca. 25 Mbit/s ca. 1 Mbit/svideo bit rate

    required ca. 30 : 1 ca. 300 : 1 ca. 500 : 1compression

    Required video bit rate is typically much smaller than raw data rate Required bit rate can only be achieved with lossy compression Decoded signal represents approximation of the input signal Goal: Represent signal with small impact on perceived quality

    T. Wiegand (TU Berlin) — Image and Video Coding: JPEG 3 / 49

  • Introduction

    Coding Efficiency

    Coding efficiency Trade-off between bit rate and reconstruction quality Measured for sets of representative test sequences

    Bit rate Use average bit rate in our comparisons (count bits in bitstream) In practice: Bit distribution among frames is also important

    Reconstruction quality Human perception of visual quality: Difficult to evaluate Most reliable method: Subjective viewing tests (expensive, time consuming) Use mean squared error (MSE) and peak signal-to-noise ratio (PSNR)

    MSE = 1

    W · H

    W−1∑ x=0

    H−1∑ y=0

    ( s ′[x , y ]− s[x , y ]

    )2 PSNR = 10 · log10

    (2B − 1)2

    MSE (B: bit depth)

    T. Wiegand (TU Berlin) — Image and Video Coding: JPEG 4 / 49

  • Introduction

    Rate-Distortion Curves & Bit-Rate Savings

    35

    36

    37

    38

    39

    40

    41

    0 1000 2000 3000 4000 5000

    P S

    N R

    (Y C

    bC r)

    [ dB

    ]

    bit rate [kbit/s]

    codec B

    codec A RB

    RA

    example target quality

    ( 39 dB )

    0

    10

    20

    30

    40

    50

    60

    70

    36 37 38 39 40 41

    bi t-r

    at e

    sa vi

    ng [%

    ]

    PSNR (YCbCr) [dB]

    BBA = (RA - RB) / RA target quality

    ( 39 dB )

    example

    bit rate saving of codec B vs. codec A

    Rate-distortion curve Measure average bit rate and average PSNR for multiple operation points

    Bit-rate savings Determine bit-rate savings using interpolated rate-distortion curves

    BBA = (RA − RB ) /RA (codec B relative to codec A)

    Calculate average bit-rate savings by integrating curve T. Wiegand (TU Berlin) — Image and Video Coding: JPEG 5 / 49

  • Intra-Picture Coding

    Still Image Coding / Intra-Picture Coding

    Still Image Coding Exchange, transmission and storage of images Used in virtually all digital cameras and picture editing applications Most widely used image coding standard: JPEG Further standards: JPEG-2000, JPEG-XR, HEVC-Intra

    Simple Approach for Video Coding Separate coding of pictures of a video sequence

    Hybrid Video Coding: Two types of block coding modes Intra-picture coding modes

    Represent blocks of samples without referring to other pictures Utilize only dependencies inside pictures

    Inter-picture coding modes Utilize dependencies between pictures (motion-compensated prediction)

    T. Wiegand (TU Berlin) — Image and Video Coding: JPEG 6 / 49

  • Intra-Picture Coding

    Intra-Picture Coding in Hybrid Video Codecs

    Two different settings:

    Intra Pictures All blocks of a picture are coded using intra-picture coding modes First picture of a video sequence Required for “clean” random access / bitstream splicing Can be advantageous in error-prone environments

    Individual Intra Blocks Some blocks of an inter picture are coded in intra-picture coding modes Increases error robustness Older standards: Stops accumulation of transform mismatches Main reason: Coding efficiency (remember: Non-matched prediction can decrease coding efficiency)

    T. Wiegand (TU Berlin) — Image and Video Coding: JPEG 7 / 49

  • Intra-Picture Coding

    Intra Blocks in Inter Pictures — Coding Efficiency

    32

    33

    34

    35

    36

    37

    38

    0 2 4 6 8 10

    P S

    N R

    (Y )

    [d B

    ]

    bit rate [Mbit/s]

    Basketball Drive, IPPP

    all coding modes

    intra-picture coding modes are disabled in inter pictures

    0

    5

    10

    15

    20

    25

    30

    32 33 34 35 36 37 38

    bi t-r

    at e

    in cr

    ea se

    [% ]

    PSNR (Y) [dB]

    Basketball Drive, IPPP

    bit-rate increase due to disabling of intra-picture

    coding modes in inter pictures (on average, 10.9%)

    Example: IPPP coding with H.265 | MPEG-H HEVC Disabling of intra blocks in inter pictures ca. 11% bit-rate increase Certain regions of a picture cannot be well predicted using MCP

    Uncovered background Regions with non-translational motion ...

    T. Wiegand (TU Berlin) — Image and Video Coding: JPEG 8 / 49

  • JPEG Overview

    The JPEG Standard

    Joint Photographic Experts Group (JPEG) Standard is named after the group which created it Joint committee of ITU-T and ISO/IEC JTC 1 ITU-T Study Group 16, Question 6 (Visual Coding Experts Group: VCEG) ISO/IEC Joint Technical Committee 1, Subcommittee 29, Working Group 1 (ISO/IEC JTC 1/SC 29/WG 1)

    Standard: Digital Compression and Coding of Continuous-Tone Still Images Officially ITU-T Rec. T.81 and ISO/IEC 10918-1 Commonly referred to as JPEG Specifies compression for gray-level and color images Work commences in 1986 Standard published in 1992 Still most widely used image compression

    T. Wiegand (TU Berlin) — Image and Video Coding: JPEG 9 / 49

  • JPEG Overview

    JPEG: Color Components and Partitioning

    Color components (e.g., Y, Cb, Cr) are coded independently of each other Color components are partitioned into 8×8 blocks (padding at borders) The 8×8 blocks are coded using transform coding

    T. Wiegand (TU Berlin) — Image and Video Coding: JPEG 10 / 49

  • JPEG Overview

    JPEG: Block Transform Coding (8× 8 Blocks)

    2D

    transform

    scalar

    quantization

    entropy

    coding

    block of

    samples

    sequence

    of bits

    2D inverse

    transform

    decoder

    mapping

    entropy

    decoding

    reconstructed

    block

    2D Block Transform Energy compaction (reduce dependencies between coefficients)

    Scalar Quantization Approximate signal in a suitable way

    Entropy Coding Represent quantization indexes with a small number of bits

    T. Wiegand (TU Berlin) — Image and Video Coding: JPEG 11 / 49

  • 2D Block Transform / Orthogonal Block Transform

    Orthogonal Block Transform

    Linear Transform General case: Samples of a block are arranged in a vector svec Forward and inverse transforms are given by

    tvec = A · svec and s′vec = B · t′vec s′vec is given by weighted sum of transform basis functions (columns of B)

    s′vec = t ′0 · b0 + t ′1 · b1 + t ′2 · b2 + · · ·

    Perfect Reconstruction Perfect reconstruction in the absence of quantization

    =⇒ A = B−1

    Orthogonal Transforms Transform basis functions bk are orthogonal to each other Transform basis functions bk have unit norms (bTk bk = 1)

    =⇒ A = B−1 = BT

    T. Wiegand (TU Berlin) — Image and Video Coding: JPEG 12 / 49

  • 2D Block Transform / Orthogonal Block Transform

    Orthogonal Block Transform

    Orthogonal Block Transforms Rotation (and reflection) in signal space (axes remain orthogonal) Forward and inverse transforms are given by

    tvec = BT · svec and s′vec = B · t′vec

    Main Advantage SSD distortion in signal space = SSD distortion in transform domain

    D = (s − s′)T (s − s′) = (t − t′)T BTB (t − t′) = (t − t′)T (t − t′) = ∑ k

    (tk − t ′k)2 = ∑ k

    (sk − s ′k)2

    SSD distortion can be minimized with independent scalar quantizers Lagrangian costs D + λ · R can be minimized using simple algorithms

    T. Wiegand (TU Berlin) — Image and Video Coding: JPEG 13 / 49

  • 2D Block Transform / Separable 2D Block Transforms

    Separable 2D Block Transforms

    Separable Transforms N×M blocks of samples and transform coefficients Forward and inverse transforms are given by

    t = BTV · s · BH and s′ = BV · t′ · B T H

    Interpretation (forward transform) 1 Transform all columns of the block using the vertical transform 2 Transfor