Video Compression with Film Grain

Aesthetics Preservation

Mickael RAULETCTO @ ATEME

2

By considering dedicated features

• Grain• Legacy• Artistic intent

• Like it or not, people want grain• But grain is uncompressible• AV1 has a very unique feature:

• Film Grain Synthesis (FGS)

• Not exactly new, but, • Normative and MANDATORY for the first time

• Highly regarded by Netflix (proponent)• Gain is huge• FGS was previously available in AVC and

HEVC through SEIs (partially supported by decoders)

3

AV1: Royalty-Free Codec (October 2018)

• AV1 (AOM Video codec 1)• Alliance for Open Media (AOM)

• Interoperable and open

• Optimized for OTT delivery

• High Video Quality and Real-time delivery modes

• Compression gains up to 20% w.r.t state-of-the-art

• From low to high resolutions (including UHD, HDR, WCG)

• Classical structure + • new coding tools

• additional features (Film grain synthesis, scalability, SCC)

4

Coding Structure: HEVC vs AV1HEVC AV1

Multiples sizes/forms blocks: 128x128 to 4x4 (mixing intra and inter)Rectangular separable TUs: 64x64 to 4x4 (4 kernel types)

Recursive tree coding structure

56 Intra prediction directions + 11 non-directional (DC + Paeth + 3 Smooth + 5 Filter + Chroma from Luma)

Coding block128x128 to 4x4

Multiples sizes/forms PUs: 64x64 to 4x4 (intra or inter for all PUs)Square non separable TUs: 32x32 to 4x4 (2 kernel types)

Quadtree coding structure

33 Intra prediction directions + 2 non-directional (DC + PLANAR)

17 1

6 1

5 1

4 1

3 1

2 1

1 1

0 9

8 7

6 5

4 3

2

18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34

0 : Intra_Planar

1 : Intra_DC

35: Intra_FromLuma

Coding Unit64x64 to 8x8

Efficient spatio-temporal mv prediction (Merge, AMVP) Efficient spatio-temporal mv prediction (Refmv, Newmv)

5

Film Grain Synthesis process

• Straightforward idea

• Challenging implementation

What we have

What we encode

What we synthesize

6

Denoising and modeling

• 2 steps denoising• Rough + fine denoising

• Or pre-analysis + fine denoising

• Grain parameters computation

L = 2

L = 2

7

For all the flat blocks indexed from 1 to m, we look for 𝜑 minimizing the quantity:

control on the AR coefficients + unicity of the solution guaranteed

andwith

𝜑 is solution of the linear system:

.

Computation of the AR coefficients 𝜑1,…, 𝜑𝑝

Ridge regression

Advantages : - Taking λ hi h enough, the scheme is guaranteed to be stationary- Taking λ hi h enough, the coefficients are < 2

8

About Unpleasing Patterns

• It can happen by design

• Detecting and avoiding them

Pattern selectedPattern not selected

9

Compression Gains

Grain Encoding (no film grain synthesis) Film Grain Synthesis

Tag Bitrate(Mbps)

Comments Bitrate(Mbps)

Gain(in %)

Chimera-DinnerSceneUHD 10bits

112 33.90 Grain preserved 5.23 84,57

118 10.25 Grain washed out 3.33 67,49

ZombieLongPS_cut2UHD 10bits

112 29.16 Grain preserved 15.02 48,48

124 10.71 Ugly unstable grain patterns 7.78 27,33

ebu_farytale1HD 8bits

85 19.26 Grain preserved 9.52 50,61

90 9.94 Grain already washed out, even though bitrate is quite high

5.99

39,79

10

Parts of the Reconstructed Frames

Coded at 10 709 kbps

Coded at 9 944 kbps

Coded at 7 782 kbps

Coded at 5 987 kbps

Grain Management

ZombieLongPS_cut2UHD 10bits

ebu_fairytale1HD 8bits

Source #1 version #2 version

Source #1 version #2 version

13

Considering dedicated features: FGS

• Still ongoing effort• Patents

• Research papers

• Practical tests / interoperability• TV LG

• STB broadcom

• NAB DEMO AV1 8k HDR + grain, with compatible TV set !

• Differentiating feature for VOD operators

14

MERCI, Danke, gracias , obrigado, tack, Dank, grazie, Thank you,dankie, faleminderit, Barak Allahu fiik, chnorakaloutioun, çox sag olun, a ni kié,bedankt, waita, eskerrik, dhanyabaad, dziakuju, thint ko, kyay tzu tin pa te, a nikié, trugéré, blagodaria, gràcies, salamat, kam sah hamnida, grazie, mèsi, tak,dankon, akpé, kiitos, multumesc, gracie, a dank, tesekkur ederim, tapadh leat,go raibh maith agat, diolch, efharisto, aguyjé, meherbani, mahalo, toda, tau,dhanyavad, köszönöm, terima kasih, takk, arigatô, tanemirt, hvala,dhanyavadagalu, akun, murakoze, sobodi, tenki, spas, mercé, khob chai, gratiasago, paldies, choukrane, aciu, blagodaram, terima kasih, misaotra, nizzik hajr,aabhari aahe, bayarlalaa, dank u wel, mercé, shukriya, danki, motashakkeram,mamnun, dzikuj, dakujem, multumesc, spacibo, marci, faafetai lava, gratzias,ahsante, salamat po, mauruuru, nanedri, rahmat, dkuji, kop khun krap,yekeniele, sagolun, diakuiu, cám Ön, grces, djiere dieuf, enkosi, ngiyabonga