FrameFree Video Compression

Critical Points versus Block Matching

FrameFree Technologieshttp://framefree.com

Igor Borovikov

Overview

• Motion compensation for video compression,• Critical Points Filter: brief introduction,• FrameFree Video: a bird eye view and side by

side comparison of schematics with MPEG,• Occlusions and CPF,• Side by side comparison CPF3 and H.264,• FFV features overview,• Future work.

Motion Compensation

• Block Matching Algorithm – dominant motion predictor.

• Advantages: – Conceptually simple yet effective algorithm,– Low CPU requirement on decoder side,– Motion vectors data is well suited for entropy

compression.

BMC Downsides

Inherently discrete nature of blocks leads to:• Resolution dependence

(partially addressed by adaptive block size),

• Block artifacts (addressed by deblocking filters),

• Optimal motion vectors (exhaustive search) are usually prohibitively expensive,

• No simple way to reconstruct good quality non-integer frames,

• BMC has no direct relation to Human Visual System.

Alternatives

• Optical flow and related methods. A large group of pixel matching algorithms, yet not directly suitable for motion compensation in video compression.

• Critical Points Filter. The exceptional robustness of CPF pixel matching and adaptive mesh brings CPF to practical applications.

Critical Points Filter (CPF)

• Y. Shinagawa, and T. L. Kunii, Unconstrained Automatic Image Matching Using Multi-resolutional Critical-Point Filters, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.20, no.9, pp994-1001, Sep 1998

• K. Habuka and Y. Shinagawa, Image Interpolation Using Enhanced Multi-resolutional Critical-Point Filters, International Journal of Computer Vision, vol.58, no.1, pp19–35, 2004

CPF mimics HVS

HVS can reconstruct transition between these two images

<====>So can CPF.

Motion Compensation with CPF

“I Frame” “B-Frame” “P-Frame”Original frame compressed with, say, JPEG.

CPF- Interpolated between I and P frames: information from both sides of interval is used.

Forward motion compensated and corrected frame.

Motion Compensation with CPF

BMC vs. CPF

Motion compensationwith CPF (uncorrected)

Destination image Block motion compensation,Search radius 16 (uncorrected)

Source image

CPF vs. BMC

BMC Metaphor:

- “moving around tiles”;

CPF Metaphor:

- “deforming flexible film fixed on a rectangular frame, no tearing or folding allowed”.

• Bijectivity – no “holes” left after compensation,• Adaptive mesh - resolution independence,• Continuous time - no discretization

whatsoever,• “Key features” – critical points – are matched

with high precision, thus: • Remaining artifacts are not too conspicuous for

HVS,• Residual error of motion compensation is

relatively well suited for combination of custom JPEG-like and entropy coding.

Motion Compensation with CPF

MPEG vs. FFV encoding comparison

MPEG vs. FFV encoding comparison

“Practical Scheme”

TRANS: Image Translator TRANS: Image Translator －－ : : Differential Encoder & Noise FilterDifferential Encoder & Noise FilterINTERP: Interpolation Unit INTERP: Interpolation Unit ＋＋ : Differential Decoder: Differential Decoder

Occlusions and CPF“Boy and Toys” (NHK test sequence) with ball bouncing off the floor:

CP

F3 O

riginal C

PF

“CPF3” is the latest modification of FFV Practical Scheme that addresses occlusion problem.

Compression ratio and PSNR

Sequence FFV “Practical Scheme”,

12.2005FFV “CPF3”

12.2006

H.264 Med H.264 Low

Yachting 313Kb,27.93

328Kb,30.12

301Kb,34.82

94Kb,30.71

Intersection 257Kb,30.48

206Kb,31.47

253Kb,33.91

63Kb,29.96

Horse race 653Kb,28.35

807Kb,31.13

582Kb,35.25

215Kb,32.07

Boy and Toys

265Kb,31.14

162Kb,32.80

209Kb,33.94

54Kb,30.44

FFV is approaching H.264: many sequences are in between Med and Low settings in terms of PSNR and compression ratio. The table represents work in progress.

Side by Side Comparison

Original Original CPF* CPF3 H.264Low H.264Med

*) The “worst” frames (occlusion artifacts are apparent) were picked for original CPF to show CPF3 improvements.

FFV/CPF3 Features Overview

1. Continuous time – arbitrary playback speed,2. Mesh based rendering – eliminates resolution

dependence.Combination of 1 and 2 “liberates video from frames”, hence “FrameFree video” ;

3. Playback fits well modern graphics hardware – it is nothing else but rendering of textured meshes with simple pixel shader enhancements (ps_1_1),

4. CPF3 allows fallback Practical Scheme playback in case of dropped packets or CPU-bounded client.

Future work• Simplify encoding and decoding by better occlusion handling,• Coming soon:

– DirectShow filters and QT components,– CODEC,– Integration with popular players on both Mac and PC,– Parallel and GPU-based encoding,– Hardware support,– Mobile platforms support,– Lightweight decoder/player.

• Some applications:– Sports video: arbitrary speed playback and web streaming,– Security video archiving,– Integration with FrameFree Studio,– Much more.

•Thank you!

•Q&A

•Business contact: Tom Randolph tom@framefree.us

•Engineering contact: Igor Borovikov igor@framefree.us

•General information: www.framefree.com