View
221
Download
0
Category
Tags:
Preview:
Citation preview
FrameFree Video Compression
Critical Points versus Block Matching
FrameFree Technologieshttp://framefree.com
Igor Borovikov
Overview
• Motion compensation for video compression,• Critical Points Filter: brief introduction,• FrameFree Video: a bird eye view and side by
side comparison of schematics with MPEG,• Occlusions and CPF,• Side by side comparison CPF3 and H.264,• FFV features overview,• Future work.
Motion Compensation
• Block Matching Algorithm – dominant motion predictor.
• Advantages: – Conceptually simple yet effective algorithm,– Low CPU requirement on decoder side,– Motion vectors data is well suited for entropy
compression.
BMC Downsides
Inherently discrete nature of blocks leads to:• Resolution dependence
(partially addressed by adaptive block size),
• Block artifacts (addressed by deblocking filters),
• Optimal motion vectors (exhaustive search) are usually prohibitively expensive,
• No simple way to reconstruct good quality non-integer frames,
• BMC has no direct relation to Human Visual System.
Alternatives
• Optical flow and related methods. A large group of pixel matching algorithms, yet not directly suitable for motion compensation in video compression.
• Critical Points Filter. The exceptional robustness of CPF pixel matching and adaptive mesh brings CPF to practical applications.
Critical Points Filter (CPF)
• Y. Shinagawa, and T. L. Kunii, Unconstrained Automatic Image Matching Using Multi-resolutional Critical-Point Filters, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.20, no.9, pp994-1001, Sep 1998
• K. Habuka and Y. Shinagawa, Image Interpolation Using Enhanced Multi-resolutional Critical-Point Filters, International Journal of Computer Vision, vol.58, no.1, pp19–35, 2004
CPF mimics HVS
HVS can reconstruct transition between these two images
<====>So can CPF.
Motion Compensation with CPF
“I Frame” “B-Frame” “P-Frame”Original frame compressed with, say, JPEG.
CPF- Interpolated between I and P frames: information from both sides of interval is used.
Forward motion compensated and corrected frame.
Motion Compensation with CPF
BMC vs. CPF
Motion compensationwith CPF (uncorrected)
Destination image Block motion compensation,Search radius 16 (uncorrected)
Source image
CPF vs. BMC
BMC Metaphor:
- “moving around tiles”;
CPF Metaphor:
- “deforming flexible film fixed on a rectangular frame, no tearing or folding allowed”.
• Bijectivity – no “holes” left after compensation,• Adaptive mesh - resolution independence,• Continuous time - no discretization
whatsoever,• “Key features” – critical points – are matched
with high precision, thus: • Remaining artifacts are not too conspicuous for
HVS,• Residual error of motion compensation is
relatively well suited for combination of custom JPEG-like and entropy coding.
Motion Compensation with CPF
MPEG vs. FFV encoding comparison
MPEG vs. FFV encoding comparison
“Practical Scheme”
TRANS: Image Translator TRANS: Image Translator -- : : Differential Encoder & Noise FilterDifferential Encoder & Noise FilterINTERP: Interpolation Unit INTERP: Interpolation Unit ++ : Differential Decoder: Differential Decoder
Occlusions and CPF“Boy and Toys” (NHK test sequence) with ball bouncing off the floor:
CP
F3 O
riginal C
PF
“CPF3” is the latest modification of FFV Practical Scheme that addresses occlusion problem.
Compression ratio and PSNR
Sequence FFV “Practical Scheme”,
12.2005FFV “CPF3”
12.2006
H.264 Med H.264 Low
Yachting 313Kb,27.93
328Kb,30.12
301Kb,34.82
94Kb,30.71
Intersection 257Kb,30.48
206Kb,31.47
253Kb,33.91
63Kb,29.96
Horse race 653Kb,28.35
807Kb,31.13
582Kb,35.25
215Kb,32.07
Boy and Toys
265Kb,31.14
162Kb,32.80
209Kb,33.94
54Kb,30.44
FFV is approaching H.264: many sequences are in between Med and Low settings in terms of PSNR and compression ratio. The table represents work in progress.
Side by Side Comparison
Original Original CPF* CPF3 H.264Low H.264Med
*) The “worst” frames (occlusion artifacts are apparent) were picked for original CPF to show CPF3 improvements.
FFV/CPF3 Features Overview
1. Continuous time – arbitrary playback speed,2. Mesh based rendering – eliminates resolution
dependence.Combination of 1 and 2 “liberates video from frames”, hence “FrameFree video” ;
3. Playback fits well modern graphics hardware – it is nothing else but rendering of textured meshes with simple pixel shader enhancements (ps_1_1),
4. CPF3 allows fallback Practical Scheme playback in case of dropped packets or CPU-bounded client.
Future work• Simplify encoding and decoding by better occlusion handling,• Coming soon:
– DirectShow filters and QT components,– CODEC,– Integration with popular players on both Mac and PC,– Parallel and GPU-based encoding,– Hardware support,– Mobile platforms support,– Lightweight decoder/player.
• Some applications:– Sports video: arbitrary speed playback and web streaming,– Security video archiving,– Integration with FrameFree Studio,– Much more.
•Thank you!
•Q&A
•Business contact: Tom Randolph tom@framefree.us
•Engineering contact: Igor Borovikov igor@framefree.us
•General information: www.framefree.com
Recommended