Upload
eileen-malone
View
222
Download
2
Tags:
Embed Size (px)
Citation preview
Video Coding
Past, Present & Future
Georgios Diamantopoulos
Digital Systems and Vision Processing Group
Today’s agenda
History of video compression standards State-of-the-art Showcase Relative performance analysis
Advanced video coding technology Motion Compensation demo
Image coding with JPEG2000 Features, how it works, demo
Future of video coders Video Quality Evaluation
Relation to Image Quality and overview of Metrics Getting started with video coding
About me
Graduated with BEng in Computer Systems Engineering in 2004
Started PhD the same year: Improve video compression Link psychology of vision with engineering Investigating how to re-arrange bits such that
perceptual quality is increased
Personal research interests: Video Quality Evaluation [Video Quality Studio]
How to use vision models to evaluate video automatically
The history
The early days (1950-1990)
First picture phone systems in the ’50s (B&W) First standardized effort (1982):
International Telecommunications Union-Telecommunication (ITU-T) H.120: a Differential Pulse Code Modulation (DPCM) codec
Target bitrate: 2Mbit/s for Europe, 1.544 for North America Poor temporal quality
The early days (1950-1990)
Need for less than 1 bit per pixel Solution: Block based coding
Discrete Cosine Transform versus Vector Quantization DCT won because of progressive image transmission JPEG 1984-1988
H.261: Video conferencing ITU-T, completed in 1990 Combination of DPCM and DCT Motion Compensation P x 64kbps
Familiar things (1990-1995)
MPEG-1: “Coding of moving pictures and associated audio for digital storage media” (1992) Target was VHS Quality at 1.5MBits/s Basis of Video-CD MP3 is still with us! (MPEG-1 Layer 3)
Familiar things (1995-2000)
MPEG-2: “Generic coding of Moving Pictures and Associated Audio”
Broadcasting and storage Bitrates: 4-9 MBits/s Satellite TV, DVD
MPEG-3? Aimed to do High Definition TV
(HDTV) MPEG-2 could do that anyway Folded into MPEG-2
MPEG-4: “Coding of audio-visual objects” Started as very low-bitrate project Turned out to be much more:
Coding of media objects 64kbps to 240Mbps (Part 10/H.264) Synthetic/Semi-synthetic objects XMT: Like HTML, but to build videos First standard with Intellectual Property
Management
Present & Future (2000-2010)
MPEG-4 Part 10: Advanced Video Coding / H.264 Designed by a Joint MPEG and
ITU-T group Claims 50% bitrate savings to
MPEG-2, 30% over MPEG-4!
H.265 aims to have 50% better compression
MPEG-7: “Multimedia Content Description Interface” (2001) Describing audio/video
Applications Indexing of video databases Search & Retrieval Browsing
Independent players
Efforts from independent companies in video compression
Main competitors: RealNetworks – http://www.real.com Microsoft Windows Media (WM) – http://www.microsoft.com Apple Quicktime – http://www.apple.com On2 – http://www.on2.com
Standards have wider acceptance Recently Microsoft wrapped their WM technology
Standard: VC-1 An attempt to compete with MPEG/ITU… succeeded
The next DVD format (Blu-Ray) will support H.264 and VC-1
Where is MPEG used? Most probably.
MPEG-1 Video-CD Usually .mpg or .mpeg files are MPEG-1 DAB Digital Radio is MP2 (MPEG-1 Layer 2) MP3 files (MPEG-1 Layer 3)
MPEG-2: .vob, .m2v, rarely .mpg files Anything to do with DVD
Camcorders, DVD players, DVD recorders, TiVo Digital TV
MPEG-4: High Quality AVI files Video Phones DivX Some advanced audio players support MPEG-4 Advanced Audio Coding (AAC)
NetMeeting and similar video-chat H.263/+/++
H.264 Some content has appeared recently, mainly trailers
Even if you haven’t…
MPEG-1 VideoCD-like (352 x 240/288 @ 1200 kbps)
MPEG-2 DVD-like (720 x 576 @ 6500-7000 kbps)
MPEG-4 High Quality (720 x 576 @ 1000-1800 kbps)
HDTV (1920 x 1080) H.264 Windows Video 9
* Material encoded from H.264/WMV9 original with TMPGEnc (MPEG-1/2) and XviD (MPEG-4).
R-D Performance of MPEG Codecs
32
34
36
38
40
42
44
46
48
50
350 450 550 650 750 850 950 1050
Bit rate (kbps)
PS
NR
(Y
)
MPEG-1 MPEG-2 MPEG-4 H.264
Mission Impossible 3 HDTV trailer (1080p) resized to 720x304. Encoders: TMPGEnc (MPEG-1/2), XviD (MPEG-4), x264 (H.264)
CODEC Design
Reproduced from “H.264 and MPEG-4 Video Compression” by I. E. G. Richardson.
Motion Compensation
It aims to reduce the data transmitted by detecting the motion of objects Use the previous as reference In steps:
Split the current frame in blocks. For each one: Find the best-matching block in the reference frame
How? We search an area in the reference frame and compare, using Sum of Absolute Differences (SAD) Mean SAD (MSAD)
The best matching block is coded and transmitted Next frame can be used a reference too
Demo
Advanced Video Coding
Bidirectional frames When performing motion estimation, each block can
be assigned a match from either a backward or forward reference
Global Motion Compensation A set of vectors that describe the whole object plane
can be transmitted
Advanced Video Coding
Other tools Quarter-pixel motion compensation precision
Bigger vectors but better prediction (less error) Spatial prediction (intra-level)
Prediction of pixels spatially Motion vectors off-picture 4x4 integer transform
Faster, reversible, exploits spatial correlation better Deblocking filter
Executed on the encoder, reduces blocking Multiple reference frames
JPEG2000 – What’s new
Resolution and quality scalability Encode once, decode at a resolution and quality
appropriate to the client e.g. Mobile phone: low resolution/quality High performance PC: high resolution/quality
Random access Any part of the stream can be decoded independently
Lossless coding is possible Fast, highly efficient in terms of compression Data is organized in layers with different
capabilities
JPEG2000 – How it works Component transform is
RGB to YUV Wavelet transform gives a
multi-resolution representation of the image
Quantization is optional
Bytes are organized in bit planes
Post-compression R-D optimisation “truncates” the planes to achieve optimal distortion for the specified rate
JPEG2000 - Showcase
NASA Image 8688 x 8296 pixels RGB Data = 206MBytes
JPEG2000 Tiles 200x300 Wavelet decomposition/Resolution levels = 8 6 Quality layers = 0.01, 0.05, 0.1, 0.25, 1.0, lossless Output size = 101MBytes
Demo
The future standard is…
MPEG-21: JPEG2000 for videoScalability
Resolution Quality But also… Temporal
In other words, from one stream, you can get streams with variable resolution, quality and frame-rate
In the same fashion we saw with JPEG2000
The future product is…
High Definition DVD1080 pixels vertical resolution, 60 HzBlu-Ray discs
25/50GB of storage 4 hours of HD video
H.264 and Windows Media Video 9First prototype set-top boxes have appeared
Video Quality Evaluation
Similarities to Image Quality Evaluation Eye more sensitive to chrominance Some areas attract more attention that others Blocking/ringing artifacts are annoying …
Differences Temporal dimension
Moving versus stationary objects Frame rate plays a significant role
Any less than 24 Hz is not perceived as moving images but as slideshow
Video Quality Evaluation
SubjectiveA human “subject” rates the video on a scale
Double Stimulus Continuous Quality Scale Method Hidden scale of 0-100 Difference is calculated as the
actual rating
Video Quality Evaluation
ObjectiveA computer algorithm judges the distortion
between videosAttempts to model a human observerThere is currently no standard method
Objective Metrics
Peak Signal-To-Noise Ratio (PSNR)Used widely in evaluating coding performancePurely mathematical difference
Can be tricked quite easily
Objective Metrics
How to trick PSNRTake a natural imageGive more bits to areas you look at moreGive less bits to areas you look at lessSubjective rating will be high, PSNR low
Better metrics Not perfect
Video Quality Experts Group
Evaluation of objective metrics
Video Quality Studio
Getting started - Programming
Learn to read and write RAW images Process them somehow
You can move on to YUV video sequences Get visual
Win32 API, Windows Forms (.NET) Processing video files
AVI files – Video For Windows API Tons of open source MPEG/H.xxx libraries (libmpeg2, libavcodec, MPEG4IP,
XviD etc.) Advanced editing
DirectShow API Very complex, get a book
Languages C/C++ is the de-facto standard C++/CLI (Managed C++) if you want to access .NET libraries VB: Definitely a no-no, slow and cumbersome C#: Better than VB but still not for performance coding
DirectShow
Getting started - Tools
Jasc Paint Shop Pro Now owned by Corel GIMP is a free alternative
IrfanView Excellent & free image viewer with saving capabilities Supports JPEG2000!
Virtualdub Free video processing toolkit Book: VirtualDub Video: Capture, Processing and Encoding
Media Player Classic Lightweight, open source media player Replica of Windows Media Player 6.4 A LOT of options
TMPGEnc Free MPEG-1/2 encoder
Getting started – Codecs/Content Mostly open-source
XviD Ogg Theora (wavelets) BBC Dirac (wavelets) On2 VP3 x264 OpenJPEG Jasper
Commercial, free versions DivX
Where to get content Extract them from your DVDs – be careful who you show it to YUV Sequences
http://www.cipr.rpi.edu/resource/sequences/ ftp://ftp.crc.ca/crc/vqeg/TestSequences/
High Definition Apple trailers: http://www.apple.com/trailers/ Microsoft showcase:
http://www.microsoft.com/windows/windowsmedia/content_provider/film/ContentShowcase.aspx
Summary
Historic development of video coding Who makes codecs/standards Examined samples of each generation and their relative performance
Codec design Motion estimation demo Advanced video coding
Wavelet coding: JPEG2000 Features Technology Demo
What the future holds Video Quality Evaluation
Objective/Subjective Evaluation/Metrics Video Quality Studio
Video coding how-to
Get in touch
If you like programming and want to get involved with cool real-world projects, get in touch
I am willing to give more technical/practical seminars if there is interest so let me know
Any Questions?
Contact information
Web: http://postgrad.eee.bham.ac.uk/gxd186/
E-mail: [email protected]
Bibliography
Video Quality Studio: http://www.visumalchemia.com/vqstudio/
History: http://www.pha.jhu.edu/~sundar/intermediate/history.html http://myhome.hanafos.com/~soonjp/vchx.html
F. Pereira, T. Ebrahimi. The MPEG-4 Book. I. Richardson. H.264 and MPEG-4 Video Compression. MPEG: http://www.chiariglione.org/mpeg/ VQEG: http://www.vqeg.org Video For Windows:
http://msdn.microsoft.com/library/en-us/multimed/htm/_win32_video_for_windows.asp
DirectShow: http://msdn.microsoft.com/directx/sdk/