Upload
leonard-mervyn-bruce
View
219
Download
1
Embed Size (px)
Citation preview
December, 20, 2006AV Compression / Alain Bouffioux
2
Agenda
About video formats
Video compression
Audio/Video synchronisation
The MPEG model
The MPEG model and its situation in a communication/storage context
December, 20, 2006AV Compression / Alain Bouffioux
3
Agenda
About video formats
Video compression
Audio/Video synchronisation
The MPEG model
The MPEG model and its situation in a communication/storage context
December, 20, 2006AV Compression / Alain Bouffioux
4
Some video formats (1)
Max. component video signal bandwidth: 6 MHz.
CCIR601 (CCIR is now ITU-R): Video sampling frequency: 13.5 MHz for 525 & 625 line standards(Shannon requirement)
Synchronous with line (& image) sampling frequencyFsampling= 864*Fh for 625 line system (50Hz countries) Fsampling= 858*Fh for 525 line system (60Hz countries)
Why synchronous? Points at the same place
RGB format
December, 20, 2006AV Compression / Alain Bouffioux
5
Some video formats (2)
YCbCr formatCb = B-Y, Cr = R-YEye is more sensitive to luminance than to chrominance (lower resolution needed for chrominance)
R ed
B lueG reen
M atrixLP F
Y
C b
C r
M odula torS ubC
+C om posite
V ideo
December, 20, 2006AV Compression / Alain Bouffioux
6
Some video formats (3)
The 4:2:2 format– Y sampling @ 13.5 MHz– C sampling @ 6.75 MHz– 8 bits per pixel– 720 active points per line– 576 lines active lines per image (2 fields) (625 lines)
and 480 active lines (525 lines) – Pixels are not square (e.g. for 480 lines, only 640 active points are needed
- VGA format)– Image size 720*576 or 720*480
The 4:2:0 format– Vertical chrominance resolution reduced by a factor 2
(average on two successive lines)
December, 20, 2006AV Compression / Alain Bouffioux
7
Some video formats (4)
SIF format (Source Intermediate Format)Half the vertical & horizontal resolution of 4:2:0For 50Hz countries:
– Luminance: 360*288– Chrominance: 180*120
CIF format (Common Intermediate Format)– Intermediate format used in videoconferencing
(communication between US & Europe)– resolution: 360*288 – Sampling frequency: 30 Hz
QCIF (Quarter CIF)– Half the vertical & horizontal resolution of CIF.
December, 20, 2006AV Compression / Alain Bouffioux
8
Agenda
About video formats
Video compression
Audio/Video synchronisation
The MPEG model
The MPEG model and its situation in a communication/storage context
December, 20, 2006AV Compression / Alain Bouffioux
9
Video compression in MPEG-1&2 (1/6)
Principles– removal of intra-picture redundancy :
Image is decomposed in 8*8 pixels sub-images.Each sub-image contains redundant information DCT transformation (in frequency domain) de-correlates the input signal.( most energy in low spatial frequencies)
– removal of inter-picture redundancy :coding of difference with an interpolated picture (moving vectors)
– high frequent spatial frequencies quantized with lower resolution than low ones(remove irrelevancy)
– Zig-zag scan and VLC (remove redundancy)
December, 20, 2006AV Compression / Alain Bouffioux
10
Video compression in MPEG-1&2 (2/6)
Result– 4:2:2 CCIR 601 resolution : 166 Mbps
(=25images/sec *576lines* 720pixels* 2(lum & chrom) *8bits) ± 3-4 Mbps (mean) in MPEG2
– 4:2:0 SIF resolution : 30 Mbps (=25 images/sec *288 lines *352pixels* 1.5(lum & chrom) *8bits) ±1.2 Mbps (CBR) in video CD (MPEG1)
December, 20, 2006AV Compression / Alain Bouffioux
11
Video compression in MPEG-1&2 (3/6)
Spatial redundancy reduction (DCT example)
158 0 -1 0 0 0 0 0 -1 -1 0 0 0 0 0 0 -1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
139 144 149 153 155 155 155 155144 151 153 156 159 156 156 156150 155 160 163 158 156 156 156159 161 162 160 160 159 159 159159 160 161 162 162 155 155 155161 161 161 161 160 157 157 157162 162 161 163 162 157 157 157162 162 161 161 163 158 158 158
158 0 -1 -1 -1 -1 EOBzig-zag scan
1260 -1 -12 -5 2 -2 -3 1 -23 -17 -6 -3 -3 0 0 -1 -11 -9 -2 2 0 -1 -1 0 -7 -2 0 1 1 0 0 0 -1 -1 1 2 0 -1 1 1 2 0 2 0 -1 1 1 -1 -1 0 0 -1 0 2 1 -1 -3 2 -4 -2 2 1 -1 0
DCT
Quantisation
December, 20, 2006AV Compression / Alain Bouffioux
12
Video compression in MPEG-1&2 (4/6)
Temporal redundancy reduction
B
5
Bi-directional prediction
I : Intra-coded pictureP: Predicted pictureB: Bi-directionally interpolated picture
4
B
Order ofpresentation
Order oftransmission
BI P
0 3
B P
1 2 6
B
Prediction
I B P B
Increase of compressionrate
0 1 2 3 4
7
B P B
5 9
B I P
8
P B B P I B
86 7 9
December, 20, 2006AV Compression / Alain Bouffioux
13
Video compression in MPEG-1&2 (5/6)
Model of a possible encoder
Buffer
Data
Regulator
Motion estimation
Motionvectorsand modes
Format conversion and picture reorder
Inputpictures
Picturestore andprediction
1/DCT
DCT VLC
1/Q
Q Multi-plex
December, 20, 2006AV Compression / Alain Bouffioux
14
Video compression in MPEG-1&2 (6/6)
MPEG1 en MPEG2 video features– MPEG1
• sequential picture• resolution : SIF format 288(240)*356*24,25 or 30 Hz
– MPEG2• sequential or interlaced• various levels : low level (SIF: 288*356), main level (CCIR601: 576 *
720), high 1440 level (HDTV: 1152*1440), high level (EQTV: 1152*1920)
• various profiles (toolboxes) : simple profile (No B picture), main profile (=MPEG1+interlaced), SNR scalable profile (allows graceful degradation (noise improvement at same resolution), spatial scalable profile (hierarchical coding : improvement at higher resolution), high profile.
December, 20, 2006AV Compression / Alain Bouffioux
15
MPEG-4 AVC or H264
Defined about 10 years after MPEG-2
Similar structure used as for MPEG-2, but with some improvements
And two additional functional blocks:– Intra-frame prediction: exploit the redundancy between neighboring blocks within a
frame. A block is predicted using previously decoded neighboring blocks.– Deblocking filter: filter applied at the block edges (except picture boundaries) in order to
improved perceptual quality
Cost/benefits– Higher complexity– Compression efficiency: factor 2
More than compressionNetwork adaptation layer defining packets adapted for each transport network
Uses – HD-DVD (MPEG2 & H264) & HD-camcorder– Video services over IP networks – Mobile television
December, 20, 2006AV Compression / Alain Bouffioux
16
Agenda
About video formats
Video compression
Audio/Video synchronisation
The MPEG model
The MPEG model and its situation in a communication/storage context
December, 20, 2006AV Compression / Alain Bouffioux
17
Synchronisation
Synchronisation in the multimedia context
refers to the mechanism that ensures a temporal
consistent presentation of the audio-visual
information to the user
December, 20, 2006AV Compression / Alain Bouffioux
18
Intramedia synchronisation
T between capture & presentation = Constant Same clock frequency & Data on time Need for corresponding tools
Network
T1T2 =
T1
Capture time
Audio signal Encoder
T = Constant
Decoder
Presentation time
Audio signal
December, 20, 2006AV Compression / Alain Bouffioux
19
Intermedia synchronisation
T_Audio = T_Video Sampled at the same time Presented at the same time) Possible tools : common time base and presentation control (media
synchronisation with the common time base)
Ex.: Lip_sync (requirement: |delay_difference| < 80msec)
Network
T_audio = Constant
T_video = Constant = Capture time
Video signal
Audio signal
Encoder
Capture time
T_Audio Presentation time
Decoder
Video signal
Audio signal
Presentation time
December, 20, 2006AV Compression / Alain Bouffioux
20
Recovery of clock in CBRCBR = Constant Bit Rate
if the clock to recover is synchronous with transport clock Recovery of clock but not of common time base
Remark : possibility to slave stream from DSM (Digital Storage Media) to local clock
Filter
CBR stream
Phaseerror
Filling level
50%
Time Information carriedby each sample
VCO
Processing
Recoveredclock
time
December, 20, 2006AV Compression / Alain Bouffioux
21
Recovery of clock and time base in VBRVBR = Variable Bit Rate
Need for insertion of time stamps (OUTPUT TIME)Output time stamp says for example : “It is now 16h25”Receiver adjusts its own horloge to the received time stamp
Recovery of clock & of common time baseTime information carriedonly by time stamps
Time stamp extraction
Time counter
Recovery of clock & time
First time stamp
Others
Clock
Data stream
Data stream
Time stamps
Counter
Recovered clock
Error Filter VCO
VBR Stream
Counter sample(=Time stamps)
Channel
Time stamps
December, 20, 2006AV Compression / Alain Bouffioux
22
Synchronisation with common time base
Insertion of time stamp (=INPUT TIME)Input time stamp says : “Input has been sampled at 16h29”.Receiver presents the sample at (its input time stamp + maximum encoding and decoding delay).Alternative: transmission of presentation time stamp (input time+delay)
Buffering
Mediaoutput
Media input
Time clock(Recovered)
Comparison of time clockwith sampled time clock
Sample "Time clock"Assemble frame
Time clock
Timestamp
Processing
Processing
Channel
December, 20, 2006AV Compression / Alain Bouffioux
23
Getting data on time
“On time” Not too late, not too earlyNo buffer over- or underflow
Flow control : not applicable in broadcasting
Common time base and Definition of a standard target decoder that describes the data consumption pattern of the receiver.
Remark: Direct MPEG (Microsoft) does not use time information for clock recovery but relies on flow control
December, 20, 2006AV Compression / Alain Bouffioux
24
Streams
Idea of continuity (pipelining)
Carry time information for clock recovery
No flow control (allows broadcasting)The emitter must have a precise knowledge of the receiver data consumption pattern (explicit in MPEG STD)
Just-in-timeShorter delay and smaller buffer size than with flow control
Two aspects in synchronisation :Clock recovery & timing control (model & buffering)
December, 20, 2006AV Compression / Alain Bouffioux
25
Requirement on for stream transport
Data information BER (Bit Error Rate) requirementNo repetition of frame possible FEC (Forward Error Correction)
Time information No jitter
December, 20, 2006AV Compression / Alain Bouffioux
26
Agenda
About video formats
Video compression
Audio/Video synchronisation
The MPEG model
The MPEG model and its situation in a communication/storage context
December, 20, 2006AV Compression / Alain Bouffioux
27
What is MPEG ? (1/2)
Moving Picture Expert Group
International standard (ISO/IEC) Interoperability & economy of scale
Various standard developed/under developmentNot only audio/video compression
MPEG-1/2/4 - Compression of audio and video and multiplexing in a single stream
Definition of the interface not of the codecs room for improvement
MPEG-1 : until 1.5 Mbps, for DSMProgressive picture, stereo (Dolby surround)
December, 20, 2006AV Compression / Alain Bouffioux
28
What is MPEG ? (2/2)
MPEG-2 : Various bit rates (CBR & VBR)Program stream for DSM, transport stream for networkInterlaced picture, 5.1 audio channels Definition of various video levels (e.g. CCIR601 resolution: 4-9 Mbps, HDTV:15-25 Mbps) and profiles
MPEG-3 : Cancelled, integrated in MPEG-2(Initially : for HDTV)
MPEG-4 : standard for audio, video and graphics in interactive 2D and 3D multimedia communication. (Initially : low bit rate for real-time personal communication)
MPEG-7 : Multimedia contents description interface
MPEG-21 : Focus on multimedia distribution and on DRM aspects.
December, 20, 2006AV Compression / Alain Bouffioux
29
The MPEG model (1/2)
Audiodecoder
Audio signal
Videosignal
Presented signals
Multiplexer
Videodecoder
Captured signals
Audioencoder
Videoencoder
Audio signal
Videosignal Digital storage medium
orNetwork
Transmission channel
Demulti-plexer
December, 20, 2006AV Compression / Alain Bouffioux
30
The MPEG model (2/2)
Compression of audio & video and multiplexing in a single stream
Guarantees intramedia and intermedia synchronisation.
MPEG defines an interface– bitstream syntax– timing of the bitstream STD specifying timing requirement (ideal
model)
Consequences:– Decoder should compensate deviations from STD– Network should correct jitter introduced by the channel (RTD-LJ)
MPEG stream must be adapted to transmission channel formatting, error correction, channel coding (b.v.video-CD)
December, 20, 2006AV Compression / Alain Bouffioux
31
Components of the MPEG standard
The MPEG standard is composed of 3 main parts :– Audio : Specifies the compression of audio signals– Video : Specifies the compression of video signals– System : specifies how the compressed audio and video signals are
combined in the multiplexed stream (program stream or transport stream).
Each part specifies :– The bitstream syntax – The timing requirement and the related information (bit rate, buffer
needs)
December, 20, 2006AV Compression / Alain Bouffioux
32
Synchronisation Mechanism (1/2)
System decoderanddemultiplexer
Multiplexer and systemencoder
Audiodecoder
Videodecoder
ComparisonPTS and STCand presentation
Videooutput
ComparisonPTS and STCand presentation
Audiooutput
Extractionof PCR (SCR)
STC
Audioencoder
Videoencoder
Assemble pictures,Sample STC for PTS
Videoinput
Assembleaudio frames,Sample STCfor PTS
Audioinput
Sample STCfor PCR(SCR)
STC
Transmission channel
December, 20, 2006AV Compression / Alain Bouffioux
33
Synchronisation Mechanism (2/2)
PCR for TS & SCR for PS (but same concept)
Clock & time base recovery: Time-stamping at OUTPUT (PCR included in TS multiplex, SCR in pack header)
Audio & video clock locked to STC easy recovery (see next slide)
Synchronisation of audio & video to common time base (Time stamping at Input)
STD is defined (because of the absence of flow control)streams are such that STD buffers never over- or underflow
In TS, many program in a single stream but unique clock per program.
Time information “No Jitter” requirement for transport
December, 20, 2006AV Compression / Alain Bouffioux
34
Clock recovery in receiver
VCO Audio clockdivider
Video clockdivider Video
clock
Audioclock
Audiooutput
STC
ComparisonPTS and STCandPresentationDecoded
audio
PTS
STC(Counter)
PCR
STC
Error Low Pass Filter(Integrator)
Load first PCR
December, 20, 2006AV Compression / Alain Bouffioux
35
MPEG-2 program & transport streams
Program streams:– Relatively error free environment– program stream packet may have variable and great length– Single time base
Transport streams:– environment where errors are likely– many programs (independent time base)– Transport stream packet : fixed, 188 bytes– Contains tables
December, 20, 2006AV Compression / Alain Bouffioux
36
Channelencode
Channeldecode
Bit-stream
Sourcedecode
Informationsink
Digitaloutput
Format Decrypt
SourceencodeFormat
Informationsource
Digitalinput
Encrypt
Synchro-nisation
Digitalwaveform
Otherdestination
Demodu-late
Demulti-plex
Multipleaccess
Modu-late
Multi-plex
Channelbits
Othersources
Multipleaccess
Channel
MPEG in a communication context (1)
“Typical” communication system
December, 20, 2006AV Compression / Alain Bouffioux
37
MPEG in a communication context (2)
MPEG : Source coding only (bit rate reduction) + multiplexing
The MPEG stream must be adapted to the channel in what concern its physical characteristics and in order to get the required QoS (Quality of Service) & Security
– Encryption – Channel coding (forward error correction, interleaving, modulation codes)– multiplexing & formatting– modulation (frequency allocation)– multiple access method
Some channels : CD/DVD - satellite - cable - ATM - 1394
December, 20, 2006AV Compression / Alain Bouffioux
38
VideoEncoder
MPEG2 compression layer
Audioencoder
Audio,videosources
ES(ElementaryStream)
Adap-tationto thechannel
PS(1 pro-gram)
MPEG2 system layer
PSMulti-plexing
Adap-tationto thechannel
DVB, DVD ...
Disc
Satellite
TSMulti-plexing
TS(n pro-grams)
Adap-tationto thechannel Cable
TS (Transport Stream)orPS (Program Stream)
MPEG-2 in a communication context (3) A simple view of MPEG-2 in the communication context