View
8
Download
0
Category
Preview:
Citation preview
Quantization
C.M. LiuPerceptual Signal Processing Lab College of Computer ScienceNational Chiao-Tung University
Office: EC538(03)5731877
cmliu@cs.nctu.edu.tw
http://www.csie.nctu.edu.tw/~cmliu/Courses/Compression/
Contents
Quantization Problem
Uniform Quantization
Adaptive Quantization
Nonuniform Quantization
Entropy-Coded Quantization
Vector Quantization
Rate-Distortion Function
2
Quantization
Definition:
The process of representing a large—possibly infinite—set of values with a smaller set.
Example:Source:
Real numbers in the [-10.0, 10.0]
Quantization
Q(x) = ⎣x+0.5⎦[-10.0, -10.0] {-10, -9, …, -1, 0, 1, 2, …, 9, 10}
Scalar vs. vector quantizationScalar: applied to scalars
Vector: applied to vectors
3
The Quantization Process
Two aspectsEncoder mapping
Map a range of values to a codewordIf source is analog A/D converterKnowledge of the source can help pick more appropriate ranges
Decoder mappingMap the codeword to a value in the rangeIf output is analog D/A converterKnowledge of the source distribution can help pick better approximations
Quantizer = encoder + decoder
4
Quantization Example 5
Quantizer Input-Output Map6
Quantization Problem Formulation
Input:X – random variablefX(x) – probability density function (pdf)
Output:{bi}i=0..M decision boundaries
{yi}i=1..M reconstruction levels
Discrete processes are often approximated by continuous distributions
E.g.: Laplacian model of pixel differenceIf source is unbounded, then first/last decision boundaries = ±∞
7
Quantization Error
Mean squared quantization error
8
( )
( ) dxfyx
dxfxQx
X
M
i
b
b i
Xq
i
i∑∫
∫
=
∞
∞−
−
−=
−=
1
2
22
1
)(σ
iii bxbyxQ ≤<= −1iff)(
Quantization error is a.k.a.Quantization noiseQuantizer distortion
Quantization Problem Formulation (2)
Bit rates w/ fixed-length codewordsR = ⎡log2M⎤E.g.: M = 8 R = 3
Quantizer design problemGiven:
input pdf fX(x) & MFind:
decision boundaries {bi} and Reconstruction levels {yi}
Such that: MSQE is minimized
9
Quantization Problem Formulation (3)
Bit rates w/ variable-length codewordsR depends on boundary selectionExample
10
∑ ==
M
i ii yPlR1
)(
∫−
=i
i
b
b Xi dxxfyP1
)()(
∑ ∫=−
=M
i
b
b Xii
i
dxxflR1
1
)(
Quantization Problem Formulation (4)
Rate-optimization formulation
Given:
Distortion constraint σq2 ≤ D*
Find:{bi}, {yi}, binary codes
Such that:R is minimized
Distortion-optimization formulation
Given:Rate constraint R ≤ R*
Find:{bi}, {yi}, binary codes
Such that:
σq2 is minimized
11
Uniform Quantizer
All intervals of the same sizei.e., boundaries are evenly spaced (Δ)Outer intervals may be an exception
ReconstructionUsually the midpoint is selected
Midrise quantizerZero is not an output level
Midtread quantizerZero is an output level
12
Midrise vs. Midtread Quantizer13
Midrise Midtread
Uniform Quantization of Uniform Source
InputUniform [-Xmax, Xmax]
OutputM-level uniform quantizer
14
MX max2
=Δ
1221
2122
22
1)1(
max
22 Δ
=⎟⎠⎞
⎜⎝⎛ Δ
−−= ∑∫
=
Δ
Δ−
M
i
i
iq dxX
ixσ
Alternative MSQE Derivation
Consider quantization error instead:q = x – Q(x)
q ∈ [-Δ/2, Δ/2]
15
∫Δ
Δ−
Δ=
Δ=
2
2
222
121 dqqqσ
( ) ( )
dB02.6)2(log20)(log10
212
122log1012
122log10log10)dB(SNR
102
10
2max
2max
102
2max
102
2
10
nM
MX
XX
n
q
s
===
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
⎟⎠⎞
⎜⎝⎛
=⎟⎟⎠
⎞⎜⎜⎝
⎛Δ
=⎟⎟⎠
⎞⎜⎜⎝
⎛=
σσ
Examples (8 1, 2, 3 bits/pixel)16
Darkening, contouring & dithering
Uniform Quantization of Nonuniform Sources
Example nonuniform source:x ∈ [-100, 100], P(x ∈ [-1, 1]) = 0.95
ProblemDesign an 8-level quantizer
The naïve approach leads to95% of samples values represented by two numbers:
-12.5 and 12.5
Max quantization error (QE)= 12.5
Min QE = 11.5 (!)
Consider an alternativeStep = 0.3
Max QE = 98.5, however 95% of the time QE < 0.15
17
Optimizing MSQE18
Numerically solvable for specific PDF
Example Optimum Step Sizes19
QE for 3-bit Midrise Quantizer20
Overload/Granular Regions21
"4"4,stdev
valuegranularmax11 loadingff σ→==
The step selectionTradeoff between overload noise and granular noise.
Variance Mismatch Effects22
Variance Mismatch Effects (2)23
Distribution Mismatch Effects
8-level quantizers, SNR
24
Adaptive Quantization
IdeaInstead of a static scheme, adapt to the actual data:
Mean, variance, pdf
Forward adaptive (off-line)Divide source in blocksAnalyze block statisticsSet quantization scheme
Side channel
Backward adaptive (on-line)Adaptation based on quantizer outputNo side channel necessary
25
Forward Adaptive Quantization (FAQ)
Choosing block sizeToo large
Not enough resolutionIncreased latency
Too smallMore side channel information
Assuming a mean of zeroVariance estimate:
26
∑ −
= +=1
022 1ˆ N
i inq xN
σ
Speech Quantization Example
16-bit speech 3-bit fixed
27
Speech Quantization Example (2)
16-bit speech 3-bit FAQBlock = 128 samples
8-bit variance quantization
28
FAQ Refinement
So far we assumed uniform pdf
RefinementAssume uniform pdf butRecord min/max values for each block
Example:Sena image8x8 blocks3-bit quantizationOverhead = 16/8x8 = 0.25 bits/pixel
29
FAQ Refinement Example30
Original: 8 bits/pixel Quantized: 3.25 bits/pixel
Backward Adaptive Quantization (BAQ)
ObservationOnly encoder sees inputAdaptation can only be based on quantized output
ProblemHow do we deduce mismatch information from output only?It is possible, if we know the pdf …… and we are very patient
31
Jayant Quantizer
IdeaIf input falls in the outer levels
Expand step size
If input falls in the inner levelsContract step size
The product of expansions & contraction should be 1
Multipliers: Mk
If input Sn-1 falls in the kth interval, thenstep is multiplied by Mk
Inner Mk < 1, outer Mk > 1
32
1)1( −− Δ=Δ nnln M
3-bit Jayant Quantizer Output Levels33
Jayant Example
M0 = M4 = 0.8, M1 = M5 = 0.9
M2 = M6 = 1.0, M3 = M7 = 1.2, Δ0 = 0.5
Input: 0.1 -0.2 0.2 0.1 -0.3 0.1 0.2 0.5 0.9 1.5
34
Picking Jayant Multipliers
Δmin / Δmax to prevent under/overflow.
Adaption speed affected by γ.
35
⎣ ⎦
∑∏
∏
∏
∏
==
=
=
=
=⇒=
=>=
==
=
=
M
kkk
M
k
Pl
kkl
k
M
k
kk
P
M
k
Nn
M
k
n
Pl
llMLetNnPM
M
M
kk
k
k
k
k
k
k
k
00
0
0
0
01
,1where,
where,1
1
1
γ
γγ
Jayant Example36
“Ringing”
Jayant Performance37
Expands more rapidly than contracts to avoid overload errors.Robustness over changing input statistics.
Non-uniform Quantization
Idea: Pick the boundaries such that error is minimized
i.e., smaller/bigger step for smaller/bigger values
e.g.:
38
Non-uniform Quantization-- pdf-optimized Quantization
Problem: Given fX, minimize MSQE:
39
( ) dxfyx X
M
i
b
b iqi
i∑∫
= −
−=1
22
1
σ
Set derivative w.r.t. yj to zero and solve for yj:
( )
( )dxxf
dxxfxy
j
j
j
j
b
b X
b
b X
j
∫∫
−
−=
1
1
Set derivative w.r.t. bj to zero and solve for bj:( ) 21 jjj yyb += +
Non-uniform Quantization-- Lloyd-Max Algorithm
Observation:Circular dependency b/w bj and yj
Lloyd/Max/Lukaszewics/Steinhaus approach:Solve the two iteratively until an acceptable solution is found
Example:
40
Non-uniform Quantization-- Lloyd-Max Algorithm (2)
Boundaries: { b1, b2, …, bM/2-1}b0 = 0, bM/2-1 = MAX_INPUT
Reconstruction levels: { y1, y2, …, yM/2-1}
41
( ) ( )dxxfdxxfxyb
b X
b
b X ∫∫=1
0
1
01
One equation, two unknowns: b1, y1Pick a value for b1 (e.g. b1 = 1), solve for y1 and continue:
112 2 yby += ( ) ( )dxxfdxxfxyb
b X
b
b X ∫∫=2
1
2
12
… and so on until all { bn} and { ym} are found
Non-uniform Quantization-- Lloyd-Max Algorithm (3)Terminating condition:
42
12122 2ˆ −− += MMM yby
( ) ( )dxxfdxxfxy M
M
M
M
b
b X
b
b XM ∫∫−−
=2/
12/
2/
12/2
ε≤− 22 ˆMM yy
Else: pick a different b1 & repeat
Non-uniform Quantization— Example: pdf-Optimized Quantizers
43
Significant improvement over the uniform quantizer.
Lloyd-Max Quantizer Properties
1. MeanOUTPUT = MeanINPUT
2. VarianceOUTPUT ≤ VarianceINPUT
3. MSQE:
44
[ ]jj
M
jjxq bxbPy ≤≤−= −
=∑ 1
1
222 σσ
4. If N is a random variable representing QE
[ ] 2qXNE σ−=
5. Quantizer output and QE are orthogonal (uncorrelated)
[ ] 0,,,|)( 10 =MbbbNXQE K
Mismatch Effects45
4-bit Laplacian pdf-optimized quantizer
Companded Quantization (CQ)46
Compressor Uniform quantizer Expander
CQ Example: Compressor47
CQ Example: Uniform Quantizer48
CQ Example: Expander49
CQ Example: Equivalent Non-uniform Quantizer50
Vector Quantization
Definition
in alues vector v) stic(determini L of one only takemay it in that
on distributi special a has y vector e vector thrandom ldimensiona -N : ,
variablesrandom real : 1 , )( , )(
)]( .......... )2( )1([ )]( .......... )2( )1([
NR
yx
Niiyix
NyyyyNxxxx
≤≤
==
Vector Quantization (c.1)
Vector quantization
the vector quantization of x may be viewed as a pattern recognition problem involving the classification of the outcomes of the random variable x into a discrete number of categories or cell in N-space in a way that optimizes some fidelity criterion, such as mean square distortion.
y Q x= ( )
Vector Quantization (c.2)
VQ Distortion
VQ Optimizationminimize the average distortion D.
D P x C E d x y x C
d x yR l l l
k k kk
L
kN
= ∈ ∈=
∞
∑ ( ) { ( , )| }
( , )1
1 2
are typically the distance measures in , including , , norm
Vector Quantization (c.3)
Two conditions for optimalityNearest Neighbor Selection
minimize average distortion
=> applied to partition the N-dimensional space into cell when the joint pdf is known.
.1,for ),( ),( , )(
LjjkyxdyxdiffCxyxQ
jk
kk
≤≤≠≤∈=
NnxCxy
ky
ky
k
ddfyxd
CxyxdEDy
kξξξξ .....)....(),(....=
}|),({
11minarg
minargminarg
∈∫∫
∈===
{ , }C k Lk 1≤ ≤ fx( )•
55
Appendix C: Rate-Distortion Functions
Introduction
Rate-Distortion Function for a Gaussian Source
Rate-Distortion Bounds
Distortion Measure Methods
1. Introduction
Considering questionGiven a source-user pair and a channel, under what conditions is it possible to design a communication system that reproduces the source output for the user with an average distortion that does not exceed some specified upper limit D?
The capacity (C) of a communication channel. The rate distortion function ( R(D) )of a source-user pair.
Rate-distortion function R(D)A communication system can be designed that achieves fidelity D if and only if the capacity of the channel that connects the source to user exceeds R(D).
The lower limit for data compression to achieve a certain fidelity subject to a predetermined distortion measure D.
1. Introduction (cont.)
Equations representations :D is to r t io n D :
M u tu a l in fo rm a tio n :
R a te d is to r t io n fu n c t io n R (D ) :
d is to r t io n m e a su re fo r th e s o u rc e w o rd
= (
D d q p x q y x x y d x d y
I q p x q y xq y x
q yd x d y
R D I q Q q y x d q D
x y
x x
q Qd
n
D
= =
=
= = =
∫ ∫
∫ ∫
∈
( ) ( ) ( | ) ( , )
( ) ( ) ( | ) lo g( | )
( )
( ) in f ( ) , { ( | ) : ( ) }
( , ) :
, . . . , )
ρ
ρx 1 r e p ro d u c e d a s = (y
T h e fa m ily is c a l le d th e
s in g le - le t te r f id e li ty c r i te r io n g e n e ra te d b y .
1
n
y
x ,y
,. . . , )
( ) ( , )
{ , }
y
n x y
F n
n
t t
t
n
n
ρ ρ
ρ
ρρ
=
= ≤ < ∞
−
=∑1
1
1
2. Rate-Distortion Bounds
Introduction
Rate-Distortion Function for A Gaussian Source R(D) for a memoryless Gaussian sourceSource coding with a distortion measure
Rate-Distortion Bounds
Conclusions
3. Rate-Distortion Function for A Gaussian Source
Rate-Distortion for a memoryless Gaussian source
The minimum information rate (bpn) necessary to represent the output of a discrete-time, continuous-amplitude, memorylessstationary Gaussian source based on an MSE distortion measure per symbol.
R DD D
Dg
x x
x
( )log ( ),
,=
≤ ≤
≥
⎧⎨⎪
⎩⎪
12 2
2 2
20
σ σ
σ
0
3. Rate-Distortion Function for A Gaussian Source (c.1)
Source coding with a distortion measure (Shannon, 1959)
There exists a coding scheme that maps the source output into codewords such that for any given distortion D, the minimum rate R(D) bpn is sufficient to reconstruct the source output with an average distortion that is arbitrarily close to D.
Transform the R(D) to distortion-rate function D(R)
21010
22
log106)(log10dBin Express
2)(
xg
xR
g
RRD
RD
σ
σ
+−=
= −
Comparison between different quantizations
3. Rate-Distortion Function for A Gaussian Source (c.2)
4. Rate-Distortion Bounds
Source:Memoryless, continuous-amplitude source with zero mean and finite variance with respect to the MSE distortion measure.
Upper boundAccording to the theorem of Berger (1971), it implies that the Gaussian source requires the maximum rate among all other sources for a specified level of mean square distortion.
R DD
R D D
D R D R
xg x
gR
x
( ) log ( ),
( ) ( )
≤ = ≤ ≤
≤ = −
12
0
2
2
22
2 2
σσ
σ
σx2
Lower bound (Shannon lower bound)
where
4. Rate-Distortion Bounds (c.1)
)(2*
2*
22
1)(
2log21)()(
xHR
eRD
eDxHDR
−−=
−=
π
π
DeDeDR
exH
exf
xx
xg
x
xx
x
2
222
2*
22
2
log212log
212log
21)(
2log21)(
21)(
:sourceGaussian For 22
σπσπ
σπ
σπσ
=−=⇒
=
= −
∫∞
∞−−= ξξξ dffxH
xH
nxnx
def)(log)()(
entropy aldifferenti :)(
)(2)(
)()()(
)()()(*
*
RDRDRD
DRDRDR
g
g
≤≤
≤≤
For Gaussian source, the rate-distortion, upper bound and lower bound are all identical to each other.The bound of differential entropy
4. Rate-Distortion Bounds (c.2)
)(by boundedupper isentropy aldifferenti The
)]()([6
)]()([6)()(
log10
)]()([66)(log10
*
*10
*10
xH
DRDR
xHxHRDRD
xHxHRRD
g
g
gg
g
⇒
−=
−=
−−−=
Rate-distortion R(D) to channel capacity CFor C > Rg(D)
The fidelity (D) can be achieved.
For R(D)<C< Rg(D) Achieve fidelity for stationary sourceMay not achieve fidelity for random source
For C<R(D)Can not be sure to achieve fidelity
4. Rate-Distortion Bounds (c.3)
4. Rate-Distortion Bounds (c.4)
5. Distortion Measure Methods
{ }GG m r k c x m
k rcr
c rr
rrrrr
r( , , ) exp ( )
( / )( / )
( / ):
σ
σ
= − −
= =
= →= →= →= ∞ →
and
For different Laplacian pdf Gaussian pdf constant pdf uniform pdf
2 13
1
120
2ΓΓ
Γ
Problems
Homeworks (Sayood 3rd, pp. 270-271)
3, 6.
Referenceshttp://en.wikipedia.org/wiki/Rate_distortion_theory.J.R. Deller, J.G. Proakis, and J.H.L. Hansen, “Discrete-Time Processing of Speech Signals,” IEEE Press
69
Recommended