Fundamentals of Vector Quantization

8/18/2019 Fundamentals of Vector Quantization

1/87


2/87


3/87


4/87


5/87


6/87


7/87


8/87


9/87


10/87


11/87


12/87


13/87


14/87


15/87


16/87


17/87


18/87


19/87


20/87


21/87


22/87


23/87


24/87


25/87


26/87


27/87


28/87


29/87


30/87


31/87


32/87


33/87


34/87


35/87


36/87


37/87


38/87


39/87


40/87


41/87


42/87


43/87


44/87


45/87


46/87


47/87


48/87


49/87


50/87


51/87


52/87


53/87


54/87


55/87


56/87


57/87


58/87


59/87


60/87


61/87


62/87


63/87


64/87


65/87


66/87


67/87


68/87


69/87


70/87


71/87


72/87


73/87


74/87


75/87


76/87


77/87


78/87


79/87

If N very large, then lattice codes chosen so thatVoronoi cell has minimum normalized moment of

inertia are nearly optimal in the ECVQ problem.Tradeoff: Lattice codes mean simple NN selectionand nearly optimal variable rate performance,but need to entropy code to attain performance.ECVQ design can do better, and sometimesmuch better if rate not asymptotically big.

Note:

•For the xed rate case, cells should haveroughly equal partial distortion.•For the variable rate case, cells should haveroughly equal volume.

In neither case should you try to make cells haveequal probability (maximum entropy).

79


80/87

Summary

D(Q) ≈E [ m(X )λ(X )2/k ]N −

2/k

D̂k,f (R) ≈C k||f X ||k/ (k+2) 2−2R

D̂k,v(R)

≈C k2

2h(X )k 2−2R

Can use high rate quantization theory toquantify the gains of vector quantization overscalar quantization as a function of dimension k,and to separate out the gains as those duememory, to space lling (moment of inertia), andto shape (of density function).

80


81/87

Gish & Pierce

Asymptotic theory implies that for iid sources,high rates (low distortion)

D̂1,v(R)D̂k,v(R) ≤

C 1C ∞≈

1.533dB

Or, equivalently, for low distortionR̂1,v(D) − R̂k,v(D) ≤0.254 bits

famous “quarter bit” result.

Suggests at high rates there may be little to begained by using vector quantization (but stillneed to use vectors to do entropy coding!)

Ziv (1985) showed that for all distortions

R̂1,v

(D)

− R̂

k,v(D)

≤0.754 bits

using a dithering argument: X̂ = Q(X + Z ) −Z with Z uniform and independent of X .(subtractive dither)

81


82/87

Comment: If x R and let k → ∞, thenD̂

k,v(R) →

k → ∞D(R)

D̂k,f (R) →k → ∞D(R)

the Shannon DRF.

Bennett theory recently extended toinput-weighted squared error measures.

82


83/87

Recent Extensions

Can generalize to input-dependent squared errorand to non-difference distortion measures thatbehave locally in this way (have a well-behavedTaylor series expansion)

d(X, ˆX ) = (X −

ˆX )∗BX (X −

ˆX ),Fixed rate:

D ≥DL(Qopt) = kk + 2

C −2k

k N −2k×

[f (x)(det(B (x)))1k]

11+2/k dx

}k+2

k .Optimal point density

λ(x)∝( p(x)(det(B (x)))1k)

kk+2 .

Variable rate:

D ≥ DL(Qopt)=

kC −2k

kk + 2 ·e−

2k(H Q−h( p)−12 log(det(B(x)))f (x)dx)

83


84/87

λopt

(x) = (det(B (x)))

12

x∈G(det(B (x)))12dx

. (4)

84


85/87

Final Comments

•The Shannon compression model:codes, rate, and distortion Mathematical model of compression system.Successfully applied to asymptoticperformance bounds and design algorithms.

•Optimalility properties and quantizer design Lloyd optimal codes. For high rate, latticecodes + lossless codes nearly optimal (variablerate)Rate-distortion ideas can be used to improvestandards-compliant and wavelet codes. (e.g.,Ramchandran, Vetterli, Orchard)Multiple distortion measures (include othersignal processing)MPEG 4 style image structure?

85


86/87

Compression on networks, joint source andchannel coding

•Structured vector quantizers Transform/pyramid/subband/wavelet arecurrently best (rate, distortion, complexity)for image compression, especially embeddedzerotrees

Many variations and alternative structurescome and go, some stay.

•Optimal achievable performance:quantization and source coding theory Very general Shannon theorems exist. Recenturry of work on universal and adaptive codes.Multiple distortion measures, non-difference(perceptual) distortion measures.Current work on convergence and rate of convergence (Vapnic-Cernovenkis theory), usein classication and regression (Devroye,Gyor, Gabor, Nobel, Olshen)

86


87/87

Open problems: Gersho’s conjecture, correctcombination of wavelet theory + Bennett

(Goldberg did for traditional bit allocationapproach, not yet done for embedded zerotreecodes and related), other improvedcombinations of wavelet and compressiontheory and design (Mallat, Orchard).

Another issue: Perceived quality of compressedaudio and images.Different applications have different fundamentalquality requirements:Entertainment, browsing, screening, diagnostic,legal, scienticQuantitative as predictors for subjective anddiagnostic.

Documents

Fundamentals of Vector Quantization