Upload
others
View
17
Download
0
Embed Size (px)
Citation preview
Lossy Image CompressionLossy Image Compression and JPEG standard
Yao WangPolytechnic Institute of NYU, Brooklyn, NY 11201
With contribution from Zhu Liu, andGonzalez/Woods, Digital Image Processing, 2ed and A. K. Jain,
Fundamentals of Digital Image Processingg g g
Lecture Outline• Introduction• Quantization Revisited• Quantization Revisited
– General description of quantizerUniform quantization– Uniform quantization
• Transform codingReview of linear and unitary transform 1D and 2D– Review of linear and unitary transform, 1D and 2D
– DCT– Quantization of transform coefficientsQuantization of transform coefficients– Runlength coding of DCT coefficients
• JPEG standard overview
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 2
A Typical Compression System
Input Transformed Quantized Binary
Transfor- Quanti- Binary
InputSamples
Transformedparameters
Quantizedparameters
Binarybitstreams
Transformation
Quantization
BinaryEncoding
PredictionTransformsM d l fitti
Scalar QVector Q
Fixed lengthVariable length
Model fitting…...
g(Huffman, arithmetic, LZW)
• Motivation for transformation ---
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 3
To yield a more efficient representation of the original samples.
Quantization (Review)
• General descriptionUniform q anti er• Uniform quantizer
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 4
General Description of Quantization
Quantizerf Q(f)Q(f)
r7
r5
Decision Levels {tk, k = 0, …, L}
Reconstruction Levels {rk, k = 0, …, L-1}
)[f
r6
r7
ft1 t2 t5
If ),[ 1 kk ttfThen Q(f) = rk t6L levels need bits LR log
t3 t7r3
t4
r4
r1
Quantizererror
L levels need bits LR 2logr2
r3
x returns the smallest integerthat is bigger than or equal to x
r0
f
r1 r2 r3 r4 r5 r6 r7r0
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 5
t1 t2 t3 t4 t5 t6 t7t0=-∞ t8=∞
Uniform Quantization• Equal distances between adjacent
decision levels and between adjacentdecision levels and between adjacent reconstruction levels– tl – tl-1 = rl – rl-1 = ql l 1 l l 1 q
• Parameters of Uniform Quantization– L: levels (L = 2R)( )– B: dynamic range B = fmax – fmin– q: quantization interval (step size)q q ( p )– q = B/L = B2-R
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 6
Uniform Quantization: Functional Representation
Q(f)
r7=fmax-q/2t i (f f )/L
qff r5
r6
stepsize q=(fmax-fmin)/L
minmin
2)( fqq
qfffQ
r3
r4
f /2
r1
r2 x returns the biggest integer that is smaller than or equal to x
ft0 t1 t2 t3 t4 t5 t6 t7 t8
fmin fmax
r0=fmin+q/2than or equal to x
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 7
min max
.for used is leveltion reconstruc which indicates which index, leveltion reconstruc thecalled is )( min fqfffI
Uniform Quantization on Images=2
56
32
Orig
inal
, L=
q=8,
L=
O16
, L=1
6
4, L
=4
q=1
q=64
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 8
Implementation• Setup the quantization function
Q(f) for all possible input level into a look-up table first. Actual x = imread('lena256_8.bmp');
[h i ht idth] i ( )
Matlab code:
a look up table first. Actual quantization can be implemented as a table look-up.
[height, width] = size(x);B = 256;x = double(x);% This is the quantization tableQ (256 1)Q = zeros(256, 1);% Quantized to 64 levelsL = 16;q = B / L;f i 0 255for i = 0:255,
Q(i+1, 1) = floor(i / q) * q + q /2; endy = zeros(size(x));f i 1 h i htfor i = 1:height,
for j = 1:width,y(i, j) = Q(x(i,j) + 1);
endE d
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 9
L = 16, MSE = 21.6
EndMSE = mean(mean((x-y).^2));
Transform Coding• Represent an image (or an image block) as the linear combination of
some basis images and specify the linear coefficients.
+t1 t2 t3 t4
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 10
Components in Transform Coding
Bl kf t )(tI BBlocktransform Quantizer Binary Encoder
f t )(tI Bt
Encoder
Binary Decoder DequantizerInverseblock
transform
f̂)(tIBtt̂
Decoder
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 11
For block transform coders, we usually use Unitary (othonormal) transforms
Why Using Transform?
• When the transform basis is chosen properlyproperly– Many coefficients have small values and can
be quantized to 0 w/o causing noticeablebe quantized to 0 w/o causing noticeable artifacts
– The coefficients are uncorrelated and henceThe coefficients are uncorrelated, and hence can be coded independently w/o losing efficiency.
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 12
What Transform Basis to Use?• The transform should
– Minimize the correlation among resulting coefficients, so that scalar quantization can be employed without losing too much in coding efficiency compared to vector quantization
– Compact the energy into as few coefficients as possible
• Optimal transform – Karhunan Loeve Transform (KLT): signal statistics dependent
Suboptimal transform• Suboptimal transform– Discrete Cosine transform (DCT): nearly as good as KLT for
common image signals
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 13
One Dimensional Linear Transform• Let CN represent the N dimensional complex
space.space.• Let h0, h1, …, hN-1 represent N linearly
independent vectors in CN.p• Any vector f є CN can be represented as a linear
combination of h0, h1, …, hN-1 :0 1 N 1
,)(1
0
ktN
kk Bthf
AffBt 1
.)1()0(
],,...,,[ 110
tt
where N
thhhB
AffBt
f and t form a transform pair
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 14
)1(
Nt
Inner Product
• Definition of inner product1
* )()(N
H ffffff
• Orthogonal
0
212121 )()(,n
H nfnfffff
• Norm of a vector0, 21 ff
Norm of a vector
Normalized vector: unit norm
1
0
22 |)(|,N
n
H nffffff
12 f• Normalized vector: unit norm• Orthonomal = orthogonal + normalized
1f
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 15
Orthonormal Basis Vectors (OBV)• {hk, k=0,…N-1} are OBV if
lklk
lklk 01
, ,hh
• With OBVNN 11
fhhhhhfh Hlkl
kk
kll ltktkt
)(,)()(,,00
h
H
0
AffBfhh
t
HH
1
0
h
HN 1
.,1 IBBBBBB HHH or B is unitary
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 16
., IBBBBBB or y
Definition of Unitary Transform• Basis vectors are orthonormal• Forward transform
fh
N
fhk1
* )()()(
h
fh
H
nkk nfnhkt
0
0
* ,)()(,)(
I t f
AffBf
h
ht
H
H
H1
• Inverse transform h N 1
N
k nhktnf
1
,)()()(
tABtthhhhf HN
N
k
kk
kt
nhktnf
110
10
)(
,)()()(
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 17
k0
Example of 1D Unitary Transform
2/12/1
2/12/1
2/12/1
2/12/1
,2/12/1
,2/12/1
,2/1
2/1,
2/12/1
3210 hhhh
51
2/12/12/12/1
0t
02
32 1
0
tt
f
10
43
3
2
tt
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 18
1D DFT as a Unitary Transform
;1...,,1,0,)(1)(1
0
2
Nkenf
NkF
N
n
Nknj
.1...,,1,0,)(1)(1 2
0
NnekFN
nf
NN
Nknj
n
,1)(2
oreN
nh Nknj
k
0N k
1
12
N
Nkj
.1,...,1,0,1
)1(2
Nke
NN
kNj
N
k
h
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 19
e N
Example: 1D DFT, N=2
)02exp(
:vectorsbasisonly two are there:case 2kj
N
hhh
1
12
1,11
21:
)12
2exp(
)02
2exp(
21
10kj
jk
fh
f
obtain we,, Using
, determine ,21
if 10
t
tt
kk
111
212*11*1
21
21
,1
12
1,2
32*11*12
121
,11
21
10 tt
kk
fhh
21
11
21
11
23 :Verify 1100 tt
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 20
Another Example: 1D DFT, N=4
111
11
111
11)1
42exp(
)04
2exp(
1 jjkj
kj
hhhhh
12
1;
111
21;
121;
111
21:yields
)34
2exp(
)24
2exp(4
21 using :case 4 3210
j
j
j
j
kj
kjN k
f
);3(213542
21;73542
21
42
4
10 jjjtt
f
8)3()3(14
.3213542
21;03542
21
222
35
32
jj
jjjtt
fhhhh
122016
41
)3()3(14)3()3(14)3()3(14
41 :Verify 33221100
jjjjjj
jjjjtttt
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 21
1D Discrete Cosine Transform (DCT)
2)12(cos)()(
:Vectors Basis
N
knknhk
1
0
)()()( :Transform ForwardN
k nhnfkT
1,...,10
2
1)(
2
Nkk
N
Nkwhere
N
1
0
0
)()()( :Transforms InverseN
uk
n
nhkTnf
27060983cos
50682cos
65330381cos
501381cos
:case 4tion RepresentaVector
k
N
hhhhh
3210
2706.06533.06533.0
2706.0
821cos
815cos
89cos
21;
5.05.05.0
5.0
814cos
810cos
86cos
21;
6533.02706.0
2706.06533.0
87cos
85cos
83cos
21;
5.05.05.05.0
1111
21:yields
87cos
85cos
83cos
)(
k
k
kkk
f
10
3827065330*)45(27060*)32(235421
;9239.02706.0*)54(6533.0*)32(;7354221
542
8888
tt
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 22
fhhhh
33221100
32
:Verify
.3827.06533.0*)45(2706.0*)32(;2354223
5
tttt
tt
Two Dimensional Decomposition
• Decompose a MxN 2D matrix F=[F(m,n)] into a linear combination of some basicinto a linear combination of some basic images, Hk,l=[Hk,l(m,n)], so that:
1
0
1
0, ,),(
M
k
N
llklkT HF
1 1
,
0 0
),(),(),(M N
lk
k l
nmHlkTnmF 0 0k l
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 23
Graphical Interpretation
+t1 t2 t3 t4t1 t2 t3 t4
Inverse transform: Represent a vector (e.g. a block of image samples) as the superposition of some basis vectors (block patterns)Forward transform: Determine the coefficients associated with each basis vector
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 24
Two Dimensional Inner Product
• Inner Product
1 1
*M N
• Norm of a Matrix
0 0
2*
121 ),(),(,m n
nmFnmFFF
• Norm of a Matrix
1
0
1
0
2),(,M N
nmFFFF
• A set of basis images {Hk,l, k=0,1,…,M-1, l=0 1 N-1} is orthonormal if
0 0m n
l=0,1,…,N-1} is orthonormal if
otherwise
jlikifjlikjilk 0
,,1, ,,,, HH
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 25
otherwise,0
Two Dimensional Unitary Transform
• {Hk,l} is an orthonormal set of basis imagesFor ard transform• Forward transform
1 1
* )()()(M N
nmFnmHlkT FH
• Inverse transform
0 0
,, ),(),(,),(m n
lklk nmFnmHlkT FH
Inverse transform
1
0
1
0, ,),(),(),(
M
k
N
llk ornmHlkTnmF
1
0
1
0,),(
M
k
N
llklkT HF
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 26
Example of 2D Unitary Transform
5)0,0(
,2/12/12/12/1
,2/12/12/12/1
,2/12/1
2/12/1,
2/12/12/12/1
11100100
T
HHHH
0)1,1(1)0,1(2)1,0(
),(
4321
TTT
F
0)1,1(T
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 27
Separable Unitary Transform
• Let hk, k=0, 1, …, M-1 represent orthonormal basis vectors in CMorthonormal basis vectors in C ,
• Let gl, l=0, 1, …, N-1 represent orthonormal basis vectors in CNorthonormal basis vectors in CN,
• Hk,l=hkglT, or Hk,l(m,n)=hk(m)gl(n).
• Then Hk,l will form an orthonormal basis set in CMxN.
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 28
Example of Separable Unitary Transform
• Example 1 2/12/1
2/12/12/12/1
.2/1
2/1,2/12/1
10
TT hhHhhH
hh
2/12/12/12/1
2/12/12/12/1
2/12/12/12/1
11110110
10010000
TT hhHhhH
hhHhhH
2/12/12/12/1
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 29
Example: 4x4 DFT
1
1
21;
11
1
21;
1
1
21;
111
21: are basis DFT 1D theRecall 3210
jjhhhh
1111
111111111
11111
111111111
1
:yields using
11
jjjj
jjjj
jjT
lkk,l hhH
1111
11111
11111
11111
1
11114
1,
111111114
1,
11114
1,
111111114
13,02,01,00,0
jjjj
jjjjjjjj
jjjj
jjjjjj
jjjjjj
HHHH
HHHH
1111
111111111
11111
111111111
1
11114
,11114
,
11114
,11114 3,12,11,10,1
jjjj
jjjj
jjjj
jjjjjjjj
jjjj
HHH
HHHH
1111
1,
11111,
1111
1,
11111
11114
,
111111114
,
11114
,
111111114
33231303
3,22,21,20,2
jjjj
jjjjjjjj
jjjj
jjjj
jjjj
HHHH
HHH
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 30
11
114,
11114,
11114
,11114 3,32,31,30,3
jjjj
jjjjjjjj
jjjj
Example: 4x4 DFT
comp te13100221
For T
F
02211111
e.g, yields, , Using
compute ,
12211210
For
,,
,
T
T
lklk
lk
FH
F
022111
4181221121013100221
41
122112101310
,
111111111111
41,
*
0,00,0
jj
T
FH
)1(4122123221
41
122112101310
,
1111
1141,3,23,2 jjjjjjjj
jjjjjj
T
FH
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 31
Basis Images of 8x8 DCT
01)(
2)12(cos
2)12(cos)()(),(,
kNkwhere
Nln
NkmlknmH lk
Low-Low High-Low
1,...,12)(
Nk
Nkwhere
o o
Low-High High-High
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 32
Property of Separable Transform
• When the transform is separable, we can perform the 2D transform separatelyperform the 2D transform separately.– First, do 1D transform for each row using
basis vectors gbasis vectors gl,– Second, do 1D transform for each column of
the intermediate image using basis vectors hkthe intermediate image using basis vectors hk.– Proof:
11 11 1 MM NM N
0
*
0 0
**
0 0
*, ),()(),()()(),(),(),(
mm nl
m nlk lmUmhnmFngmhnmFnmHlkT
kk
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 33
DCT on a Real Image Block
>>imblock = lena256(128:135,128:135)-128
imblock=imblock
54 68 71 73 75 73 71 45
47 52 48 14 20 24 20 -8
20 -10 -5 -13 -14 -21 -20 -21 >>dctblock =dct2(imblock)
-13 -18 -18 -16 -23 -19 -27 -28
-24 -22 -22 -26 -24 -33 -30 -23
-29 -13 3 -24 -10 -42 -41 5
-16 26 26 -21 12 -31 -40 23
dctblock=
31.0000 51.7034 1.1673 -24.5837 -12.0000 -25.7508 11.9640 23.2873
113.5766 6.9743 -13.9045 43.2054 -6.0959 35.5931 -13.3692 -13.0005
195.5804 10.1395 -8.6657 -2.9380 -28.9833 -7.9396 0.8750 9.5585
17 30 50 -5 4 12 10 5 35.8733 -24.3038 -15.5776 -20.7924 11.6485 -19.1072 -8.5366 0.5125
40.7500 -20.5573 -13.6629 17.0615 -14.2500 22.3828 -4.8940 -11.3606
7.1918 -13.5722 -7.5971 -11.9452 18.2597 -16.2618 -1.4197 -3.5087
1 4562 13 3225 0 8750 1 3248 10 3817 16 0762 4 4157 1 1041-1.4562 -13.3225 -0.8750 1.3248 10.3817 16.0762 4.4157 1.1041
-6.7720 -2.8384 4.1187 1.1118 10.5527 -2.7348 -3.2327 1.5799
Note that low-low coefficients are much larger than high-high coefficients
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 34
In JPEG, “imblock-128” is done before DCT to shift the mean to zero
Note that low low coefficients are much larger than high high coefficients
DCT Coefficient Distribution
ianc
eVa
r
Coefficient Index (Zig-Zag Order)
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 35
Coefficient Index (Zig Zag Order)
Approximation by DCT Basis
Original With 16/64Coefficients
With 8/64Coefficients With 4/64
Coefficients
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 36
Quantization of DCT Coefficients• Use uniform quantizer on each coefficient• Different coefficient is quantized with different• Different coefficient is quantized with different
step-size (Q):– Human eye is more sensitive to low frequencyy q y
components– Low frequency coefficients with a smaller Q– High frequency coefficients with a larger Q– Specified in a normalization matrix (aka quantization
matrix)matrix)– Normalization matrix can then be scaled by a scale
factor (aka quality factor or QP)
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 37
Default Normalization Matrix in JPEG
NormalizationMatrix
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 38
Actual step size for C(i,j): Q(i,j) = QP*M(i,j)
Example: Quantized Indices
>>dctblock =dct2(imblock)( )
dctblock=
31.0000 51.7034 1.1673 -24.5837 -12.0000 -25.7508 11.9640 23.2873
113.5766 6.9743 -13.9045 43.2054 -6.0959 35.5931 -13.3692 -13.0005>>QP=1;
>>QM=Qmatrix*QP;195.5804 10.1395 -8.6657 -2.9380 -28.9833 -7.9396 0.8750 9.5585
35.8733 -24.3038 -15.5776 -20.7924 11.6485 -19.1072 -8.5366 0.5125
40.7500 -20.5573 -13.6629 17.0615 -14.2500 22.3828 -4.8940 -11.3606
7.1918 -13.5722 -7.5971 -11.9452 18.2597 -16.2618 -1.4197 -3.5087
>>qdct=floor((dctblock+QM/2)./(QM))
qdct =
2 5 0 -2 0 -1 0 0
9 1 -1 2 0 1 0 0-1.4562 -13.3225 -0.8750 1.3248 10.3817 16.0762 4.4157 1.1041
-6.7720 -2.8384 4.1187 1.1118 10.5527 -2.7348 -3.2327 1.579914 1 -1 0 -1 0 0 0
3 -1 -1 -1 0 0 0 0
2 -1 0 0 0 0 0 0
0 0 0 0 0 0 0 00 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 39
Only 19 coefficients are retained out of 64
Example: Quantized Coefficients%dequantized DCT block
>> iqdct=qdct.*QM
iqdct=
32 55 0 -32 0 -40 0 0
108 12 -14 38 0 58 0 0
196 13 -16 0 -40 0 0 0
42 17 22 29 0 0 0 042 -17 -22 -29 0 0 0 0
36 -22 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 Original DCT block0 0 0 0 0 0 0 0 dctblock=
31.0000 51.7034 1.1673 -24.5837 -12.0000 -25.7508 11.9640 23.2873
113.5766 6.9743 -13.9045 43.2054 -6.0959 35.5931 -13.3692 -13.0005
195 5804 10 1395 8 6657 2 9380 28 9833 7 9396 0 8750 9 5585195.5804 10.1395 -8.6657 -2.9380 -28.9833 -7.9396 0.8750 9.5585
35.8733 -24.3038 -15.5776 -20.7924 11.6485 -19.1072 -8.5366 0.5125
40.7500 -20.5573 -13.6629 17.0615 -14.2500 22.3828 -4.8940 -11.3606
7.1918 -13.5722 -7.5971 -11.9452 18.2597 -16.2618 -1.4197 -3.5087
Note that many coefficients become zero after quantization!
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 40
-1.4562 -13.3225 -0.8750 1.3248 10.3817 16.0762 4.4157 1.1041
-6.7720 -2.8384 4.1187 1.1118 10.5527 -2.7348 -3.2327 1.5799
q
Example: Reconstructed Image
%reconstructed image block
>> qimblock=round(idct2(iqdct))Original image block
imblock=qimblock=
58 68 85 79 61 68 67 38
45 38 39 33 22 24 19 -2
21 2 11 12 13 19 24 27
imblock
54 68 71 73 75 73 71 45
47 52 48 14 20 24 20 -8
20 -10 -5 -13 -14 -21 -20 -2121 2 -11 -12 -13 -19 -24 -27
-8 -19 -31 -26 -20 -35 -37 -15
-31 -17 -21 -20 -16 -39 -41 0
-33 3 -1 -14 -11 -37 -44 1
-13 -18 -18 -16 -23 -19 -27 -28
-24 -22 -22 -26 -24 -33 -30 -23
-29 -13 3 -24 -10 -42 -41 5
-16 26 26 -21 12 -31 -40 23-16 32 18 -10 1 -16 -30 8
3 54 30 -6 16 11 -7 2317 30 50 -5 4 12 10 5
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 41
Coding of DCT Coefficients• Many coefficients are zeros after quantization.• Specify which coefficients are non-zero by using• Specify which coefficients are non-zero by using
run-lengths of zeros.• Run-Length Coding - Used in JPEG standardRun Length Coding Used in JPEG standard
• Ordering coefficients in the zig-zag order• Each symbol=(length-of-zero, non-zero-value)• Code each symbol using Huffman coding
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 42
Zig-Zag Ordering of DCT Coefficients
0 1 5 6 14 15 27 280 1 5 6 14 15 27 28
2 4 7 13 16 26 29 42
3 8 12 17 25 30 41 43
9 11 18 24 31 40 44 53
10 19 23 32 39 45 52 54
20 22 33 38 46 51 55 6020 22 33 38 46 51 55 60
21 34 37 47 50 56 59 61
35 36 48 49 57 58 62 63
Zig-Zag ordering: converting a 2D matrix into a 1D array, so that the frequency (horizontal+vertical) increases in this order and the coefficient variance decreases in this order
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 43
order, and the coefficient variance decreases in this order.
Exampleqdct =
2 5 0 -2 0 -1 0 0
9 1 -1 2 0 1 0 0
14 1 -1 0 -1 0 0 0
3 -1 -1 -1 0 0 0 0
2 -1 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 00 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
Run-length symbol representation:{2 (0 5) (0 9) (0 14) (0 1) (1 2) (0 1) (0 1) (0 3) (0 2) (0 1) (0 1) (0 2) (1{2,(0,5),(0,9),(0,14),(0,1),(1,-2),(0,-1),(0,1),(0,3),(0,2),(0,-1),(0,-1),(0,2),(1,-1),(2,-1), (0,-1), (4,-1),(0,-1),(0,1),EOB}
EOB: End of block, one of the symbol that is assigned a short Huffman
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 44
, y gcodeword
What is JPEG
• The Joint Photographic Expert Group (JPEG), under both the International Standards Organization (ISO) and the International g ( )Telecommunications Union-Telecommunication Sector (ITU-T)
– www.jpeg.org
H bli h d l t d d• Has published several standards– JPEG: lossy coding of continuous tone still images
• Based on DCT
– JPEG-LS: lossless and near lossless coding of continuous tone still images
• Based on predictive coding and entropy codingp g py g
– JPEG2000: scalable coding of continuous tone still images (from lossy to lossless)
• Based on wavelet transform
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 45
Based on wavelet transform
The 1992 JPEG Standard• Contains several modes:
– Baseline system (what is commonly known as JPEG!): y ( y )lossy
• Can handle gray scale or color images (8bit)– Extended system: lossyExtended system: lossy
• Can handle higher precision (12 bit) images, providing progressive streams, etc.
– Lossless versionLossless version• Baseline system
– Each color component is divided into 8x8 blocksp– For each 8x8 block, three steps are involved:
• Block DCT• Perceptual-based quantization
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 46
• Perceptual-based quantization• Variable length coding: Runlength and Huffman coding
Coding of Quantized DCT CoefficientsCoefficients
• DC coefficient: Predictive coding– The DC value of the current block is predicted from that of the previous
block, and the error is coded using Huffman codingblock, and the error is coded using Huffman coding• AC Coefficients: Runlength coding
– Many high frequency AC coefficients are zero after first few low-frequency coefficients
– Runlength Representation:• Ordering coefficients in the zig-zag order• Specify how many zeros before a non-zero value• Each symbol=(length-of-zero non-zero-value)Each symbol (length of zero, non zero value)
– Code all possible symbols using Huffman coding• More frequently appearing symbols are given shorter codewords• For more details on the actual coding table, see Handout (Sec.8.5.3 in
[G l 02][Gonzalez02]• One can use default Huffman tables or specify its own tables.• Instead of Huffman coding, arithmetic coding can be used to achieve
higher coding efficiency at an added complexity
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 47
higher coding efficiency at an added complexity.
Coding of DC Symbols• Example:
– Current quantized DC index: 2q– Previous block DC index: 4– Prediction error: -2
Th di ti i d d i t t– The prediction error is coded in two parts:• Which category it belongs to (Table of JPEG Coefficient
Coding Categories), and code using a Huffman code (JPEG Default DC Code)Default DC Code)
– DC= -2 is in category “2”, with a codeword “100”• Which position it is in that category, using a fixed length code,
length=category numberlength category number– “-2” is the number 1 (starting from 0) in category 2, with a fixed
length code of “01”.– The overall codeword is “10001”
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 48
JPEG Tables for Coding DC
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 49
Coding of AC Coefficients• Example:
– First symbol (0,5)Th l ‘5’ i t d i t t• The value ‘5’ is represented in two parts:
• Which category it belongs to (Table of JPEG Coefficient Coding Categories), and code the “(runlength, category)” using a Huffman code (JPEG Default AC Code)( )
– AC=5 is in category “3”, – Symbol (0,3) has codeword “100”
• Which position it is in that category, using a fixed length code, length=category numberlength=category number
– “5” is the number 5 (starting from 0) in category 3, with a fixed length code of “101”.
– The overall codeword for (0,5) is “100101”S d b l (0 9)– Second symbol (0,9)
• ‘9’ in category ‘4’, (0,4) has codeword ‘1011’,’9’ is number 9 in category 4 with codeword ‘1001’ -> overall codeword for (0,9) is ‘10111001’
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 50
– ETC
JPEG Tables for Coding AC (Run,Category) Symbols(Run,Category) Symbols
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 51
JPEG Performance for B/W images
65536 Bytes8 bpp
4839 Bytes0.59 bppCR=13.6CR 13.6
3037 Bytes0.37 bppCR=21.6
1818 Bytes0.22 bppCR=36.4
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 52
JPEG for Color Images• Color images are typically stored in (R,G,B) format• JPEG standard can be applied to each component separately
– Does not make use of the correlation between color components– Does not make use of the lower sensitivity of the human eye to
chrominance samplesAlt t h• Alternate approach – Convent (R,G,B) representation to a YCbCr representation
• Y: luminance, Cb, Cr: chrominance D l th t h i t– Down-sample the two chrominance components
• Because the peak response of the eye to the luminance component occurs at a higher frequency (3-10 cpd) than to the chrominance components (0.1-0.5 cpd). (Note: cpd is cycles/degree)
• JPEG standard is designed to handle an image consists of many (up to 100) components
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 53
RGB <-> YCbCr Conversion
0114.0587.0299.0 RY
128128
081.0419.0500.0500.0331.0169.0
r
b
BG
CC
128714034400001402.1001.0000.1
CY
GR
128128
001.0772.1000.1714.0344.0000.1
r
b
CC
BG
Note: Cb ~ Y-B, Cr ~ Y-R, are known as color difference signals.
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 54
Chrominance Subsampling
4:2:04:2:24:4:4 4:1:1 4:2:0For every 2x2 Y Pixels
1 Cb & 1 Cr Pixel(Subsampling by 2:1 bothhorizontally and vertically)
4:2:2For every 2x2 Y Pixels
2 Cb & 2 Cr Pixel(Subsampling by 2:1
horizontally only)
4:4:4For every 2x2 Y Pixels
4 Cb & 4 Cr Pixel(No subsampling)
4:1:1For every 4x1 Y Pixels
1 Cb & 1 Cr Pixel(Subsampling by 4:1
horizontally only)
Y Pixel Cb and Cr Pixel
4 2 0 i th t f t
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 55
4:2:0 is the most common format
Coding Unit in JPEG
4 8x8 Y blocks 1 8x8 Cb blocks 1 8x8 Cr blocks
Each basic coding unit (called a data unit) is a 8x8 block in any color component.In the interleaved mode, 4 Y blocks and 1 Cb and 1 Cr blocks are processed as a group (called a minimum coding unit or MCU) for a 4:2:0 image.
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 56
Default Quantization Table
For luminance For chrominance
17 18 24 47 99 99 99 99
18 21 26 66 99 99 99 99
24 26 56 99 99 99 99 99
47 66 99 99 99 99 99 99
99 99 99 99 99 99 99 99
99 99 99 99 99 99 99 99
99 99 99 99 99 99 99 99
99 99 99 99 99 99 99 99
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 57
The encoder can specify the quantization tables different from the default ones as part of the header information
Performance of JPEG• For color images at 24 bits/pixel (bpp)
– 0.25-0.5 bpp: moderate to good0.25 0.5 bpp: moderate to good– 0.5-0.75 bpp: good to very good– 0.75-1.5 bpp: excellent, sufficient for most
applications– 1.5-2 bpp: indistinguishable from original
F G K W ll Th JPEG Still i t– From: G. K. Wallace: The JPEG Still picture compression standard, Communications of ACM, April 1991.p
• For grayscale images at 8 bpp– 0.5 bpp: excellent quality
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 58
JPEG Performance
487x414 pixels, Uncompressed, 600471 Bytes,24 bpp 487x414 pixels
41174 Bytes, 1.63 bpp, CR=14.7
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 59
85502 Bytes, 3.39 bpp, CR=7
JPEG Pros and Cons
• ProsLow
• Cons– Single resolution– Low
complexityMemory
Single resolution– Single quality
No target bit rate– Memory efficient
– Reasonable
– No target bit rate– Blocking artifacts at low
bit rateReasonable coding efficiency
bit rate– No lossless capability
Poor error resiliencey – Poor error resilience– No tiling
No regions of interest
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 60
– No regions of interest
Homework1. For the 2x2 image S given below, compute its 2D DCT, reconstruct it by retaining
different number of coefficients to evaluate the effect of different basis imagesdifferent number of coefficients to evaluate the effect of different basis images.a) Determine the four DCT basis images.b) Determine the 2D-DCT coefficients for S, Tk,l, k=0,1;l=0,1. c) Show that the reconstructed image from the original DFT coefficients equal to the original
imageimage.d) Modify the DCT coefficients using the given window masks (W1 to W5) and reconstruct
the image using the modified DCT coefficients. (for a given mask, “1” indicates to retain that coefficient, “0” means to set the corresponding coefficient to zero) What effect do you see with each mask and why?y
1001
,1000
,0100
,0010
,0001
,9119
54321 WWWWWS
2. For the same image S as given in Prob. 1, quantize its DCT coefficients using the quantization matrix Q. Determine the quantized coefficient indices and quantized values. Also, determine the reconstructed image from the quantized coefficients
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 61
coefficients.
5333
Q
Homework (cnt’d)3. Describe briefly how JPEG compresses an image. You may want to break down your
discussion into three parts:a) How does JPEG compress a 8x8 image block (three steps are involved)b) H d JPEG l ib) How does JPEG compress a gray-scale imagec) How does JPEG compress a RGB color image
4. Suppose the DCT coefficient matrix for an 4x4 image block is as shown below (dctblock).
a) Quantize its DCT coefficients using the quantization matrix Q given below assuminga) Quantize its DCT coefficients using the quantization matrix Q given below, assuming QP=1. Determine the quantized coefficient indices and quantized values.
b) Represent the quantized indices using the run-length representation. That is, generate a series of symbols, the first being the quantized DC index, the following symbols each consisting of a length of zeros and the following non-zero index, the last symbol is EOB ( d f bl k)(end of block).
c) Encode the DC index and the runlength symbols from (b) using the JPEG coding method, with the coding tables given in the lecture note. For this problem, assuming the quantized index for the DC coefficient of the previous block is 60.
,
0.0085 0.0044 0.0086 0.0046- 0.0019 0.0036 0.0207- 0.0086- 0.0071 0.0877 0.0033- 0.0134 0.0912 0.0466- 0.0500- 1.3676
* 0030.1
edctblock ,
120 103 78 49 103 68 37 18 69 40 16 14 51 24 10 16
Q
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 62
Computer Assignment1. Computer assignment: Write a program to examine the effect of
quantizing the DCT coefficients. For each 8 x 8 block in the image, your program should first calculate the DCT of the block, quantize th ffi i t d th t k th i DCT F ti tithe coefficients, and then take the inverse DCT. For quantization, using a scaled version of the JPEG quantization matrix as the stepsizes. Examine the resulting image quality with the followngvalues for the scaling factor: 0.5, 1, 2, 4, 8. What is the largest
l f th li f t t hi h th t t d ivalue of the scaling factor at which the reconstructed image quality is very close to the original image? Hint: you should make use of the ”blkproc” function in MATLAB to speed up your program.
2. Modify the program above, so that it keeps only first N coefficients in the zig-zag order. N should be a user-selectable parameter. Examine the resulting image quality when N=1, 4, 16, 32. For the image you have chosen, how many DCT coefficients should be g y yretained to maintain a sufficient quality?
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 63
Reading
• R. Gonzalez, “Digital Image Processing,” Section 8 5 - 8 6Section 8.5 - 8.6
• A.K. Jain, “Fundamentals of Digital Image Processing ” Section 11 4 11 5Processing,” Section 11.4 – 11.5
Yao Wang, NYU-Poly EL5123: Transform Coding and JPEG 64