View
219
Download
0
Tags:
Embed Size (px)
Citation preview
Xinqiao Liu Rate constrained conditional replenishment
1
Rate-Constrained Conditional Replenishment with Adaptive
Change Detection
Xinqiao Liu
December 8, 2000
EE368B Project
Xinqiao Liu Rate constrained conditional replenishment
2
Motivation
• Conditional replenishment ---- method of reducing temporal redundancy between successive frames
– Efficient in video conferencing with stationary cameras and slow motion.
– Study shows that less than 3% of the pixels need to be replenished in most head-and-shoulders scenes in desktop video
• Computational complexity is significant simpler than other video compression methods– Software-only CODEC is possible
– Appealing for on-sensor compression where pixel array and simple image processing are integrated on the same chip, i.e, camera system-on-chip
Xinqiao Liu Rate constrained conditional replenishment
3
Previous Work
• Most of the research concentrate mainly on the image quality (Haskell,
et al’72, Haskell’79)
• Recently, a perception-based change detection method was proposed
(Chiu&Berger ’96, Chiu&Berger’99)
– Reduces the perceptual redundancy in addition to the spatial and temporal redundancy
– Change detection threshold is set based on Web’s law
• However, the correlation between transmission bit-rate and the choice
of change detection schemes still need to be explored.
Xinqiao Liu Rate constrained conditional replenishment
4
Outline
• Introduction & Problem formulation
• Context-based Arithmetic Encoder
• Change detection --- direct methods
– Subsampling
– Threshold adjusting
• Adaptive change detection
– Noise characteristic
– Adaptive algorithm
• Conclusion
Xinqiao Liu Rate constrained conditional replenishment
5
Conditional Replenishment Diagram
Goal: Given a rate-constrained transmission channel, find the optimal change detection algorithm that minimizes the distortion
Xinqiao Liu Rate constrained conditional replenishment
6
Model and Assumptions
Assumptions:1. Transmitted separately under certain bit-rate constrain R1, R2 2. Lossless coding for both mask and signal3. Only intra-frame compression is considered
Current frame
Change detector
Change mask
Changed signal
CAE encoder
Signal encoder
Current frame
Change mask
Changed signal
CAE decoder
Signal decoder
Frame store Frame
store
Channel #1, rate = R1
Channel #2, rate = R2
#1
#2
Xinqiao Liu Rate constrained conditional replenishment
7
Rate-Constrained Change Detection
• Three ways to control the bit rate in the change detector:
1. Subsampling the mask and signal after detection
2. Adjusting the detection threshold
3. Using adaptive threshold for each pixel based on the noise characteristics -----eliminate those pixels that have changed due to noise rather than the input
• Use unconstrained Lagrangian cost function to find the optimum detection parameters for each method
Change mask
ABS Average of 3x3 window
Threshold
Previous frame
Current frame
Xinqiao Liu Rate constrained conditional replenishment
8
Problem Formulation (I)
CACAA 212 )1(ˆ
The mean-square distortion is defined as: 2
212
2222 ))1(()ˆ()ˆ,( CACAEAAEAAD
The above assumption allows us to study the rate-distortion function of conditional replenishment by only implementing the compression scheme of the mask.
Assume R1 = kR2 since they are proportional to the number of changed
pixels. The total bit-rate R is
1221 )1()()( RkCAHCHRRR
Given previous frame A1, current frame A2, binary change mask C, the reconstructed frame at decoder end is:
Xinqiao Liu Rate constrained conditional replenishment
9
Problem Formulation (II)The constrained problem of:
)()(),( sRsDsJ
Can be converted to the unstrained problem by introducing the Lagrangian cost function given Lagrange multiplier :
max222 osubject t )ˆ|)ˆ,(min( RRAAAD
0),(
s
sJ
where s is the adjustable change detection parameter. The optimal
value of s is given by:
The desired optimal slop value is not known a priori but can be obtained using a fast bisection search algorithm
Xinqiao Liu Rate constrained conditional replenishment
10
Outline
• Introduction & problem formulation
• Context-based Arithmetic Encoder
• Change detection --- direct methods
• Adaptive change detection
• Conclusion
Xinqiao Liu Rate constrained conditional replenishment
11
Test Video Sequence
• Captured by a stationary high-speed digital camera with a person moving cross the screen:
Xinqiao Liu Rate constrained conditional replenishment
12
Context-based Arithmetic Encoder (CAE)
• Binary bitmap-based shape coding scheme used in the MPEG-4 standard
• Three types of 16x16 macroblocks:
– "black" block: none of the pixel changed (all 0)
– "white" block: all pixels changed and to be replenished (all 1)
– “boundary” block: encoded with a template of 10 pixels to define the causal context for predicting the binary value of the current pixel (S0).
S8 S9 S10
S3 S4 S5 S6 S7
S2 S1 S0 )),,(log(),,,(),,|( 1010101010210
100 1
ssspssspSSSSHss s
For black and white blocks, only the block type need to be transmitted
For boundary blocks, use conditional entropy:
Xinqiao Liu Rate constrained conditional replenishment
13
Outline
• Introduction & problem formulation
• Context-based Arithmetic Encoder
• Change detection --- direct methods
– Subsampling
– Threshold adjusting
• Adaptive change detection
• Conclusion
Xinqiao Liu Rate constrained conditional replenishment
14
Change Masks With Subsampling • Subsample the macroblock by a factor of 2, 4 or 8
• Subblocks are encoded using the CAE
• Upsample at the decoder end using pixel replication filter combined with a 3x3 median filter
Xinqiao Liu Rate constrained conditional replenishment
15
Rate-distortion of Subsampling
Xinqiao Liu Rate constrained conditional replenishment
16
Change Masks With Threshold-adjusting
• Control the bit-rate by globally adjusting the change detector threshold. As the threshold increased, few pixels will be detected
Xinqiao Liu Rate constrained conditional replenishment
17
Rate-distortion of Threshold-adjusting
Xinqiao Liu Rate constrained conditional replenishment
18
Outline
• Introduction & problem formulation
• Context-based Arithmetic Encoder
• Change detection --- direct methods
• Adaptive change detection
– Noise characteristics
– Adaptive algorithm
• Conclusion
Xinqiao Liu Rate constrained conditional replenishment
19
Noise Characteristics
• A fundamental problem in designing an optimum change detector is how to separate pixels whose change is due to noise from pixels whose change is due to real input signal change
• For cameras using either CCD or CMOS image sensors, the final image is formed by the photo-charge Qi,j (or voltage) integrated on each photo-detector
during the exposure time. Two independent additive noise corrupt the output signal:
– Shot noise Ui,j which is zero mean signal dependent gaussian distribution
with:
– Readout circuit and reset noise Vi,j (including quantization noise) with
zero mean and variance V
.
),0(~ ,, jiji qQNU
Xinqiao Liu Rate constrained conditional replenishment
20
Adaptive Change Detection
• Thus the total noise variance of pixel (i,j) is:
– The noise is signal dependent
– The stronger the luminance level, the noisier the pixel will be
• The threshold Ti,j is thus set as:
– where m is the sensitivity factor that is set globally
– is local average value over a small area with size 8x8.
• Note that by changing m, we effectively adjusting the detection sensitivity while the threshold is still locally adapted
2,,
2Vjiji qQ
2,,, Vjijiji QqmmT
jiQ ,
Xinqiao Liu Rate constrained conditional replenishment
21
Adaptive Threshold
Xinqiao Liu Rate constrained conditional replenishment
22
Change Masks With Adaptive Threshold
Xinqiao Liu Rate constrained conditional replenishment
23
Rate-distortion of Adaptive Threshold
Xinqiao Liu Rate constrained conditional replenishment
24
Performance Comparison
•Subsampling is the most efficient in reducing bit-rate•Adaptive thresholding achieves the best PSNR
Xinqiao Liu Rate constrained conditional replenishment
25
Conclusion
• Studied three change detection algorithms:
1. Subsampling
2. Threshold-adjusting
3. Adaptive threshold based on the noise characteristics
• The adaptive change detection algorithm efficiently separates pixels whose change is due to noise from pixels whose value change is due to real input signal change
• Simulation proves that the adaptive change detection algorithm achieves the best PSNR among all the three algorithms