Upload
rumor
View
66
Download
0
Embed Size (px)
DESCRIPTION
Stereo Matching with Color-Weighted Correlation, Hierarchical Belief Propagation, and Occlusion Handling. Qingxiong Yang, Student Member, IEEE, Liang Wang, Student Member, IEEE, Ruigang Yang, Member, IEEE, Henrik Stewe ´ nius , Member, IEEE, and David Niste ´ r, Member, IEEE. - PowerPoint PPT Presentation
Citation preview
Stereo Matching with Color-Weighted Correlation, Hierarchical Belief Propagation,and Occlusion HandlingQingxiong Yang, Student Member, IEEE, Liang Wang, Student Member, IEEE,Ruigang Yang, Member, IEEE, Henrik Stewe´ nius, Member, IEEE, and David Niste´ r, Member, IEEE
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 31, NO. 3, MARCH 2009
Outline• Introduction• System Overview•Methods• Initialization• Pixel Classification• Iterative Refinement• Fast-Converging Belief Propagation
• Depth Enhancement• Experiments• Conclusion
Introduction
Introduction• Stereo is one of the most extensively researched
topics in computer vision.
• Energy Minimization framework:• Graph Cut• Belief Propagation(BP)
Objective(Contribution)• To formulate stereo model with careful handling
of:• Disparity• Discontinuity• Occlusion
• Differs from the normal framework in the final stages of the algorithm
• Outperforms all other algorithms on the average
System Overview
•1) Initialization
•1) Initialization
•1) Initialization
•2) Pixel Classification
•3) Iterative Refinement
Initialization(Block 1)
Initialization• Input:• Left Image IL
• Right Image IR
• Output:• Initial Left Disparity Map DL
(0)
• Initial Right Disparity Map DR
• Initial Data Term ED(0)
Image
Color-Weighted Correlation
Correlation Volume
Data Term Initialization
Hierarchical BP
Disparity Map Initialization
ED(0)
CL CR
DL(0) DR
Image
Color-Weighted Correlation
Correlation Volume
Data Term Initialization
Hierarchical BP
Disparity Map Initialization
Initialization• Color-weighted Correlation
• To build the Correlation Volume
• Makes the match scores less sensitive to occlusion boundaries
• By using the fact that occlusion boundaries most often cause color discontinuities as well
ED(0)
DL(0) DR
CL CR
Correlation Volume• Color difference Δxy between pixel x and y
(in the same image)
Ic: Intensity of the color channel c
• The weight of pixel x in the support window of y:
Color Difference Spatial Difference
10 21
Correlation Volume• The Correlation Volume[27]:
• Wx : support window around x• d(yL, yR ) : pixel dissimilarity[1]
• xL , yL : pixels in left image IL
• xR , yR : corresponding pixels in right image IR
• dx : disparity value of pixel XL in IL
weight
Pixels in the window Dissimilarity[1]
dx = arg min CL,x (yL, yR)
weight
xR = xL – dx
yR = yL – dx
Correlation VolumeDisparity Map Bad Pixel
[27] K.-J. Yoon and I.-S. Kweon, “Adaptive Support-Weight Approach for Correspondence Search,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 28, pp. 650-656, 2006.
[1] S. Birchfield and C. Tomasi, “A Pixel Dissimilarity Measure That Is Insensitive to Image Sampling,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 20, pp. 401-406, 1998.
Image
Color-Weighted Correlation
Correlation Volume
Data Term Initialization
Hierarchical BP
Disparity Map Initialization
Initialization• Initial Data Term• Total energy = Data Term + Smooth Term
• Computed from Correlation Volume
• Given an iteration index i = 0 here because it will be iteratively refined
ED(0)
DL(0) DR
CL CR
Initial Data Term• Initial Data Term:
• Ƞbp : twice the average of correlation volume to exclude the outliers
Correlation Volume0.2
X 2Average
Correlation Volume
Image
Color-Weighted Correlation
Correlation Volume
Data Term Initialization
Hierarchical BP
Disparity Map Initialization
Initialization• hierarchical Belief Propagation
• Employed with the data term and the reference image
• Resulting in the initial left and right disparity maps DL
(0) and DR
DL(0) DR
CL CR
ED(0)
Pixel Classification
(Block 2)
Pixel ClassificationInput
Output
Pixel Classification•Mutual Consistency Check• Requires that the disparity value from the left and
right disparity maps are consistent, i.e.,
• Not Pass : occluded pixel• Pass : unoccluded pixel => Correlation Confidence Measure
• Correlation Confidence• Based on how distinctive the highest peak in a
pixel's correlation profile is
• : the cost for the best disparity value• : the cost for the second best disparity value
Pixel Classification
If > αs stableElse unstable
0.04
dx = arg min CL,x (yL, yR)
Iterative Refinement
(Block 3)
• Goal: to propagate information from the stable pixels to the unstable and the occluded pixels
Input
Iteration
• Color Segmentation• Color segments in IL are extrated by Mean Shift[6]
Iterative Refinement
[6] D. Comaniciu and P. Meer, “Mean Shift: A Robust Approach Toward Feature Space Analysis,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 24, pp. 603-619, 2002.
• Plane Fitting
• Using the disparity values for the stable pixels in each color segment
• Disparity values are taken from the current hypothesis for the left disparity map DL
(i). (Initial: DL(0))
• The plane-fitted depth map is used as a regularization for the new disparity estimation.
Iterative Refinement
• Plane Fitting• Using RANSAC[10]
• Iterates until the plane parameters converge
Iterative Refinement[10] M.A. Fischler and R.C. Bolles, “Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography,” Comm. ACM, vol. 24, pp. 381-395, 1981.
• Plane Fitting output : D(i)
• The ratio of stable pixels of each segment:• If Ratio > ȠS
• Stable pixels: D(i)
• Unstable, Occluded pixels: D(i)
• If Ratio ≤ ȠS
• All pixels : D(i)
Iterative Refinementpf
L
pf
pf
0.7
Iteration
• Absolute Difference:
• D(i+1) : New Disparity Map• D(i) : Plane-fitted Disparity Map
• Data Term:
Iterative Refinement
L
2.0
pf
0.5
0.05
• The core energy minimization of our algorithm is carried out via the hierarchical BP algorithm.
Belief Propagation
Total Energy for Pixel X
Data Term Smooth Term
•Max-Product BP[25] :
• : Message vector passed from pixel X to one of its neighbors Y
Max-Product Belief Propagation
Data Term
Jump Cost
[25] Y. Weiss and W. Freeman, “On the Optimality of Solutions of the Max-Product Belief Propagation Algorithm in Arbitrary Graphs,” IEEE Trans. Information Theory, vol. 2, pp. 732-735, 2001.
Max-Product Belief Propagation
xY
• Jump Cost:
• dx : Disparity of pixel X
• d : Disparity of pixel Y (X’s neighbor)
• αbp : The number of disparity levels / 8
• ρs : 1 – (normalized average color difference)
• ρbp : The rate of increase in the cost
Max-Product Belief Propagation
Disparity Differenceof pixel X and its neigbor Y
1
• Total Energy for pixel X:
• Finally the label d that minimizes the total Energy for each pixel is selected.
Max-Product Belief Propagation
Smooth TemData Tem
• Standard loopy BP algorithm is too slow.
• Hierarchical BP[9] runs much faster while maintaining comparable accuracy.
• Works in a coarse-to-fine manner
Hierarchical Belief Propagation
[9] P.F. Felzenszwalb and D.P. Huttenlocher, “Efficient Belief Propagation for Early Vision,” Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, vol. 1, pp. 261-268, 2004.
Hierarchical Belief PropagationCoarser(Level 1)
Finer(Level 0)
• A large number of iterations is required to guarantee convergence in a standard BP algorithm.
• Fast-Converging BP effectively removes the redundant computation.
• Only updating the pixels that have not yet converged (value bigger than ȠZ )
Fast-Converging Belief Propagation
0.1
Fast-Converging Belief Propagation
Depth Enhancement
• To reduce the discontinuities caused by the quantization
• Sub-pixel Estimation algorithm is proposed.
• Cost Function:
Depth Enhancement
• The depth with the minimum of the cost function:
• d: the discrete depth with the minimal cost• d+: d+1• d- : d-1
• Replace each value with the average of those values that are within one disparity over a 9 x 9 window
Depth Enhancement
Experiments
ExperimentsParameter Settings Used Throughout:
ExperimentsParameter Settings Used Throughout:
ExperimentsResults on the Middlebury Data Sets with Error Threshold 1
nonocc : The subset of the nonoccluded pixelsdisc : The subset of the pixels near the occluded areas. all : The subset of the pixels being either nonoccluded or half-occluded
Error%
ExperimentsResults on the Middlebury Data Sets with Error Threshold 0.5
Color-Weighted Correlation Voume :
Initial Hierarchical BP:
Plane fitting:
Integer-Based Disparity Map:
Depth Enhancement:
Ground Trueh:
Conclusion
Conclusion• Propose a stereo model based on • energy minimization• color segmentation• plane fitting• repeated application of hierarchical BP• depth enhancement
• A fast converging BP approach is proposed.• Preserves the same accuracy as the standard BP• The runtime is sublinear to the number of iterations.
Conclusion• The algorithm is currently outperforming the
other algorithms on the Middlebury data sets on average.
• There’s space for Improvement:• Only refined the disparity map for the reference
image• [19] suggests that, by generating a good disparity
map for the right image, the occlusion constraints can be extracted more accurately.
J. Sun, Y. Li, S.B. Kang, and H.-Y. Shum, “Symmetric Stereo Matching for Occlusion Handling,” Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, vol. 2, pp. 399-406, 2005.