56
Stereo Matching with Color-Weighted Correlation, Hierarchical Belief Propagation, and Occlusion Handling Qingxiong Yang, Student Member, IEEE, Liang Wang, Student Member, IEEE, Ruigang Yang, Member, IEEE, Henrik Stewe´ nius, Member, IEEE, and David Niste´ r, Member, IEEE IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 31, NO. 3, MARCH 2009

Qingxiong Yang, Student Member, IEEE, Liang Wang, Student Member, IEEE,

  • Upload
    rumor

  • View
    66

  • Download
    0

Embed Size (px)

DESCRIPTION

Stereo Matching with Color-Weighted Correlation, Hierarchical Belief Propagation, and Occlusion Handling. Qingxiong Yang, Student Member, IEEE, Liang Wang, Student Member, IEEE, Ruigang Yang, Member, IEEE, Henrik Stewe ´ nius , Member, IEEE, and David Niste ´ r, Member, IEEE. - PowerPoint PPT Presentation

Citation preview

Page 1: Qingxiong  Yang, Student Member, IEEE,  Liang  Wang, Student Member, IEEE,

Stereo Matching with Color-Weighted Correlation, Hierarchical Belief Propagation,and Occlusion HandlingQingxiong Yang, Student Member, IEEE, Liang Wang, Student Member, IEEE,Ruigang Yang, Member, IEEE, Henrik Stewe´ nius, Member, IEEE, and David Niste´ r, Member, IEEE

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 31, NO. 3, MARCH 2009

Page 2: Qingxiong  Yang, Student Member, IEEE,  Liang  Wang, Student Member, IEEE,

Outline• Introduction• System Overview•Methods• Initialization• Pixel Classification• Iterative Refinement• Fast-Converging Belief Propagation

• Depth Enhancement• Experiments• Conclusion

Page 3: Qingxiong  Yang, Student Member, IEEE,  Liang  Wang, Student Member, IEEE,

Introduction

Page 4: Qingxiong  Yang, Student Member, IEEE,  Liang  Wang, Student Member, IEEE,

Introduction• Stereo is one of the most extensively researched

topics in computer vision.

• Energy Minimization framework:• Graph Cut• Belief Propagation(BP)

Page 5: Qingxiong  Yang, Student Member, IEEE,  Liang  Wang, Student Member, IEEE,

Objective(Contribution)• To formulate stereo model with careful handling

of:• Disparity• Discontinuity• Occlusion

• Differs from the normal framework in the final stages of the algorithm

• Outperforms all other algorithms on the average

Page 6: Qingxiong  Yang, Student Member, IEEE,  Liang  Wang, Student Member, IEEE,

System Overview

Page 7: Qingxiong  Yang, Student Member, IEEE,  Liang  Wang, Student Member, IEEE,

•1) Initialization

Page 8: Qingxiong  Yang, Student Member, IEEE,  Liang  Wang, Student Member, IEEE,

•1) Initialization

Page 9: Qingxiong  Yang, Student Member, IEEE,  Liang  Wang, Student Member, IEEE,

•1) Initialization

Page 10: Qingxiong  Yang, Student Member, IEEE,  Liang  Wang, Student Member, IEEE,

•2) Pixel Classification

Page 11: Qingxiong  Yang, Student Member, IEEE,  Liang  Wang, Student Member, IEEE,

•3) Iterative Refinement

Page 12: Qingxiong  Yang, Student Member, IEEE,  Liang  Wang, Student Member, IEEE,

Initialization(Block 1)

Page 13: Qingxiong  Yang, Student Member, IEEE,  Liang  Wang, Student Member, IEEE,

Initialization• Input:• Left Image IL

• Right Image IR

• Output:• Initial Left Disparity Map DL

(0)

• Initial Right Disparity Map DR

• Initial Data Term ED(0)

Image

Color-Weighted Correlation

Correlation Volume

Data Term Initialization

Hierarchical BP

Disparity Map Initialization

ED(0)

CL CR

DL(0) DR

Page 14: Qingxiong  Yang, Student Member, IEEE,  Liang  Wang, Student Member, IEEE,

Image

Color-Weighted Correlation

Correlation Volume

Data Term Initialization

Hierarchical BP

Disparity Map Initialization

Initialization• Color-weighted Correlation

• To build the Correlation Volume

• Makes the match scores less sensitive to occlusion boundaries

• By using the fact that occlusion boundaries most often cause color discontinuities as well

ED(0)

DL(0) DR

CL CR

Page 15: Qingxiong  Yang, Student Member, IEEE,  Liang  Wang, Student Member, IEEE,

Correlation Volume• Color difference Δxy between pixel x and y

(in the same image)

Ic: Intensity of the color channel c

• The weight of pixel x in the support window of y:

Color Difference Spatial Difference

10 21

Page 16: Qingxiong  Yang, Student Member, IEEE,  Liang  Wang, Student Member, IEEE,

Correlation Volume• The Correlation Volume[27]:

• Wx : support window around x• d(yL, yR ) : pixel dissimilarity[1]

• xL , yL : pixels in left image IL

• xR , yR : corresponding pixels in right image IR

• dx : disparity value of pixel XL in IL

weight

Pixels in the window Dissimilarity[1]

dx = arg min CL,x (yL, yR)

weight

xR = xL – dx

yR = yL – dx

Page 17: Qingxiong  Yang, Student Member, IEEE,  Liang  Wang, Student Member, IEEE,

Correlation VolumeDisparity Map Bad Pixel

[27] K.-J. Yoon and I.-S. Kweon, “Adaptive Support-Weight Approach for Correspondence Search,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 28, pp. 650-656, 2006.

[1] S. Birchfield and C. Tomasi, “A Pixel Dissimilarity Measure That Is Insensitive to Image Sampling,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 20, pp. 401-406, 1998.

Page 18: Qingxiong  Yang, Student Member, IEEE,  Liang  Wang, Student Member, IEEE,

Image

Color-Weighted Correlation

Correlation Volume

Data Term Initialization

Hierarchical BP

Disparity Map Initialization

Initialization• Initial Data Term• Total energy   = Data Term + Smooth Term

• Computed from Correlation Volume

• Given an iteration index i = 0 here because it will be iteratively refined

ED(0)

DL(0) DR

CL CR

Page 19: Qingxiong  Yang, Student Member, IEEE,  Liang  Wang, Student Member, IEEE,

Initial Data Term• Initial Data Term:

• Ƞbp : twice the average of correlation volume to exclude the outliers

Correlation Volume0.2

X 2Average

Correlation Volume

Page 20: Qingxiong  Yang, Student Member, IEEE,  Liang  Wang, Student Member, IEEE,

Image

Color-Weighted Correlation

Correlation Volume

Data Term Initialization

Hierarchical BP

Disparity Map Initialization

Initialization• hierarchical Belief Propagation

• Employed with the data term and the reference image

• Resulting in the initial left and right disparity maps DL

(0) and DR

DL(0) DR

CL CR

ED(0)

Page 21: Qingxiong  Yang, Student Member, IEEE,  Liang  Wang, Student Member, IEEE,

Pixel Classification

(Block 2)

Page 22: Qingxiong  Yang, Student Member, IEEE,  Liang  Wang, Student Member, IEEE,

Pixel ClassificationInput

Output

Page 23: Qingxiong  Yang, Student Member, IEEE,  Liang  Wang, Student Member, IEEE,

Pixel Classification•Mutual Consistency Check• Requires that the disparity value from the left and

right disparity maps are consistent, i.e.,

• Not Pass : occluded pixel• Pass : unoccluded pixel      => Correlation Confidence Measure

Page 24: Qingxiong  Yang, Student Member, IEEE,  Liang  Wang, Student Member, IEEE,

• Correlation Confidence• Based on how distinctive the highest peak in a

pixel's correlation profile is

•   : the cost for the best disparity value•   : the cost for the second best disparity value     

Pixel Classification

If > αs stableElse unstable

0.04

dx = arg min CL,x (yL, yR)

Page 25: Qingxiong  Yang, Student Member, IEEE,  Liang  Wang, Student Member, IEEE,

Iterative Refinement

(Block 3)

Page 26: Qingxiong  Yang, Student Member, IEEE,  Liang  Wang, Student Member, IEEE,

• Goal: to propagate information from the stable pixels to the unstable and the occluded pixels

Input

Iteration

Page 27: Qingxiong  Yang, Student Member, IEEE,  Liang  Wang, Student Member, IEEE,

• Color Segmentation• Color segments in IL are extrated by Mean Shift[6]

Iterative Refinement

[6] D. Comaniciu and P. Meer, “Mean Shift: A Robust Approach Toward Feature Space Analysis,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 24, pp. 603-619, 2002.

Page 28: Qingxiong  Yang, Student Member, IEEE,  Liang  Wang, Student Member, IEEE,

• Plane Fitting

• Using the disparity values for the stable pixels in each color segment

• Disparity values are taken from the current hypothesis for the left disparity map DL

(i). (Initial: DL(0))

• The plane-fitted depth map is used as a regularization for the new disparity estimation.

Iterative Refinement

Page 29: Qingxiong  Yang, Student Member, IEEE,  Liang  Wang, Student Member, IEEE,

• Plane Fitting• Using RANSAC[10]

• Iterates until the plane parameters converge

Iterative Refinement[10] M.A. Fischler and R.C. Bolles, “Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography,” Comm. ACM, vol. 24, pp. 381-395, 1981.

Page 30: Qingxiong  Yang, Student Member, IEEE,  Liang  Wang, Student Member, IEEE,

• Plane Fitting output : D(i)

• The ratio of stable pixels of each segment:• If Ratio > ȠS

• Stable pixels: D(i)

• Unstable, Occluded pixels: D(i)

• If Ratio ≤ ȠS

• All pixels : D(i)

Iterative Refinementpf

L

pf

pf

0.7

Page 31: Qingxiong  Yang, Student Member, IEEE,  Liang  Wang, Student Member, IEEE,

Iteration

Page 32: Qingxiong  Yang, Student Member, IEEE,  Liang  Wang, Student Member, IEEE,

• Absolute Difference:

• D(i+1) : New Disparity Map• D(i)   : Plane-fitted Disparity Map

• Data Term:

Iterative Refinement

L

2.0

pf

0.5

0.05

Page 33: Qingxiong  Yang, Student Member, IEEE,  Liang  Wang, Student Member, IEEE,

• The core energy minimization of our algorithm is carried out via the hierarchical BP algorithm.

Belief Propagation

Total Energy for Pixel X

Data Term Smooth Term

Page 34: Qingxiong  Yang, Student Member, IEEE,  Liang  Wang, Student Member, IEEE,

•Max-Product BP[25] :

•      : Message vector passed from pixel X to one of its neighbors Y

Max-Product Belief Propagation

Data Term

Jump Cost

[25] Y. Weiss and W. Freeman, “On the Optimality of Solutions of the Max-Product Belief Propagation Algorithm in Arbitrary Graphs,” IEEE Trans. Information Theory, vol. 2, pp. 732-735, 2001.

Page 35: Qingxiong  Yang, Student Member, IEEE,  Liang  Wang, Student Member, IEEE,

Max-Product Belief Propagation

xY

Page 36: Qingxiong  Yang, Student Member, IEEE,  Liang  Wang, Student Member, IEEE,

• Jump Cost:

• dx : Disparity of pixel X

• d : Disparity of pixel Y (X’s neighbor)

• αbp : The number of disparity levels / 8

• ρs : 1 – (normalized average color difference)

• ρbp : The rate of increase in the cost

Max-Product Belief Propagation

Disparity Differenceof pixel X and its neigbor Y

1

Page 37: Qingxiong  Yang, Student Member, IEEE,  Liang  Wang, Student Member, IEEE,

• Total Energy for pixel X:

• Finally the label d that minimizes the total Energy for each pixel is selected.

Max-Product Belief Propagation

Smooth TemData Tem

Page 38: Qingxiong  Yang, Student Member, IEEE,  Liang  Wang, Student Member, IEEE,

• Standard loopy BP algorithm is too slow.

• Hierarchical BP[9] runs much faster while maintaining comparable accuracy.

• Works in a coarse-to-fine manner

Hierarchical Belief Propagation

[9] P.F. Felzenszwalb and D.P. Huttenlocher, “Efficient Belief Propagation for Early Vision,” Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, vol. 1, pp. 261-268, 2004.

Page 39: Qingxiong  Yang, Student Member, IEEE,  Liang  Wang, Student Member, IEEE,

Hierarchical Belief PropagationCoarser(Level 1)

Finer(Level 0)

Page 40: Qingxiong  Yang, Student Member, IEEE,  Liang  Wang, Student Member, IEEE,

• A large number of iterations is required to guarantee convergence in a standard BP algorithm.

• Fast-Converging BP effectively removes the redundant computation.

• Only updating the pixels that have not yet converged (value bigger than ȠZ )

Fast-Converging Belief Propagation

0.1

Page 41: Qingxiong  Yang, Student Member, IEEE,  Liang  Wang, Student Member, IEEE,

Fast-Converging Belief Propagation

Page 42: Qingxiong  Yang, Student Member, IEEE,  Liang  Wang, Student Member, IEEE,

Depth Enhancement

Page 43: Qingxiong  Yang, Student Member, IEEE,  Liang  Wang, Student Member, IEEE,

• To reduce the discontinuities caused by the quantization

• Sub-pixel Estimation algorithm is proposed.

• Cost Function:

Depth Enhancement

Page 44: Qingxiong  Yang, Student Member, IEEE,  Liang  Wang, Student Member, IEEE,

• The depth with the minimum of the cost function:

• d: the discrete depth with the minimal cost• d+: d+1• d- : d-1

• Replace each value with the average of those values that are within one disparity over a 9 x 9 window

Depth Enhancement

Page 45: Qingxiong  Yang, Student Member, IEEE,  Liang  Wang, Student Member, IEEE,
Page 46: Qingxiong  Yang, Student Member, IEEE,  Liang  Wang, Student Member, IEEE,

Experiments

Page 47: Qingxiong  Yang, Student Member, IEEE,  Liang  Wang, Student Member, IEEE,

ExperimentsParameter Settings Used Throughout:

Page 48: Qingxiong  Yang, Student Member, IEEE,  Liang  Wang, Student Member, IEEE,

ExperimentsParameter Settings Used Throughout:

Page 49: Qingxiong  Yang, Student Member, IEEE,  Liang  Wang, Student Member, IEEE,
Page 50: Qingxiong  Yang, Student Member, IEEE,  Liang  Wang, Student Member, IEEE,

ExperimentsResults on the Middlebury Data Sets with Error Threshold 1

nonocc : The subset of the nonoccluded pixelsdisc : The subset of the pixels near the occluded areas. all : The subset of the pixels being either nonoccluded or half-occluded

Error%

Page 51: Qingxiong  Yang, Student Member, IEEE,  Liang  Wang, Student Member, IEEE,

ExperimentsResults on the Middlebury Data Sets with Error Threshold 0.5

Page 52: Qingxiong  Yang, Student Member, IEEE,  Liang  Wang, Student Member, IEEE,

Color-Weighted Correlation Voume :

Initial Hierarchical BP:

Plane fitting:

Page 53: Qingxiong  Yang, Student Member, IEEE,  Liang  Wang, Student Member, IEEE,

Integer-Based Disparity Map:

Depth Enhancement:

Ground Trueh:

Page 54: Qingxiong  Yang, Student Member, IEEE,  Liang  Wang, Student Member, IEEE,

Conclusion

Page 55: Qingxiong  Yang, Student Member, IEEE,  Liang  Wang, Student Member, IEEE,

Conclusion• Propose a stereo model based on • energy minimization• color segmentation• plane fitting• repeated application of hierarchical BP• depth enhancement

• A fast converging BP approach is proposed.• Preserves the same accuracy as the standard BP• The runtime is sublinear to the number of iterations.

Page 56: Qingxiong  Yang, Student Member, IEEE,  Liang  Wang, Student Member, IEEE,

Conclusion• The algorithm is currently outperforming the

other algorithms on the Middlebury data sets on average.

• There’s space for Improvement:• Only refined the disparity map for the reference

image• [19] suggests that, by generating a good disparity

map for the right image, the occlusion constraints can be extracted more accurately.

J. Sun, Y. Li, S.B. Kang, and H.-Y. Shum, “Symmetric Stereo Matching for Occlusion Handling,” Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, vol. 2, pp. 399-406, 2005.