Adaptation from:

Computer Vision : CISC 4/689

Adaptation from:

Prof. James M. Rehg, G.Tech

Computer Vision : CISC 4/689Public Library, Stereoscopic Looking Room, Chicago, by Phillips, 1923

Computer Vision : CISC 4/689Teesta suspension bridge-Darjeeling, India

Computer Vision : CISC 4/689Mark Twain at Pool Table", no date, UCR Museum of Photography

Computer Vision : CISC 4/689Woman getting eye exam during immigration procedure at Ellis

Island, c. 1905 - 1920 , UCR Museum of Phography


Stereo results

Ground truthScene

– Data from University of Tsukuba

(Seitz)


Results with window correlation

Window-based matching(best window size)

Ground truth

(Seitz)


Results with better method

State of the art methodBoykov et al., Fast Approximate Energy Minimization via Graph Cuts,

International Conference on Computer Vision, September 1999.

Ground truth

(Seitz)

http://www.cs.cornell.edu/rdz/Papers/BVZ-iccv99.pdf


Stereo Constraint

• Color constancy– The color of any world points remains constant from image to

image– This assumption is true under Lambertian Model– In practice, given photometric camera calibration and typical

scenes, color constancy holds well enough for most stereo algorithms.


Stereo Constraint

• Epipolar geometry– The epipolar geometry is the fundamental constraint in stereo.

– Rectification aligns epipolar lines with scanlines

Epipolar plane

Epipolar line for pEpipolar line for p’


Stereo Constraint

• Uniqueness and Continuity– Proposed by Marr&Poggio.

– Each item from each image may be assigned at most one disparity value,” and the “disparity” varies smoothly almost everywhere.


Problems with Window-based matching

• Disparity within the window must be constant.

• Bias the results towards frontal-parallel surfaces.

• Blur across depth discontinuities.

• Perform poorly in textureless regions.

• Erroneous results in occluded regions


Cooperative Stereo Algorithm

• Based on two basic assumption by Marr and Poggio:– Uniqueness: at most a single unique match exists for each pixel.

– Continuous: disparity values are generally continuous, i.e., smooth within a local neighborhood.


Disparity Space Image (DSI)

• The 3D disparity space has dimensions row r column c and disparity d. Each element (r, c, d) of the disparity space projects to the pixel (r, c) in the left image and to the (r, c + d) in the right image

• DSI represents the confidence or likelihood of a particular match.


Illustration of DSI

(r, c) slices for different d

(c, d) slice for r = 151


Definition

( , , )nL r c d

0 ( , , )L r c d

( , , )r c d

( , , )r c d

Match value assigned to element (r, c, d) at iteration n

Initial values computed from SSD or NCC

Inhibition area for element (r, c, d)

Local support area for element (r, c, d)


Illustration of Inhibitory and Support Regions

If d is disparity for (r,c),d -1 is for (r,c-1)And d+1 for (r,c+1), etc.

Vertical inhibition is self explanatory.Diagonal basically means, r,c+1in left imagecannot map to r,c+d in right image, i.e, it cannot have disparity of d-1. Similarly,r,c-1 in left image cannot map to r,c+dso it cannot have disparity of d+1.This satisfies uniqueness constraint.


Iterative Updating DSI

1

2

3

4

Questions to ponder:i) What is typical alpha?ii) Do we include inhibitionregion in numerator of Rn?


Explicit Detection of Occlusion

Identify occlusions by examining the magnitude of the converged values in conjunction with the uniqueness

constrain


Summary of Cooperative Stereo

• Prepare a 3D array, (r, c, d): (r, c) for each pixel in the reference image and d for the range of disparity.

• Set initial match values using a function of image intensities, such as normalized correlation or SSD.

• Iteratively update match values using (4) until the match values converge.• For each pixel (r, c), find the element (r, c, d) with the maximum match value.• If the maximum match value is higher than a threshold, output the disparity d, otherwise,

declare a occlusion.

0L

nL


Ordering Constraint

• If an object a is left on an object b in the left image then object a will also appear to the left of object b in the right image

Ordering constraint… …and its failure


Stereo Correspondences

… …Left scanline Right scanline

Match intensities sequentially between two scanlines


Stereo Correspondences

… …Left scanline Right scanline

Match

Match

MatchLeft occlusion Right occlusion


Search Over Correspondences

Three cases:– Sequential – cost of match– Left occluded – cost of no match– Right occluded – cost of no match

Left scanline

Right scanline

Left Occluded Pixels

Right occluded Pixels


Standard 3-move Dynamic Programming for Stereo

Dynamic programming yields the optimal path through grid. This is the best set of matches that satisfy the ordering constraint

Left Occluded Pixels

Left scanline

Right occluded P

ixels

Right scanline

Start

End


Dynamic Programming• Efficient algorithm for solving sequential decision (optimal path) problems.

1

2

3

1

2

3

1

2

3

1t 2t 3t

1i

2i

3i

1

2

3

Tt

…

How many paths through this trellis? T3


Dynamic Programming

1

2

3

1

2

3

1

2

3

1tC tC 1tC

12

22

32

Suppose cost can be decomposed into stages:

jiij state to state from going ofCost

1i

2i

3i

States:


Dynamic Programming

1

2

3

1

2

3

1

2

3

1tC tC 1tC

12

22

32

Principle of Optimality for an n-stage assignment problem

))((min)( 1 iCjC tijit

2j

1i

2i

3i


Dynamic Programming

1

2

3

1

2

3

1

2

3

1tC tC 1tC

2)2( tb

))((minarg)(

))((min)(

1

1

iCjb

iCjC

tijit

tijit

2j

1i

2i

3i


Stereo Matching with Dynamic Programming

Scan across grid computing optimal cost for each node given its upper-left neighbors.Backtrack from the terminal to get the optimal path.

Occluded Pixels

Left scanline

Dis-occluded Pixels

Right scanline

Terminal




Occluded Pixels

Left scanline

Dis-occluded Pixels

Right scanline

Terminal




Occluded Pixels

Left scanline

Dis-occluded Pixels

Right scanline

Terminal




Occluded Pixels

Left scanline

Dis-occluded Pixels

Right scanline

Terminal




Occluded Pixels

Left scanline

Dis-occluded Pixels

Right scanline

Terminal




Occluded Pixels

Left scanline

Dis-occluded Pixels

Right scanline

Terminal




For each (i,j), look for what isOptimal to get to: would it beFrom (i-1,j-1), or (i,j-1), or (i-1,j-1)?Assign large values for left occlusionand right occlusion.

Occluded Pixels

Left scanline

Dis-occluded Pixels

Right scanline

Terminal



Pseudo-code describing how to calculate the optimal match



Pseudo-code describing how to reconstruct the optimal

path


Results

Local errors may be propagated along a scan-line and no inter scan-line consistency is enforced.


Assumption Behind Segmentation-based Stereo

• Depth discontinuity tend to correlate well with color edges

• Disparity variation within a segment is small

• Approximating the scene with piece-wise planar surfaces


Segmentation-based stereo

• Plane equation is fitted in each segment based on initial disparity estimation obtained SSD or Correlation

• Global matching criteria: if a depth map is good, warping the reference image to the other view according to this depth will render an image that matches the real view

• Optimization by iterative neighborhood depth hypothesizing


Result


Another Result

Documents

Adaptation from: