Lecture9:StereoandMVS

CSE152:ComputerVisionManmohanChandraker

Recap

epipolarlines

Disparity

(x1,y1) (x2,y1)

x2-x1=thedisparityofpixel(x1,y1)

Twoimagescapturedbyapurelyhorizontal translatingcamera(rectifiedstereopair)

Estimatingthedisparityisequivalenttoestimatingdepth.

Depth from disparity

f

x x’

baseline

z

C C’

X

f

Your basic stereo algorithm

For each epipolar lineFor each pixel in the left image

• compare with every pixel on same epipolar line in right image• pick pixel with minimum match cost

Improvement: match windows

Popular matching scores

• SSD (Sum Squared Distance)

• NCC (Normalized Cross Correlation)

where

• What advantages might NCC have over SSD?

Plane sweep stereoRe-order (pixel and disparity) evaluation loops

for every pixel, for every disparityfor every disparity for every pixelcompute cost compute cost

Stereo matching framework1. For every disparity, compute raw matching

costs

Why use a robust function?• occlusions, other outliers

Can also use alternative match criteria.

Stereo matching framework2. Aggregate costs spatially

• Here, we are using a box filter(efficient moving averageimplementation)

• Can also use weighted average,or other filters

Stereo matching

I(x, y) J(x, y)

y = 141

E(x, y, d) the disparity space image (DSI)x

d

Stereo matching framework3. Choose winning disparity at each pixel

y = 141

E(x, y, d) the disparity space image (DSI)x

d

Stereo matching framework3. Choose winning disparity at each pixel

4. Interpolate to sub-pixel accuracy

d

E(d)

d*

width of a pixel

Choosingthestereobaseline

What’stheoptimalbaseline?

Large Baseline Small Baseline

width of a pixel

Choosingthestereobaseline

What’stheoptimalbaseline?– Toosmall:largedeptherror– Toolarge:difficultsearchproblem

Large Baseline Small Baseline

all of thesepoints projectto the same pair of pixels

Traditional Stereo MatchingAdvantages:

• gives detailed surface estimates• fast algorithms based on moving averages• sub-pixel disparity estimates and confidence

Limitations:• narrow baseline Þ noisy estimates• fails in textureless areas• gets confused near occlusion boundaries

Stereo as energy minimization

What defines a good stereo correspondence?1. Match quality

– Want each pixel to find a good match in the other image2. Smoothness

– If two pixels are adjacent, they should (usually) move about the same amount

Stereo as energy minimization

• Find disparity map d that minimizes an energy function

• Simple pixel or window matching

Match distance between windows I(x, y) and J(x + d(x,y), y)=

Stereo as energy minimizationBetter objective function

match cost smoothness cost

Want each pixel to find a good match in the other image

Adjacent pixels should (usually) move about the same

amount

Stereo as energy minimization

match cost:

smoothness cost:

4-connected neighborhood

8-connected neighborhood

: set of neighboring pixels

Stereo matching 20

Energy minimization1-D example: approximating splines

zx,y

dx,y

Options forsmoothness

RelaxationHow can we get the best solution?Differentiate energy function, set to 0

RelaxationIteratively improve a solution by locally

minimizing the energy: relax to solution

zx,y

dx,ydx-1,y dx+1,y

Graph cutsSolution technique for general 2D problem

Graph cutsTwo different kinds of moves:

Compute best possible match within integer disparity.

Smoothness cost

“Potts model”

L1 distance

How do we choose V?

Depth Map Results

Input image Sum Abs Diff

Graph cuts

CSE576,Spring2008 Stereomatching 27

Stereo evaluation

CSE576,Spring2008 Stereomatching 28

Stereo—best algorithms

Real-time stereo

Used for robot navigation (and other tasks)• Several software-based real-time stereo techniques have

been developed (most based on simple discrete search)

Nomad robot searches for meteorites in Antarticahttp://www.frc.ri.cmu.edu/projects/meteorobot/index.html

Multi-viewStereo

Figures by Carlos Hernandez

Input:calibratedimagesfromseveralviewpointsOutput:3Dobjectmodel

Stereo:anotherviewerror

depth

56 Flickr images taken by 8 photographers

State-of-the-artinbinocularstereo

[Zbontar andLeCun,JMLR2016]