The plan for today

Camera matrix Part A) Notation, preprocessing, and basic

concepts. Part B) 4 Stereo Algorithms

Slides are courtesy of Prof. Ronen Basri

Stereo Vision

Objective: 3D reconstruction Input: 2 (or more) images taken with calibrated

cameras Output: 3D structure of scene Steps:

Rectification Matching Depth estimation

Rectification

Image Reprojection reproject image planes onto

common plane parallel to baseline Notice, only focal point of camera

really matters(Seitz)

Rectification

Any stereo pair can be rectified by rotating and scaling the two image planes (=homography)

We will assume images have been rectified so Image planes of cameras are parallel. Focal points are at same height. Focal lengths same.

Then, epipolar lines fall along the horizontal scan lines of the images

Cyclopean Coordinates

Origin at midpoint between camera centers Axes parallel to those of the two (rectified) cameras

( / 2),

( / 2),

( ) ( ),

2 2

l l

r r

l r

l r l r

l r l r l r

f X b fYx y

Z Zf X b fY

x yZ Z

fbx x

Zb x x b y y fb

X Y Zx x x x x x

Disparity

The difference is called “disparity” d is inversely related to Z: greater sensitivity to

nearby points d is directly related to b: sensitivity to small

baseline

l r

fbZ

x x

l rd x x

Main Step: Correspondence Search What to match?

Objects?

More identifiable, but difficult to compute Pixels?

Easier to handle, but maybe ambiguous Edges? Collections of pixels (regions)?

Random Dot Stereogram

Using random dot pairs Julesz showed that recognition is not needed for stereo

Random Dot in Motion

Finding Matches

SSD error

disparity

1D Search More efficient Fewer false matches

Finding Matches

Under what conditions pixels can be matched? Ignoring specularities, we can assume that

matching pixels have the same brightness (constant brightness assumption)

Still, changes in gain and sensitivity may change the values of pixels

Possible solutions

Use larger windows

Constraint the search

Other metric e.g., Normalized correlation

Window Size

W = 3 W = 20

Constraining the Search

Restrict search to epipolar lines (1D search)

Enforce ordering

Problem: not always true

Enforce smoothness

Problem: discontinuities at object boundaries

Ordering

Ordering

Summary basic ideas

Restrict search to epipolar lines (1D search) Use larger elements (larger windows, edges,

regions)Problem: large elements may be distorted

Enforce orderingProblem: not always true

Other similarity measures (e.g., Normalized correlation)

Enforce smoothnessProblem: discontinuities at object boundaries

Comparison of Stereo Algorithms

D. Scharstein and R. Szeliski. "A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms," International Journal of Computer Vision, 47 (2002), pp. 7-42.

Ground truthScene

Scharstein and Szeliski

Results with window correlation

Window-based matching(best window size)

Ground truth

Graph Cuts

Ground truthGraph cuts

Correspondence as Optimization Most stereo algorithms attempt to minimize a

functional that usually consists of two terms:

where

- penalizes for quality of a match (unary)

- penalizes non smooth (or even non fronto-parallel) reconstructions (binary)

Many different optimization approaches were proposed

match data smoothnessE E E

smoothnessEdataE

Part B) 4 Stereo Algorithms

1. Dynamic programming

2. Minimal cut/Max flow

3. Space carving

4. Graph cut optimization

?

1D Methods: Dynamic Programming Discretize the 3-D space Find the correct curve at every slice

(A slice = epipolar plane)

Dynamic programming

Find correspondences of each epipolar

line separately

Dynamic programming

Dynamic programming

How do we find the best curve? Assign weight of all edges

insertion

matchdeletion

Dynamic programming

How do we find the best curve? Assign weight of all edges Find shortest path

Dijkstra

insertion

matchdeletion

Results

Dynamic programming

Advantages Simple, efficient Globally optimal

Disadvantages Each slice computed independently

(smoothness is not enforced between slices) Problems due to discretization (tilted planes)

Stereo Algorithms

Brief review of 4 algorithms:

1. Dynamic programming

2. Minimal cut/Max flow

3. Space carving

4. Graph cut optimization

Min Cut/Max Flow

Min Cut/Max Flow

Min Cut/Max Flow

Min Cut/Max Flow

Min Cut/Max FlowMain idea: Lets solve all DP problem

together.

Objective: find the optimal cut using all the slices simultaneously.

Min Cut/Max Flow

Min Cut/Max Flow

Construct a graph: Every voxel (3-D point in space) is a node Every node is connected to its 6 neighbors

Min Cut/Max Flow

Weights on the edges: Data cost: change in pixel value

data

data

Neighbor

In next slice/row

Neighbor

In next slice/row

Neighbor Neighbor

Min Cut/Max Flow

Weights on the edges: Data cost: change in pixel value Smoothness cost: change in depth

smooth

smooth

smooth

smooth

Min Cut/Max Flow

Weights on the edges: Data cost: change in pixel value Smoothness cost: change in depth

data

Min Cut/Max Flow

Add source and sink Find min cut

Source

…

…

Sink

∞

∞

Min Cut/Max Flow

Data penalty

Smoothness penalty

Results

Input Min cut Dynamic programming

Min Cut/Max Flow

Advantages All slices are optimized simultaneously Efficient

Disadvantages Extension to multi-camera is difficult Discretization

Multi-view stereo Every point in space

corresponds to a match in the images

Compute data term for each match

Space Carving

0.5 0.4 0.8 0.9 0.9 0.8 0.9 0.3 0.2

Space Carving

Multi-view stereo Every point in space

corresponds to a match in the images

Compute data term for each match (“photo-consistency”)

Space Carving

Dynamic data term (taking occlusion into account)

Order of sweep is important

Space Carving

Space Carving

Done for all slices simultaneously

Space Carving

Done for all slices simultaneously

Space Carving

Done for all slices simultaneously

Space Carving

Computes a bound on the object, the visual hull More camera views: better result

Space Carving: Results

Space Carving: Results

Space Carving

Advantages True multi-views stereo Handles occlusion

Disadvantages Limited to visual hull Lacks smoothness term Noise may introduce holes,

allowing for noise may thicken shape

Discretization

Graph Cut Optimization

Stereo is a minimization problem

Possible solution: local search (gradient descent) Problem: inefficient, local minima Instead, search larger areas at every iteration

match data smoothnessE E E

Graph Cut Optimization

Construct a graph to represent the problem: Nodes:

Pixels (in first image) k discrete depth values

Edges: From every pixel node to a

depth node (data term) Neighboring nodes (smoothness)

Assign weights corresponding to pixel intensities to get a global cost function

pixels

depths

…21 k

Graph Cut Optimization

Objective: Multiway cut Edges:

Every pixel remains connected to one depth node

Edges between neighboring nodes only if they are connected to same depth node

Nodes are assigned the depth that they are connected to

Multiway cut is NP-complete, solve iteratively

…

……21 k3

pixels

depths

Graph Cut Optimization

α-β swap Nodes labeled α or β, (i.e.,

connected to or )

can change their labeling to α or β

Edges between neighbors are updated according to the new labeling

Other edges are not changed Finding best swap = min cut!

α β

…

……21 k3

pixels

depths

Graph Cut Optimization

Example: 1-2 swap

…

…… k

…

…

… k1 2 1 23 3

Graph Cut Optimization

Example: 1-2 swap

…

…

… k1 2 3 … k321

Connect the nodes labeled 1 or 2 to both

labels

Graph Cut Optimization

Example: 1-2 swap… k3

2

1 … k3

2

1

Mark 1 as source and 2 as sink Find minimal cut

Graph Cut Optimization

Example: 1-2 swap

… k3

2

1

… k1 2 3

Erase edges that were on the cut

Result: a new labeling of the 1,2 nodes

Graph Cut Optimization

Start with an arbitrary labeling For every pair {α, β} є {1,…,k}

Find the best α-β swap (minimizing the function) Update the graph (add and erase edges)

Quit when no pair improves the cost function Induce pixel labels

Graph Cuts: Results

…21 k3

Advantages State of the art results Efficient Bound on approximation quality Same technique can be applied to other

problems (e.g., image restoration)Disadvantages Discretization Occlusion Still room for improvement

Graph Cut Optimization

Summary

Stereo vision: shape reconstruction from two or more images

Steps: Rectification Correspondence search Depth estimation

Algorithms: Dynamic programming Min cut/max flow Space carving Graph cuts