Lecture8 - Stereo

8/2/2019 Lecture8 - Stereo

http://slidepdf.com/reader/full/lecture8-stereo 1/67

6.801/866

Stereopsis

T. Darrell

Outline

• Reconstruction

• Rectification

• Early vs. Late

• Window Matching• Edge Matching

• Ordering Constraint

• More views

baseline

Triangulate on two images of the

same point to recover depth.

– Feature matching across views

– Calibrated cameras

Left Right

Last lecture: only needto match features across

epipolar lines…

Geometric Reconstruction

Geometric: midpoint of intersection of Op and O’p’

Multiple views?

Algebraic Reconstruction

Point must satisfy both projection equations:

and therefore:

solve using least squares.

No obvious geometric interpretation; well-defined for multiple views.

Nonlinear Reconstruction

Nonlinear technique:

search for point Q with minimal projection error:

(initialize with one of previous linear methods.)

Rectified image pair

• Choose normalized image

locations

• Common image plane

• Horizontal epipolar lines

• Correspondence search on

scanlines!

Pinhole Camera Model

Focal length f

Virtual

Center of

projection

Pinhole Camera Model

),,( Z Y X P =

Virtual

),( y x p =

f x−=

Basic Stereo Derivations

),,(1 Z Y X P =

B f x x Z ,,,of functionaasfor expressionDerive 21

),,(1 Z Y X P =1O y

Z B f x

Z B X f x

Z X f x

−=+−=−=

Ray from

camera 2

Ray from

camera 1

−=21 :disparitytheDefine

Stereo Vision

baseline

),(),(

B f y x Z =

Z ( x, y) is depth at pixel ( x, y)

d ( x, y) is disparity

Left Right

Matching across scan lines

How do matching errors affect

the depth error?

Outline

Reconstruction

Rectification

• Early vs. Late

• More views

Consider human vision

Differences?

• active foveation

• image plane

• object recognition

Human vision is very accurate at relative depth

judgements.

Julez (1960): which happens first, recognition or

fusion?

Random dot stereograms

Correspondence is ambiguous

Marr-Poggio (1976)

Three constraints:

• compatibility• uniqueness

• continuity

Works well on RDS….but not so well on natural images…

Outline

Reconstruction

Rectification

Early vs. Late

• More views

Correspondence using window matching

Points are highly individually ambiguous…

More unique matches are possible with small regionsof image.

Correspondence using window matching

Left Right

disparity

scanline

Criterion function:

Sum of Squared (Pixel) Differences

Left Right

),( L L y x ),( L L yd x −

∑∈−−=

+≤≤−+≤≤−=

),(),(

)],(),([),,(

:disparityof functionaasdifferenceintensitythemeasurescostSSDThe

},|,{),(

:functionwindowthedefineWe

pixels.of windows byingcorrespondareand

y xW vu R Lr

vd u I vu I d y xC

yv y xu xvu y xW

Image Normalization

• Even when the cameras are identical models, there can bedifferences in gain and sensitivity.

• The cameras do not see exactly the same surfaces, so their

overall light levels can differ.• For these reasons and more, it is a good idea to normalize

the pixels in each window:

pixel Normalized ),(

magnitudeWindow )],([

pixelAverage ),(

),(),(

),(),(),(

y xW vu y xW

I y x I y x I

vu I I

Images as Vectors

“Unwrap”

image to form

vector, usingraster scan order

Left Right

row 1 m

Lwrow 3

m row 2 m

Each window is a vector

in an m2 dimensional

vector space.

Normalization makes

them unit length.

Image windows as vectors

Possible metrics

)(d w R Distance?

Angle?

Correspondence Using Correlation

Left Disparity Map

Images courtesy of Point Grey Research

Foreshortening

Window methods assume fronto-parallel surface at 3-D point.

Initial estimates of the disparity can be used to warp thecorrelation windows to compensate for unequal amounts of foreshortening in the two pictures [Kass, 1987; Devernay

and Faugeras, 1994].

Outline

Reconstruction

RectificationEarly vs. Late

Window Matching

• Edge Matching

• More views

bl i h i d h d

Problems with window methods

Patch too small?

Patch too large?

Can try variable patch size [Okutomi and Kanade],or arbitrary window shapes [Veksler and Zabih]

Should match between physically meaningfulquanties, and at multiple scales [Marr]…

M P i l i h

Marr-Poggio algorithm

Search for edges, a.k.a. “zero crossings”: (more

during edge detection lectures…)

O d i i

Ordering constraint

“It is reasonable to assume that the order of matching image

features along a pair of epipolar lines is the inverse of the

order of the corresponding surface attributes along the

curve where the epipolar plane intersects the observed

object’s boundary.”

This is the so-called ordering constraint introduced by [Baker

and Binford, 1981; Ohta and Kanade, 1985].

O d i t i t

Ordering constraint

O d i t i t

Ordering constraint

DP b d h

DP-based search

Ordering constraint == smooth path through match graph.

Consider path’s cost over a graph whose nodes correspond to pairs of left and right image features, and arcs represent

matches between left and right intensity profile intervals

bounded by the features of the corresponding nodes

S h O C d

Search Over Correspondences

Occluded Pixels

Left scanline

Right scanline

Dis-occluded Pixels

Three cases: – Sequential – cost of match

– Occluded – cost of no match

– Disoccluded – cost of no match

Dynamic Programming

Occluded Pixels

Left scanline

- o c c l u d e d P i x e l s

Ri gh t s c a nl i n

Dynamic programming

yields the optimal

path through grid.This is the best set of

matches that satisfy

the orderingconstraint

Dynamic Programming

1=i 1 1 112Π

22Π2= j2 2 22=i

3 3 33=i

1−t C t C 1+t C

Principle of Optimality for an n-stage assignment problem:

)(maxarg)( 1 iC jC t ijit −Π=

Dynamic Programming

1=i 1 1 1

2)2( =t b2= j2 2 22=i

3 3 33=i

1−t C t C 1+t C

Dynamic Programming

1=i 1 1 1

3= j3 33=i

1−t C t C 1+t C

Dynamic Programming

1=i 1 1 1

1)3( =t b

3= j3 33=i

1−t C t C 1+t C

Dynamic Programming

1)1( =t b1= j1=i 1 1 1

2)2( =t b22=i

3 33=i

1−t C t C 1+t C

Dynamic Programming

1=i 1 1 1

2−T C 1−T C T C

Back-chaining recovers the optimal path and its cost:

...),(),(),(maxarg **

T T T T T T T jT sb s sC C jC s ===−

Stereo Matching with Dynamic

Programming

Occluded Pixels

Left scanline

Di s - o c c l u d e d P i x

Ri gh t s c a nl i n e

Scan across grid

computing optimal

cost for each nodegiven its upper-left

neighbors.

Backtrack from theterminal to get the

optimal path.Terminal

Programming

Occluded Pixels

Left scanline

Scan across grid

computing optimal

neighbors.

Programming

Occluded Pixels

Left scanline

Scan across grid

computing optimal

neighbors.

Programming

Occluded Pixels

Left scanline

Scan across grid

computing optimal

neighbors.

Programming

Occluded Pixels

Left scanline

Scan across grid

computing optimal

neighbors.

Programming

Occluded Pixels

Left scanline

Scan across grid

computing optimal

neighbors.

DP vs Edges

Edges:

• Which method is better? – Edges are more “meaningful” [Marr]…but hard to find!

– Edges tend to fail in dense texture (outdoors)

– Correlation tends to fail in smooth featureless areas

Computing Correspondences

Both methods fail for smooth surfaces

There is currently no good solution to thecorrespondence problem

Outline

Reconstruction

Window Matching

Edge Matching

Ordering Constraint

• More views

Three (calibrated) views

Trinocular Stereo ResultsWhy this camera

arrangement?

Trinocular stereo system available from Point

Gray Research for $5K (circa ’97)

More views reduce ambiguity

Middlebury stereo page

http://www.middlebury.edu/stereo/

Outline

Reconstruction

Window Matching

Edge Matching

Ordering Constraint

More views

[ Most figures adapted from Forsythe and Ponce ]

Lecture8 - Stereo

Documents

Lecture8 euthanasia

Lecture8 Evaluation

lecture8 volumetric stereo - Silvio Savarese...Silvio Savarese Lecture 7 - 3-Feb-15 • Active(stereo • (Structured(lighting • (Depth(sensing • Volumetric(stereo: • Space(carving

DC Lecture8

Mae546 lecture8

Lecture8: 123.101

Lecture8- Ch8

Is2215 lecture8 relational_databases

Lecture8 theromanrepublic

cpsc lecture8

Lecture8 religion

Lecture8: 123.702

Lecture8 rocks

Lecture8 - Linköping Universityusers.isy.liu.se/rt/fredrik/edu/sensorfusion/lecture8.pdf · Lecture8: Whiteboard: I SLAMproblemformulation. I FrameworkforEKF-SLAMandfastSLAM(withPFandMPF)

LECTURE8 (1)

lecture8 clipping

The Diels-Alder reaction - Massey Universitygjrowlan/stereo/lecture8.pdf · The Diels-Alder reaction ... • A class of pericyclic reactions whose stereochemical outcome is governed

Lecture8 nervous system

Lecture8 ID

BIM Lecture8