The Fundamental Matrix F

Computer Vision : CISC 4/689

The Fundamental Matrix F

• Mapping of point in one image to epipolar line in other image x ! l’ is

expressed algebraically by the fundamental matrix F

• Write this as l’ = F x• Since x’ is on l’, by the point-on-line definition we know that

x’T l’ = 0

• Substitute l’ = F x, we can thus relate corresponding points in the

camera pair (P, P’) to each other with the following:

x’T F x = 0

line point


The fundamental matrix F (courtesy: Marc Pollefeys, UNC)

geometric derivation

xHx' π

x'e'l' FxxHe' π

mapping from 2-D to 1-D family (rank 2)Note: rank of skewsymmetric matrix is

even, so 2, so F’s rank is 2


The Fundamental Matrix F• F is 3 x 3, rank 2 (not invertible, in contrast to homographies)

– 7 DOF (homogeneity and rank constraint take away 2 DOF)

• The fundamental matrix of (P’, P) is the transpose FT

from Hartley& Zisserman

x’

NOW, can get implicit equation for any x, which is epipolar line)


Computing Fundamental Matrix

Fundamental Matrix is singular with rank 2

0uFuT

In principal F has 7 parameters up to scale and can be estimatedfrom 7 point correspondences

Direct Simpler Method requires 8 correspondences

(u’ is same as x in the prev. slide, u’ is same as x)


F is the unique 3x3 rank 2 matrix that satisfies x’TFx=0 for all x↔x’

(i) Transpose: if F is fundamental matrix for (P,P’), then FT is fundamental matrix for (P’,P)

(ii) Epipolar lines: l’=Fx & l=FTx’(iii) Epipoles: on all epipolar lines, thus e’TFx=0, x

e’TF=0, similarly Fe=0(iv) F has 7 d.o.f. , i.e. 3x3-1(homogeneous)-1(rank2)(v) F is a correlation, projective mapping from a point x to

a line l’=Fx (not a proper correlation, i.e. not invertible)

i.e, mapping is (singular) correlation (projective mapping from points to lines)

The fundamental matrix F (courtesy: Marc Pollefeys, UNC)


Estimating Fundamental Matrix

0uFuT

Each point correspondence can be expressed as a linear equation

0

1

1

333231

232221

131211

v

u

FFF

FFF

FFF

vu

01

33

32

31

23

22

21

13

12

11

F

F

F

F

F

F

F

F

F

vuvvvvuuvuuu

The 8-point algorithm


The 8-point Algorithm

Lot of squares, so numbers have varied range, from say 1000 to 1. So pre-normalize.And RANSaC!


Computing F: The Eight-point Algorithm

• Input: n point correspondences ( n >= 8)– Construct homogeneous system Ax= 0 from

• x = (f11,f12, ,f13, f21,f22,f23 f31,f32, f33) : entries in F

• Each correspondence gives one equation

• A is a nx9 matrix (in homogenous format)

– Obtain estimate F^ by SVD of A

• x (up to a scale) is column of V corresponding to the least singular value

– Enforce singularity constraint: since Rank (F) = 2

• Compute SVD of F^

• Set the smallest singular value to 0: D -> D’

• Correct estimate of F :

• Output: the estimate of the fundamental matrix, F’

• Similarly we can compute E given intrinsic parameters

0lT

r pFp

TUDVA

TUDVF ˆ

TVUDF' '


Locating the Epipoles from F

• Input: Fundamental Matrix F– Find the SVD of F– The epipole el is the column of V corresponding to the null singular

value (as shown above)– The epipole er is the column of U corresponding to the null singular

value

• Output: Epipole el and er

TUDVF

el lies on all the epipolar lines of the left image

0lT

r pFp

0lT

r eFp

F is not identically zero

True For every pr

0leF

pl pr

P

Ol Orel er

Pl Pr

Epipolar Plane

Epipolar Lines

Epipoles


Special Case: Translation along Optical Axis

• Epipoles coincide at focus of expansion

• Not the same (in general) as vanishing point of scene lines

from Hartley & Zisserman


Finding Correspondences

• Epipolar geometry limits where feature in one image can be in the other image– Only have to search along a line


Simplest Case

• Image planes of cameras are parallel.

• Focal points are at same height.

• Focal lengths same.

• Then, epipolar lines are horizontal scan lines.


We can always achieve this geometry with image rectification

• Image Reprojection– reproject image planes onto common

plane parallel to line between optical centers• Notice, only focal point of camera really matters

(Seitz)


Stereo Rectification

• Rectification – Given a stereo pair, the intrinsic and extrinsic parameters, find the image

transformation to achieve a stereo system of horizontal epipolar lines

– A simple algorithm: Assuming calibrated stereo cameras

p’lp’r

P

Ol Or

X’r

Pl Pr

Z’l

Y’l Y’r

TX’l

Z’r

Stereo System with Parallel Optical AxesEpipoles are at infinity

Horizontal epipolar lines



• Algorithm– Rotate both left and right

camera so that they share the same X axis : Or-Ol = T

– Define a rotation matrix Rrect for the left camera

– Rotation Matrix for the right camera is RrectRT

– Rotation can be implemented by image transformation

pl

pr

P

Ol Or

Xl

Xr

Pl Pr

Zl

Yl

Zr

Yr

R, T

TX’l

Xl’ = T_axis, Yl’ = Xl’xZl, Z’l = Xl’xYl’








pl

pr

P

Ol Or

Xl

Xr

Pl Pr

Zl

Yl

Zr

Yr

R, T

TX’l

Xl’ = T_axis, Yl’ = Xl’xZl, Z’l = Xl’xYl’








Zr

p’lp’r

P

Ol Or

X’r

Pl Pr

Z’l

Y’l Y’r

R, T

TX’l

T’ = (B, 0, 0),

Computer Vision : CISC 4/689Public Library, Stereoscopic Looking Room, Chicago, by Phillips, 1923

Computer Vision : CISC 4/689Teesta suspension bridge-Darjeeling, India

Computer Vision : CISC 4/689Mark Twain at Pool Table", no date, UCR Museum of Photography

Computer Vision : CISC 4/689Woman getting eye exam during immigration procedure at Ellis

Island, c. 1905 - 1920 , UCR Museum of Phography


Stereo matching

• attempt to match every pixel

• use additional constraints


A Simple Stereo System

Zw=0

LEFT CAMERA

Left image:reference

Right image:target

RIGHT CAMERA

Elevation Zw

disparity

Depth Z

baseline


Let’s discuss reconstruction with this geometry before correspondence, because it’s much easier.

OOll OOrr

PP

ppll pprr

TT

ZZ

xxll xxrr

ff

T T is the stereo baselineis the stereo baselined d measures the difference in retinal position between corresponding measures the difference in retinal position between corresponding pointspoints (Camps)

Then given Z, we can compute X and Y.

Disparity:

Similarity of triangles:Xl,f -> X,Zxl,yl=(f X/Z, f Y/Z)Xr,yr=(f (X-T)/Z, f Y/Z)d=xl-xr=f X/Z – f (X-T)/Z

( -ve, +ve, referprevious slide fig.)

X(T-X)

But moving toLeft camera makesIt –(T-X)


Correspondence: What should we match?

• Objects?

• Edges?

• Pixels?

• Collections of pixels?


Extracting Structure

• The key aspect of epipolar geometry is its linear constraint on where a point in one image can be in the other

• By correlation-matching pixels (or features) along epipolar lines and measuring the disparity between them, we can construct a depth map (scene point depth is inversely proportional to disparity)

View 1 View 2 Computed depth mapcourtesy of P. Debevec


Correspondence: Photometric constraint

• Same world point has same intensity in both images.– Lambertian fronto-parallel

– Issues:

• Noise

• Specularity

• Foreshortening


Using these constraints we can use matching for stereo

For each epipolar lineFor each pixel in the left image

• compare with every pixel on same epipolar line in right image

• pick pixel with minimum match cost

• This will never work, so:

Improvement: match windows

(Seitz)


Aggregation• Use more than one pixel

• Assume neighbors have similar disparities*

– Use correlation window containing pixel

– Allows to use SSD, ZNCC, etc.


Comparing Windows: ==??

ff gg

MostMostpopularpopular

(Camps)

For each window, match to closest window on epipolar line in other image.


Compare intensities pixel-by-pixel

Comparing image regions

I(x,y) I´(x,y)

Sum of Square Differences

Dissimilarity measures


Compare intensities pixel-by-pixel

Comparing image regions

I(x,y) I´(x,y)

Zero-mean Normalized Cross Correlation

Similarity measures


Aggregation window sizes

Small windows

• disparities similar

• more ambiguities

• accurate when correct

Large windows

• larger disp. variation

• more discriminant

• often more robust

• use shiftable windows to deal with discontinuities

(Illustration from Pascal Fua)


Window size

W = 3 W = 20

Better results with adaptive window• T. Kanade and M. Okutomi,

A Stereo Matching Algorithm with an Adaptive Window: Theory and Experiment,, Proc. International Conference on Robotics and Automation, 1991.

• D. Scharstein and R. Szeliski. Stereo matching with nonlinear diffusion. International Journal of Computer Vision, 28(2):155-174, July 1998

• Effect of window size

(Seitz)

http://www.cs.cmu.edu/~ph/869/papers/kanade-okutomi.pdf

http://www.cs.cmu.edu/~ph/869/papers/kanade-okutomi.pdf

http://www.cs.washington.edu/education/courses/cse590ss/01wi/notes/papers/ScharsteinSzeliskiIJCV98.pdf


Correspondence Using Window-based matching

SSD error

disparity

Left Right

scanline


Sum of Squared (Pixel) DifferencesLeft Right

Lw Rw

LI RI

),(),(

2

2222

)],(),([),,(

:disparity offunction a as differenceintensity themeasurescost SSD The

},|,{),(

:function window thedefine We

pixels. of windowsby ingcorrespond are and

yxWvuRLr

mmmmm

RL

m

vduIvuIdyxC

yvyxuxvuyxW

mmww

Lw Rw

),( LL yx ),( LL ydx

m

m


Image Normalization• Even when the cameras are identical models, there can be differences in gain and sensitivity.• The cameras do not see exactly the same surfaces, so their overall light levels can differ.• For these reasons and more, it is a good idea to normalize the pixels in each window:

pixel Normalized ),(

),(ˆ

magnitude Window )],([

pixel Average ),(

),(

),(),(

2

),(

),(),(),(

1

yxW

yxWvuyxW

yxWvuyxW

m

mm

m

m

II

IyxIyxI

vuII

vuII


Stereo results

Ground truthScene

– Data from University of Tsukuba

(Seitz)


Results with window correlation

Window-based matching(best window size)

Ground truth

(Seitz)


Results with better method

State of the art methodBoykov et al., Fast Approximate Energy Minimization via Graph Cuts,

International Conference on Computer Vision, September 1999.

Ground truth

(Seitz)

http://www.cs.cornell.edu/rdz/Papers/BVZ-iccv99.pdf

Documents

The Fundamental Matrix F