776 Computer Vision

Jan-Michael Frahm, Enrique DunnSpring 2012

)/,/(),,( ZYfZXfZYX

101

0

0

1

Z

Y

X

f

f

Z

Yf

Xf

Z

Y

X

Pinhole camera model

PXx

Camera calibration• Given n points with known 3D coordinates Xi and

known image projections xi, estimate the camera parameters

? P

Xi

xi

slide: S. Lazebnik

Camera estimation: Linear method

• P has 11 degrees of freedom (12 parameters, but scale is arbitrary)

• One 2D/3D correspondence gives us two linearly independent equations

• Homogeneous least squares• 6 correspondences needed for a minimal solution

0pA 0

P

P

P

X0X

XX0

X0X

XX0

3

2

1111

111

Tnn

TTn

Tnn

Tn

T

TTT

TTT

x

y

x

y

slide: S. Lazebnik

Two-view geometry

• Epipolar Plane – plane containing baseline (1D family)• Epipoles = intersections of baseline with image planes = projections of the other camera center

• Baseline – line connecting the two camera centers

Epipolar geometryX

x x’

slide: S. Lazebnik

• Epipolar Plane – plane containing baseline (1D family)• Epipoles = intersections of baseline with image planes = projections of the other camera center

• Epipolar Lines - intersections of epipolar plane with image planes (always come in corresponding pairs)

• Baseline – line connecting the two camera centers

Epipolar geometryX

x x’

slide: S. Lazebnik

Representations in P2

• Homogeneous coordinates w/o Zero vector• Equivalence classes (u,v,w)T ~ s(u,v,w)T ~ (u/w, v/w,

1)T • Allows for common representation of points and lines

o L = ( a, b, c) T => ax + by + c = 0

• Ideal points correspond to points at “infinity”• Two points can define a line

o L = p1 x p2 = [p1]x p2

• Two intersecting lines define a pointo p = L1 x L2 = [L1]x L2

• Point-Line inclusion testo L.p = 0

Representations in P3

• Homogeneous coordinates w/o Zero vector• Equivalence classes

o (X,Y,Z,w)T ~ s(X,Y,Z,w)T ~ (X/w,Y/w,Z/w, 1)T

• Allows for common representation of points and planeso π = ( a, b, c, d) T => ax + by + cz + d = 0o Plane normal n= (a, b, c) T d is distance from origin

• Ideal points correspond to points at “infinity”• Plane at infinity πinf=(0,0,0,1) contains all ideal points

• Three points can define a planeo π = [(X1 –X3) x (X2-X3) , -X3 T (X1 x X2) ]’o Where X1,X2,X3 are the normlaized non-homogenous 3-vectors

• Three intersecting planes define a pointo X = null([π1T, π2T, π3T])

• Point-Plane inclusion testo πX=0

Homographies in P2

• Linear transformation (non singular 3x3 matrix)

• Homogenous Transformation (8 dof)

Plane Induced Homographies

• Given o Two cameras P=[I |0] and P’ = [A | a]o A plane π=(vT,1) T

• The homography is given by x’=Hx

H = A - avT

Plane Homography for Calibrated

Cameras

• In the calibrated case o Two cameras P=K[I |0] and P’ = K’[R | t]o A plane π=(nT,d) T

• The homography is given by x’=Hx

H = K’(R – tnT/d)K-1

Plane Homography for Calibrated

Cameras

• In the calibrated case o Two cameras P=K[I |0] and P’ = K’[R | t]o A plane π=(nT,d) T

• The homography is given by x’=Hx

H = K’(R – tnT/d)K-1

• For the plane at infinity

H = K’RK-1

The Fundamental Matrix F

P0

m0

L

l1

M

m1

M

P1

Hm0

Epipole e1

F = [e]xH = Fundamental Matrix

1 1 0Tm I 1 0I Fm 1 0 0Tm Fm

1 0Te F

The Fundamental Matrix F• The projective points e1 and (Hm0) define a plane in camera 1

(epipolar plane Πe)

• the epipolar plane intersects the image plane 1 in a line (epipolar line ue)

• the corresponding point m1 lies on line ue: m1Tue= 0

• If the points (e1),(m1),(Hm0) are all collinear, then the colinearity theorem applies: m1

T (e1 x Hm0) = 0.

1 1 0 1 1 0collinearity of , , ( ) 0

0

0

0

T

x

z y

z xx

y x

m e H m m e H m

e e

e e e

e e

1 0 0Tm Fm 1 xF e H

Fundamental Matrix F Epipolar constraint

3 3xF

Estimation of F from image

correspondences• Given a set of corresponding points, solve linearily for the 9 elements

of F in projective coordinates

• since the epipolar constraint is homogeneous up to scale, only eight elements are independent

• since the operator [e]x and hence F have rank 2, F has only 7

independent parameters (all epipolar lines intersect at e)

• each correspondence gives 1 collinearity constraint

=> solve F with minimum of 7 correspondences

for N>7 correspondences minimize distance point-line:

21, 0,

0

( ) min!N

Tn n

n

m Fm

det( ) 0 (rank 2 constraint)F1 0 0Ti im Fm

The eight-point algorithmx = (u, v, 1)T, x’ = (u’, v’, 1)T

Minimize:

under the constraintF33

= 1

2

1

)( i

N

i

Ti xFx

Problem with eight-point algorithm

• Poor numerical conditioning• Can be fixed by rescaling the data

slide: S. Lazebnik

The normalized eight-point algorithm

• Center the image data at the origin, and scale it so the mean squared distance between the origin and the data points is 2 pixels

• Use the eight-point algorithm to compute F from the normalized points

• Enforce the rank-2 constraint (for example, take SVD of F and throw out the smallest singular value)

• Transform fundamental matrix back to original units: if T and T’ are the normalizing transformations in the two images, than the fundamental matrix in original coordinates is TT F T’

(Hartley, 1995)

slide: S. Lazebnik

Comparison of estimation algorithms

8-point Normalized 8-point Nonlinear least squares

Av. Dist. 1 2.33 pixels 0.92 pixel 0.86 pixel

Av. Dist. 2 2.18 pixels 0.85 pixel 0.80 pixel

Example: Converging cameras

slide: S. Lazebnik

Example: Motion parallel to image

plane

Example: Motion perpendicular to image

plane

slide: S. Lazebnik

Example: Motion perpendicular to image

plane

slide: S. Lazebnik

e

e’

Example: Motion perpendicular to image

plane

Epipole has same coordinates in both images.Points move along lines radiating from e: “Focus of expansion”

slide: S. Lazebnik

Epipolar constraint example

slide: S. Lazebnik

X

x x’

Epipolar constraint: Calibrated case

• Assume that the intrinsic and extrinsic parameters of the cameras are known

• We can multiply the projection matrix of each camera (and the image points) by the inverse of the calibration matrix to get normalized image coordinates

• We can also set the global coordinate system to the coordinate system of the first camera. Then the projection matrix of the first camera is [I | 0].

slide: S. Lazebnik

X

x x’

Epipolar constraint: Calibrated case

R

t

The vectors x, t, and Rx’ are coplanar

= RX’ + t

slide: S. Lazebnik

Essential Matrix

Epipolar constraint: Calibrated case

0)]([ xRtx RtExExT ][with0

X

x x’

The vectors x, t, and Rx’ are coplanar slide: S. Lazebnik

1

det( ) 0, ( ) 02

T TE EE E trace EE E 'FKKE T

Cubic constraint

The Essential Matrix E E holds the relative orientation of a calibrated camera pair. It has 5

degrees of freedom: 3 from rotation matrix Rik, 2 from direction of translation e, the epipole.

E has a cubic constraint that restricts E to 5 dof (Nister 2004)

1

det( ) 0, ( ) 02

T TE EE E trace EE E

Relative Pose P from EE holds the relative orientation between 2 calibrated cameras P0 and P1:

0 3 3 3 10 , xxE e R P I P R e

Relative Pose from E and correspondence: Case c is correct relative pose in this case

Given P0 as coordinate frame, the relative orientation of P1 is determined directly

from E up to a 4-fold rotation ambiguity (P1a - P1d). The ambiguity is resolved by

correspondence triangulation: The 3D point M of a corresponding 2D image point pair must be in front of both cameras. The epipolar vector e has norm 1.