Epipolar Geometry and the Fundamental Matrix

Review about Camera Matrix P(from Lecture on Calibration)

• Between the world coordinates X=(Xs, Xs, Xs, 1) of a scene point and the coordinates x=(u’,v’,w’) of its projection, we have the following linear transformation:

• P is a 3x4 matrix that completely represents the mapping from the scene to the image and is therefore called a “camera”.

XPx =

'/'

'/'

wvy

wux

pix

pix

=

=with

Central Projection

s

si

s

si

zy

fy

zx

fx

=

=

Scene point(xs , ys , zs )

Image point(xi , yi , f)x

z

Cf

ycenter of projection

Image plane

=

1000

1000000

s

s

s

zyx

ff

wvu

wvywux ii /,/ ==

If world and image points are represented by homogeneous vectors, central projection is a linear transformation:

Pixel Components

s

si

s

si

zy

fy

zx

fx

=

=

Image point(xi , yi , f)

xpix

ypix

s

ssyiypix

s

ssxixpix

zyzy

kfyyky

zxzx

kfxxkx

00

00

+=+=

+=+=

y

x

=

1000

1000

0

'''

0

0

s

s

s

y

x

zyx

yx

wvu

αα

'/'

'/'

wvy

wux

pix

pix

=

=then

yy

xx

kfkf

==

αα

with

Image center

x0

y0

Transformation uses:

• principal point (x0, y0)

• scaling factors kx and ky Principal point

Internal Camera Parameters

=

1000

1000

'''

0

0

s

s

s

y

x

zyx

yxs

wvu

αα

yy

xx

kf

kf

−=

=

α

αwith

• αx and αy “focal lengths” in pixels

• x0 and y0 coordinates of image center in pixels

•Added parameter s is skew parameter

• K is called calibration matrix. Five degrees of freedom.

•K is a 3x3 upper triangular matrix

'/'

'/'

wvy

wux

pix

pix

=

=

[ ]33 0|IK=

=

01000010

0001

1000

00

0

1000 0

0

0

0

y

xs

y

xs

y

x

y

x

αα

αα

From Camera Coordinatesto World Coordinates

Camera center C

(Xs , Ys , Zs )

Image point(xi , yi , f)

x

z

y

Image plane

Z

X

Y(xs , ys , zs )

OM

Principal point (x0, y0 )

Principal axis Cz

Principal plane (Cx, Cy)

=

11

1S

S

S

S

S

S

ZYX

zyx

T30

TR

• We can use instead of T

Using Camera Center Position in World Coordinates

=

11

~

1S

S

S

S

S

S

ZYX

zyx

T30

CR-R

CR- ~

Linear Transformation from World Coordinates to Pixels

=

11

~

1S

S

S

S

S

S

ZYX

zyx

T30

CR-R

[ ]

=

1'''

s

s

s

zyx

wvu

33 0|IK

[ ]

=

⇒

11

~

'''

S

S

S

ZYX

wvu

T3

33 0CR-R

0|IK

=

⇒

1'''

S

S

S

ZYX

wvu

P XPx =

• Combine camera projection and coordinate transformation matrices into a single matrix P

• P has 11 degrees of freedom:• 5 from triangular calibration matrix K, 3 from R and 3 from

• P is a fairly general 3 x 4 matrix •left 3x3 submatrix KR is non-singular

Properties of Matrix P

[ ] X0

CR-R0|IKx

T3

33

=

1

~

[ ] [ ] [ ]C-|IRCR-R0

CR-R0|I 3T

333

~~1

~==

[ ]XC-|IRKx 3~

=

[ ]C-|IRKP 3~

=

C~

• Further simplification of P:

Cross-Product in Matrix Form• If a = (a1, a2, a3)T is a 3-vector, then one can

define a corresponding skew-symmetric matrix

• The cross-product of 2 vectors a and b can be written

• Matrix [a]x is singular. Its null vector (right or left) is a

••

−−

−=×

00

0

12

13

23

aaaa

aa[a]

b[a]ba ×=×

b a, 0 b[a] bb)(a . b T ∀=⇔=× ×0aF[s]F[a]F c[a] b b[a] c TTT ,∀=⇒−= ××××

Definition of Epipolar Geometry

• Projective geometry between two views

• Independent of scene structure

• Depends only on the cameras’ internal parameters and relative pose of cameras

• Fundamental matrix Fencapsulates this geometry

x’i

xi

0=iTi xFx'

for any pair of corresponding points xi and x’i in the 2 images

Relation between Image Points xi and xi’

• Camera centers C and C’, scene point Xi, image points xi and x’ibelong to a common epipolar plane

• Epipoles e and e’§ On baseline CC’

• Epipolar lines l and l’

π

C’

xi x’i

Xi

C e e'

πl l’

Pencils of Epipolar Lines

e e'C C’

X

Computation of F

• F can be computed from correspondences between image points alone

• No knowledge of camera internal parameters required

• No knowledge of relative pose required

x’i

xi

0=iTi xFx'

for any pair of corresponding points xi and x’i in the 2 images

Finding the Fundamental Matrixfrom Known Cameras P and P’ (Outline)

• Pick up an image point x in camera P• Find one scene point X on ray of x in

camera P• Find the image x’ of X in camera P’• Find epipole e’ as image of C in camera P’

is epipole = P’C• Find epipolar line l’ from e’ to x’ in P’ as

function of x• The fundamental matrix F is defined by

l’=F x• x’ belongs to l’, so x’T l’= 0, so x’T F x = 0• The fundamental matrix F is alternately

defined by x’T F x = 0

C’

x x’

X

C e e'

π

l l’

Finding the Fundamental Matrixfrom Known Cameras P and P’ (Details)

• Pick up an image point x in camera P• Find one scene point on ray from C to x§ Point X = P+x satisfies x = PX

• P+ = PT (P PT )-1, soPX = P PT (P PT )-1 x = x

• Image of this point in camera P’ is x’ = P’X = P’ P+x

• Image of C in camera P’ is epipole e’ = P’C• Epipolar line of x in P’ is

• l’=F x defines F fundamental matrix• x’ belongs to l’, so x’T l’= 0, so x’T F x = 0

xPP'][e'x)P(P')(e'l' +×

+ =×=+

×=⇒ PP'C[P'F ]

C’

x x’

X

C e e'

π

l l’

Properties of Fundamental Matrix F

• Matrix 3X3 (since x’T F x = 0 )• If F is fundamental matrix of camera pair (P, P’) then

the fundamental matrix F’ of camera pair (P’, P) is equal to FT

§ xT F’ x’ = 0 implies x’T F’T x = 0, so F’ = FT

• Epipolar line of x is l’ = F x • Epipolar line of x’ is l = FT x’

More Properties of F

• Epipole e’ is left null space of F, and e is right null space.§ All epipolar lines l’ contains epipole e’, so e’T l’= 0,

i.e. e’T F x = 0 for all x. Therefore e’T F = 0Similarly eT FT x’ = 0 implies eT FT = 0, therefore F e = 0

• F is of rank 2 because F = [e’]xP’P+ and [e’]x is of rank 2• F has 7 degrees of freedom§ There are 9 elements, but scaling is not significant§ Det F = 0 removes one degree of freedom

• Define x as intersection between line l and a line k (k does not pass through epipole e):

• Line e does not pass through point e

• Similarly

Mapping betweenEpipolar Lines (a Homography)

l[k]lkx x=×=

lF[k]x F l' x==

lF[e]x F l' x==

l'][e'Fx' F l xTT ==

e e'C C’

x l’l

k

Retrieving Camera Matrices P and P’from Fundamental Matrix F

• General form of P is• Select world coordinates as camera coordinates of first

camera, select focal length = 1, and count pixels from the principal point. Then P= [ I3 | 0]

• Then P’ = [S F | e’] with S any skew-symmetric matrix is a solution. Proof:§ x’ T F x = XT P’T F P X§ P’T F P = [S F | e’]T F [ I3 | 0 ] is skew-symmetric§ For any skew-symmetric matrix S’ and any X,

XT S’ X = 0

• S = [e’]x is a good choice. Therefore P’=[[e’]x F | e’]

[ ]C-|IRKP 3~

=

[S F | e’]T F [ I3 | 0 ] is skew-symmetric

• e’T F=0 because e’ is left null space of F• FT ST F is skew-symmetric for any F and

any skew-symmetric S

[ ] [ ]

=

=

00' T3

3TT

T3

TT

33T

00FSF

Fe0FSF

0 | I F e' | SF

Essential Matrix E• Specialization of fundamental matrix for

calibrated cameras and normalized coordinates§ x = P X = K [R | T] X§ Normalize coordinates: x0 = K-1 x = [R | T] X

• Consider pair of normalized cameras§ P = [I | 0], P’ = [R | T]

• Then we compute +×= PP'C[P'F ]

××× == T]1] 0 0 [0 T] |[[R C[P' T []]

=+

T3

3

0I

P ER[T] F RPP' , ≡=⇒= ×+

Essential Matrix and Fundamental Matrix

• The defining equation for essential matrix is x0’T E x0 = 0, with § x0 = K-1 x§ x0’ = K’-1 x’

• Therefore x’T K’-T E K-1 x = 0• Comparing with x’T F x = 0, we get

E = K’T F K

Computing Fundamental Matrixfrom Point Correspondences

• The fundamental matrix is defined by the equation

• The equation for a pair of points(x, y, 1) and (x’, y’, 1) is:

• For n point matches:

0=iTi xFx' for any pair of

corresponding points xi and x’i in the 2 images

0

'''

'''

333231

232221

131211

=+++++++

+++

ffyfx

fyfyyfxy

fxfyxfxx

0

1''''''

1'''''' 111111111111

=

= f f A

nnnnnnnnnnnn yxyyyxyxyxxx

yxyyyxyxyxxxMMMMMMMMM

Computing Fundamental Matrixfrom Point Correspondences

• We have a homogeneous set of equationsA f = 0

• f can be determined only up to a scale, so there are 8 unknowns, and at least 8 pointmatchings are needed § hence the name “8 point algorithm”

• The least square solution is the singular vector corresponding the smallest singular value of A, i.e. the last column of V in the SVD A = U D VT

Next Class

• 3D Reconstruction from Multiple Views

References

• Multiple View Geometry in Computer Vision, R. Hartley and A. Zisserman, Cambridge University Press, 2000.