lecture5 epipolar geometry - Stanford UniversityLecture 5 EpipolarGeometry. Silvio Savarese Lecture...

Lecture 5 -Silvio Savarese 24-Jan-18

ProfessorSilvioSavareseComputationalVisionandGeometryLab

Lecture5Epipolar Geometry

Lecture 5 -Silvio Savarese 24-Jan-18

• Whyisstereouseful?• Epipolar constraints• Essentialandfundamentalmatrix• EstimatingF• Examples

Lecture5Epipolar Geometry

Reading:[AZ] Chapter:4“Estimation– 2DperspectivetransformationsChapter:9“Epipolar GeometryandtheFundamentalMatrixTransformation”Chapter:11“ComputationoftheFundamentalMatrixF”

[FP] Chapter:7“Stereopsis”Chapter:8“StructurefromMotion”

Pinhole perspective projectionRecovering structure from a single view

C

Ow

Pp

Calibration rigScene

Camera K

From calibration rig

From points and lines at infinity + orthogonal lines and planes ® structure of the scene, K

® location/pose of the rig, K

Knowledge about scene (point correspondences, geometry of lines & planes, etc…

Pinhole perspective projectionRecovering structure from a single view

C

Ow

Pp

Calibration rigScene

Camera K

Why is it so difficult?

Intrinsic ambiguity of the mapping from 3D to image (2D)

Recovering structure from a single view

Intrinsic ambiguity of the mapping from 3D to image (2D)

Courtesy slide S. Lazebnik

Two eyes help!

O1 O2

pp’

Twoeyeshelp!

Thisiscalledtriangulation

K’ =knownK =known

R, T

P = l × "lP

l 'l[Eq. 1]

• FindP*thatminimizesd(p,M P*)+ d(p ',M 'P*)

O1 O2

pp’

P

Triangulation

[Eq. 2]

MP*M’P*

P*

Multi(stereo)-viewgeometry

• Camerageometry:Givencorrespondingpointsintwoimages,findcameramatrices,positionandpose.

• Scenegeometry:Findcoordinatesof3Dpointfromitsprojectioninto2ormultipleimages.

• Correspondence:Givenapointp inoneimage,howcanIfindthecorrespondingpointp’ inanotherone?

• Epipolar Plane • Epipoles e, e’

• Epipolar Lines• Baseline

Epipolar geometry

O1 O2

p’

P

p

e e’

= intersections of baseline with image planes = projections of the other camera center

Example of epipolar lines

O1 O2

P

e’p p’e

Example: Parallel image planes

• Baseline intersects the image plane at infinity• Epipoles are at infinity• Epipolar lines are parallel to u axis

u u

v v

Example:ParallelImagePlanes

Example: Forward translation

• The epipoles have same position in both images• Epipole called FOE (focus of expansion)

O2

e

e’

O1

P

- Two views of the same object - Given a point on left image, how can I find the corresponding point on right image?

Epipolar Constraint

pWhere is p’?

Epipolar geometry

O1 O2

P

p p’

Epipolar line 2

Epipolar Constraint

p p’Epipolar line 2

M P =uv1

!

"

###

$

%

&&&= p

O1 O2

pp’

P

R, T

Epipolar Constraint

!M P =!u!v1

"

#

$$$

%

&

'''= p '[Eq. 3] [Eq. 4]

M = K I 0!"

#$

M ' = K ' RT −RTT"#$

%&'

u

v

M = K I 0!"

#$

O1 O2

pp’

P

R, T

Epipolar Constraint

K = K’ are known(canonical cameras)

[ ]0IM =

Kcanonical =1 0 00 1 00 0 1

!

"

###

$

%

&&&

M ' = K ' RT −RTT"#$

%&'

M ' = RT −RTT"#$

%&'[Eq. 5] [Eq. 6]

T

O1

O2

T =O2 inthecamera1referencesystemR istherotationmatrixsuchthatavectorp’ inthecamera2isequaltoRp’ incamera1.

ThecamerasarerelatedbyR,T

p’

Camera 1

Camera 2

O1 O2

pp’

P

R, T

[ ] 0)( =¢´×® pRTpT

Epipolar Constraint

is perpendicular to epipolar planeT × ((R "p )+T ) = T × (R "p )= R !p +Tp’ in first camera reference system is

[Eq. 7]

baba ][0

00

´=úúú

û

ù

êêê

ë

é

úúú

û

ù

êêê

ë

é

--

-=´

z

y

x

xy

xz

yz

bbb

aaaaaa

Cross product as matrix multiplication

O1 O2

pp’

P

R, T

[ ] 0)pR(TpT =¢´× [ ] 0pRTpT =¢××® ´

E = Essential matrix(Longuet-Higgins, 1981)

Epipolar Constraint

[Eq. 8] [Eq. 9]

• l = E p’ is the epipolar line associated with p’ • l’ = ET p is the epipolar line associated with p • E e’ = 0 and ET e = 0 • E is 3x3 matrix; 5 DOF• E is singular (rank two)

Epipolar Constraint

O1 O2

p’

P

p

e e’l l’

pT ⋅ E p ' = 0[Eq. 10]

Epipolar Constraint

O1 O2

pp’

P

[ ]0IKM =

R, T

M ' = K ' RT −RTT"#$

%&'

pc = K−1 p !pc = K '

−1 !p[Eq. 11] [Eq. 12]

O1 O2

pp’

P

Epipolar Constraint

pTc ⋅ T×[ ] ⋅R #pc = 0 [ ] 0pKRT)pK( 1T1 =¢¢××® -´

-

[ ] 0pKRTKp 1TT =¢¢×× -´

- 0pFpT =¢® [Eq. 13]

[Eq.9]

pc = K−1 p

!pc = K '−1 !p

[Eq. 11]

[Eq. 12]

O1 O2

pp’

P

Epipolar Constraint

0pFpT =¢

F = Fundamental Matrix(Faugeras and Luong, 1992)

[ ] 1-´

- ¢××= KRTKF T[Eq. 13]

[Eq. 14]

Epipolar Constraint

O1 O2

p’

P

p

e e’

• l = F p’ is the epipolar line associated with p’ • l’= FT p is the epipolar line associated with p • F e’ = 0 and FT e = 0• F is 3x3 matrix; 7 DOF • F is singular (rank two)

pT ⋅ F p ' = 0

l’l

Why F is useful?

- Suppose F is known- No additional information about the scene and camera is given- Given a point on left image, we can compute the corresponding epipolar line in the second image

l’ = FT pp p’

WhyFisuseful?

• Fcapturesinformationabouttheepipolargeometryof2views+cameraparameters

• MOREIMPORTANTLY: Fgivesconstraintsonhowthescenechangesunderviewpointtransformation(withoutreconstructingthescene!)

• Powerfultoolin:• 3Dreconstruction• Multi-viewobject/scenematching

O1 O2

pp’

P

Estimating FThe Eight-Point Algorithm

0pFpT =¢

(Hartley, 1995)(Longuet-Higgins, 1981)

0pFpT =¢

Estimating F

Let’s take 8 corresponding points

( )11 12 13

21 22 23

31 32 33

', ,1 ' 0

1

F F F uu v F F F v

F F F

æ öæ öç ÷ç ÷ =ç ÷ç ÷ç ÷ç ÷è øè ø

( )

11

12

13

21

22

23

31

32

33

', ', , ', ', , ', ',1 0

FFFF

uu uv u vu vv v u v FFFFF

æ öç ÷ç ÷ç ÷ç ÷ç ÷ç ÷ =ç ÷ç ÷ç ÷ç ÷ç ÷ç ÷ç ÷è ø

[Eq. 13]

[Eq. 14]

!p =!u!v1

"

#

$$$

%

&

'''

p =uv1

!

"

###

$

%

&&&

Estimating F

Estimating F

uiu 'i ,uiv 'i ,ui ,viu 'i ,viv 'i ,vi ,u 'i ,vi ',1( )

F11F12F13F21F22F23F31F32F33

!

"

#############

$

%

&&&&&&&&&&&&&

= 0 [Eq. 14]

Estimating F

Lsq. solution by SVD!• Rank 8 A non-zero solution exists (unique)

• Homogeneous system

• If N>8 F̂

f

1=f

W 0=f

W

1 1 1 1 1 1 1 1 1 1 1 1

2 2 2 2 2 2 2 2 2 2 2 2

3 3 3 3 3 3 3 3 3 3 3 3

4 4 4 4 4 4 4 4 4 4 4 4

5 5 5 5 5 5 5 5 5 5 5 5

6 6 6 6 6 6 6 6 6 6 6 6

7 7 7 7 7 7

' ' ' ' ' ' 1' ' ' ' ' ' 1' ' ' ' ' ' 1' ' ' ' ' ' 1' ' ' ' ' ' 1' ' ' ' ' ' 1' '

u u u v u v u v v v u vu u u v u v u v v v u vu u u v u v u v v v u vu u u v u v u v v v u vu u u v u v u v v v u vu u u v u v u v v v u vu u u v u v

11

12

13

21

22

23

317 7 7 7 7 7

328 8 8 8 8 8 8 8 8 8 8 8

33

0

' ' ' ' 1' ' ' ' ' ' 1

FFFFFFF

u v v v u vF

u u u v u v u v v v u vF

æ öæ öç ÷ç ÷ç ÷ç ÷ç ÷ç ÷ç ÷ç ÷ç ÷ç ÷ç ÷ =ç ÷ç ÷ç ÷ç ÷ç ÷ç ÷ç ÷ç ÷ç ÷ç ÷ç ÷ç ÷è øç ÷

è ø

[Eqs. 15]

0pF̂pT =¢and estimated F may have full rank (det(F) ≠0)

0F̂F =-Find F that minimizes

Subject to det(F)=0

But remember: fundamental matrix is Rank2

Frobenius norm (*)

SVD (again!) can be used to solve this problem

^ ^

(*) Sq. root of the sum of squares of all entries

satisfies:F̂

0F̂F =-Find F that minimizes

Subject to det(F)=0

Frobenius norm (*)

F =Us1 0 00 s2 00 0 0

!

"

####

$

%

&&&&

VT

Us1 0 00 s2 00 0 s3

!

"

####

$

%

&&&&

VT = SVD(F̂)

Where:

[HZ] pag 281, chapter 11, “Computation of F”

Data courtesy of R. Mohr and B. Boufama.

Mean errors:10.0pixel9.1pixel

Problems with the 8-Point Algorithm

- Recall the structure of W:- do we see any potential (numerical) issue?

F1=f,0=fW Lsq solution

by SVD

1 1 1 1 1 1 1 1 1 1 1 1

2 2 2 2 2 2 2 2 2 2 2 2

3 3 3 3 3 3 3 3 3 3 3 3

4 4 4 4 4 4 4 4 4 4 4 4

5 5 5 5 5 5 5 5 5 5 5 5

6 6 6 6 6 6 6 6 6 6 6 6

7 7 7 7 7 7

' ' ' ' ' ' 1' ' ' ' ' ' 1' ' ' ' ' ' 1' ' ' ' ' ' 1' ' ' ' ' ' 1' ' ' ' ' ' 1' '

u u u v u v u v v v u vu u u v u v u v v v u vu u u v u v u v v v u vu u u v u v u v v v u vu u u v u v u v v v u vu u u v u v u v v v u vu u u v u v

11

12

13

21

22

23

317 7 7 7 7 7

328 8 8 8 8 8 8 8 8 8 8 8

33

0

' ' ' ' 1' ' ' ' ' ' 1

FFFFFFF

u v v v u vF

u u u v u v u v v v u vF

æ öæ öç ÷ç ÷ç ÷ç ÷ç ÷ç ÷ç ÷ç ÷ç ÷ç ÷ç ÷ =ç ÷ç ÷ç ÷ç ÷ç ÷ç ÷ç ÷ç ÷ç ÷ç ÷ç ÷ç ÷è øç ÷

è ø

Problems with the 8-Point Algorithm

• Highlyun-balanced(notwellconditioned)• ValuesofWmusthavesimilarmagnitude• ThiscreatesproblemsduringtheSVDdecomposition

0=Wf

HZpag

108

Normalization

IDEA: Transform image coordinates such that the matrix W becomes better conditioned (pre-conditioning)

For each image, apply a transformation T (translation and scaling) acting on image coordinates such that:

• Origin = centroid of image points• Mean square distance of the image points from origin is ~2 pixels

Coordinate system of the image before applying T

Coordinate system of the image after applying T

T

Example of normalization

• Origin = centroid of image points• Mean square distance of the image points from origin is ~2 pixels

Normalization

ii pTq = ii pTq ¢¢=¢

1. Normalize coordinates in images 1 and 2:

2. Use the eight-point algorithm to compute from thecorresponding points q and q’ .

1. Enforce the rank-2 constraint:

2. De-normalize Fq:

i i

ii pTq = ii pTq ¢¢=¢

0qFq qT =¢

The Normalized Eight-Point Algorithm

qF®

'TFTF qT= 0)Fdet( q =

0. Compute T and T’ for image 1 and 2, respectively

F̂q

such that:

With

nor

mal

izat

ion

With

out n

orm

aliz

atio

nMean errors:10.0pixel9.1pixel

Mean errors:1.0pixel0.9pixel

http://danielwedge.com/fmatrix/

The Fundamental Matrix Song

Lecture 5 -Silvio Savarese 24-Jan-18

Nextlecture:Stereosystems

O1 O2

P

e’p p’

e

Example: Parallel image planes

E=?K1=K2 = knownHint :

x

y

z

x parallel to O1O2

R = I T = (T, 0, 0)

u

v

E =

0 −Tz TyTz 0 −Tx−Ty Tx 0

"

#

$$$$

%

&

''''

R =0 0 00 0 −T0 T 0

"

#

$$$

%

&

'''

Essentialmatrixforparallelimages

T = [ T 0 0 ]R = I

[ ] RTE ×= ´

[Eq. 20]

O1

P

p p’

Example: Parallel image planes

x

y

z

horizontal!What are the directions of epipolar lines?

l = E p ' =0 0 00 0 −T0 T 0

"

#

$$$

%

&

'''

u 'v '1

"

#

$$$

%

&

'''=

0−TT v '

"

#

$$$

%

&

'''

l l’

u

v

e’e

u’

v’

Example: Parallel image planes

How are pand p’ related?

pT ⋅ E p ' = 0

P

e’p p’

e

y

z

xO1

u

v

u’

v’

Example: Parallel image planes

( ) ( )T

0 0 0 0p E p 0 1 0 0 0 1 0

0 0 1

uu v T v u v T Tv Tv

T Tv

¢é ù æ ö æ öç ÷ ç ÷ê ú¢ ¢ ¢= Þ - = Þ - = Þ =ç ÷ ç ÷ê úç ÷ ç ÷¢ê úë û è ø è ø

How are p and p’ related?

P

e’p p’

e

y

z

O1x

⇒ v = "v

u

v

u’

v’

Example: Parallel image planes

Rectification: making two images “parallel” Why it is useful? • Epipolar constraint ® v = v’

• New views can be synthesized by linear interpolation

P

e’p p’

e

y

z

O1x

u

v

u’

v’

Application: view morphingS. M. Seitz and C. R. Dyer, Proc. SIGGRAPH 96, 1996, 21-30

Rectification

u

u

u

v

v

v

î2

î1

îS

I1

I2

Is

P

C2

Cs

C1

H1

H2

From its reflection!

Deep view morphingD. Ji, J. Kwon, M. McFarland, S. Savarese, CVPR 2017

Deep view morphingD. Ji, J. Kwon, M. McFarland, S. Savarese, CVPR 2017

Deep view morphingD. Ji, J. Kwon, M. McFarland, S. Savarese, CVPR 2017