Multiple View ReconstructionClass 24

Multiple View GeometryComp 290-089Marc Pollefeys

Content

• Background: Projective geometry (2D, 3D), Parameter estimation, Algorithm evaluation.

• Single View: Camera model, Calibration, Single View Geometry.

• Two Views: Epipolar Geometry, 3D reconstruction, Computing F, Computing structure, Plane and homographies.

• Three Views: Trifocal Tensor, Computing T.• More Views: N-Linearities, Self-

Calibration,Multi View Reconstruction, Bundle adjustment, Dynamic SfM, Cheirality, Duality

Multi-view computation

practical structure and motion recovery from images

• Obtain reliable matches using matching or tracking and 2/3-view relations

• Compute initial structure and motion• sequential structure and motion recovery• hierarchical structure and motion recovery

• Refine structure and motion• bundle adjustment

• Auto-calibrate• Refine metric structure and motion

Sequential structure and motion recovery

Initialize structure and motion from 2 views

For each additional view• Determine pose• Refine and extend structure

Initial structure and motion

eeaFeP

0IPT

x

2

1

Epipolar geometry Projective calibration

012 FmmT

compatible with F

Yields correct projective camera setup(Faugeras´92,Hartley´92)

Obtain structure through triangulation

Use reprojection error for minimizationAvoid measurements in projective space

Compute Pi+1 using robust approach (6-point RANSAC)

Extend and refine reconstruction

)x,...,X(xPx 11 iii

2D-2D

2D-3D 2D-3D

mimi+1

M

new view

Determine pose towards existing structure

Non-sequential image collections

4.8im/pt64 images

3792

po

ints

Problem:Features are lost and reinitialized as new features

Solution:Match with other close views

For every view iExtract featuresCompute two view geometry i-1/i and matches Compute pose using robust algorithmRefine existing structureInitialize new structure

Relating to more views

Problem: find close views in projective frame

For every view iExtract featuresCompute two view geometry i-1/i and matches Compute pose using robust algorithmFor all close views k

Compute two view geometry k/i and matchesInfer new 2D-3D matches and add to list

Refine pose using all 2D-3D matchesRefine existing structureInitialize new structure

Refining and extending structure

• Refining structure

• Extending structureTriangulation

M~

P

1

3

(Iterative linear)

(Hartley&Sturm,CVIU´97)

•Initialize motion•Initialize structure•For each additional view

•Determine pose of camera•Refine and extend structure

•Refine structure and motion

Structure and motion: example

190 images

7000

poin

ts

Input sequence

Viewpoint surface mesh calibration demo

ULM

demo

Hierarchical structure and motion recovery

• Compute 2-view• Compute 3-view• Stitch 3-view reconstructions• Merge and refine reconstruction

FT

H

PM

Stitching 3-view reconstructions

Different possibilities1. Align (P2,P3) with (P’1,P’2) -1

23-1

12H

HP',PHP',Pminarg AA dd

2. Align X,X’ (and C’C’) j

jjAd HX',XminargH

3. Minimize reproj. error

jjj

jjj

d

d

x',HXP'

x,X'PHminarg 1-

H

4. MLE (merge) j

jjd x,PXminargXP,

Refining structure and motion

• Minimize reprojection error

• Maximum Likelyhood Estimation (if error zero-mean

Gaussian noise)• Huge problem but can be solved

efficiently (Bundle adjustment)

m

k

n

iikD

ik 1 1

2

kiM̂,P̂

M̂P̂,mmin

Non-linear least-squares

• Newton iteration• Levenberg-Marquardt • Sparse Levenberg-Marquardt

(P)X f (P)X argminP

f

Newton iteration

Taylor approximation

J)(P )(P 00 ffP

XJ

Jacobian

)(PX 1f

JJ)(PX)(PX 001 eff

0T-1T

0 JJJJ ee

i1i PP 0T-1T JJJ e

01-T-11-T JJJ e

normal eq.

Levenberg-Marquardt

0TT JNJJ e

0TJN' e

Augmented normal equations

Normal equations

J)λdiag(JJJN' TT

30 10λ

10/λλ :success 1 ii

ii λ10λ :failure solve again

accept

small ~ Newton (quadratic convergence)

large ~ descent (guaranteed decrease)

Levenberg-Marquardt

Requirements for minimization• Function to compute f

• Start value P0

• Optionally, function to compute J(but numerical derivation ok too)

Sparse Levenberg-Marquardt

• complexity for solving• prohibitive for large problems

(100 views 10,000 points ~30,000 unknowns)

• Partition parameters• partition A • partition B (only dependent on A and itself)

typically A contains camera parameters,and B contains 3D points

0T-1 JN' e3N

Sparse bundle adjustment

residuals:normal equations:

withJJT eJT

Sparse bundle adjustment

normal equations:

modified normal equations:

solve in two parts:

Sparse bundle adjustment

U1

U2

U3

WT

W

V

P1 P2 P3 M

Jacobian of has sparse block structure

J JJN T

12xm 3xn(in general

much larger)

im.pts. view 1

m

k

n

iikD

1 1

2

ki M̂P̂,m

Needed for non-linear minimization

Sparse bundle adjustmentEliminate dependence of camera/motion parameters on structure parametersNote in general 3n >> 11m

WT V

U-WV-1WT

NI0WVI 1

11xm 3xn

Allows much more efficient computations

e.g. 100 views,10000 points,

solve 1000x1000, not 30000x30000Often still band diagonaluse sparse linear algebra algorithms

Sparse bundle adjustment

normal equations:

modified normal equations:

solve in two parts:

Sparse bundle adjustment

• Covariance estimation

-1WVY

1a

Tb

1-a

VYY

WWVU

Yaab -

Degenerate configurations

• Camera resectioning• Two views• More views

(H&Z Ch.21)

Camera resectioning• Cameras as points

• 2D case – Chasles’ theorem

Ambiguity for 3D camerasTwisted cubic (or less) meeting lin. subspace(s) (degree+dimension<3)

Ambiguous two-view reconstructions

Ruled quadric containing both scene points and camera centers alternative reconstructions exist for which the reconstruction of points located off the quadric are not projectively equivalent• hyperboloid 1s• cone• pair of planes• single plane + 2 points• single line + 2 points

Multiple view reconstructions

• Single plane is still a problem

• Hartley and others looked at 3 and more view critical configurations, but those are rather exotic and are not a problem in practice.

Next class: Dynamic structure from motion