Multiple View Geometryin Computer Vision
Marc Pollefeys
Comp 290-089
Multiple View Geometry
a
bc
A
(a,b) A
(a,b) c
f(a,b,c)=0
a
b
c
(a,b,c) (a,b,c)(reconstruction)
(calibration)
(transfer)
Course objectives
• To understand the geometric relations between multiple views of scenes.
• To understand the general principles of parameter estimation.
• To be able to compute scene and camera properties from real world images using state-of-the-art algorithms.
Relation to other vision/image courses
• Focuses on geometric aspects• No image processing
• Comp 254: Image Processing an AnalysisMostly orthogonal to this course,
complementary
• Comp 256: Computer Vision (fall 2003)Will be much broader, based on new book:“Computer Vision: a modern approach”David Forsyth and Jean Ponce
Material
Textbook:Multiple View Geometry in Computer Visionby Richard Hartley and Andrew ZissermanCambridge University Press
Alternative book:The Geometry from Multiple Imagesby Olivier Faugeras and Quan-Tuan LuongMIT Press
On-line tutorial:http://www.cs.unc.edu/~marc/tutorial.pdf
http://www.cs.unc.edu/~marc/tutorial/
Learning approach
• read the relevant chapters of the books and/or reading assignements before the course.
• In the course the material will then be covered in detail and motivated with real world examples and applications.
• Small hands-on assignements will be provided to give students a "feel" of the practical aspects.
• Students will also read and present some seminal papers to provide a complementary view on some of the covered topics.
• Finally, there will also be a project where students will implement an algorithm or approach using concepts covered by the course.
Grade distribution
• Class participation: 20% • Hands-on assignments: 10% • Paper presentation: 10% • Implementation assignment/project: 40% • Final: 20%
Applications
• MatchMovingCompute camera motion from video (to register real an virtual object
motion)
Applications
• 3D modeling
Content
• Background: Projective geometry (2D, 3D), Parameter estimation, Algorithm evaluation.
• Single View: Camera model, Calibration, Single View Geometry.
• Two Views: Epipolar Geometry, 3D reconstruction, Computing F, Computing structure, Plane and homographies.
• Three Views: Trifocal Tensor, Computing T.• More Views: N-Linearities, Multiple view
reconstruction, Bundle adjustment, auto-calibration, Dynamic SfM, Cheirality, Duality
Multiple View Geometry course schedule(tentative)
Jan. 7, 9 Intro & motivation Projective 2D Geometry
Jan. 14, 16
(no course) Projective 2D Geometry
Jan. 21, 23
Projective 3D Geometry Parameter Estimation
Jan. 28, 30
Parameter Estimation Algorithm Evaluation
Feb. 4, 6 Camera Models Camera Calibration
Feb. 11, 13
Single View Geometry Epipolar Geometry
Feb. 18, 20
3D reconstruction Fund. Matrix Comp.
Feb. 25, 27
Structure Comp. Planes & Homographies
Mar. 4, 6 Trifocal Tensor Three View Reconstruction
Mar. 18, 20
Multiple View Geometry MultipleView Reconstruction
Mar. 25, 27
Bundle adjustment Papers
Apr. 1, 3 Auto-Calibration Papers
Apr. 8, 10
Dynamic SfM Papers
Apr. 15, 17
Cheirality Papers
Apr. 22, 24
Duality Project Demos
Fast Forward!
• Quick overview of what is coming…
Background
La reproduction interdite (Reproduction Prohibited), 1937, René Magritte.
• Points, lines & conics• Transformations
• Cross-ratio and invariants
Projective 2D Geometry
Projective 3D Geometry
• Points, lines, planes and quadrics
• Transformations
• П∞, ω∞ and Ω ∞
Estimation
How to compute a geometric relation from correspondences, e.g. 2D trafo
• Linear (normalized), non-linear and Maximum Likelihood Estimation
• Robust (RANSAC)
Evaluation and error analysis
How good are the results we get• Bounds on performance
• Covariance propagation & Monte-Carlo estimation
residual
error
JΣJΣ XP1T
Single-View Geometry
The Cyclops, c. 1914, Odilon Redon
Camera Models
X.xλor
1
10
t
1
1
1
11
λ3
PR
Z
Y
X
pf
pf
y
x
y
x
T
Mostly pinhole camera model
but also affine cameras, pushbroom camera, …
Camera Calibration
• Compute P given (m,M)(normalized) linear, MLE,…
• Radial distortion
More Single-View Geometry
• Projective cameras and planes, lines, conics and quadrics.
• Camera center and camera rotation
• Camera calibration and vanishing points, calibrating conic and the IAC
** CPPQ T
coneQCPP T
Single View Metrology
Antonio Criminisi
Two-View Geometry
The Birth of Venus (detail), c. 1485, Sandro Botticelli
Epipolar Geometry
Fundamental matrix Essential matrix 0xx' FT 0x̂[t]'x̂ RT
'PP,F 'PP,E
Two-View Reconstruction
Epipolar Geometry Computation
(normalized) linear:
minimal:
MLE:
RANSAC… and automated two view matching
0xx' FT
0λdet 21 FF
0).1,,,',',',',','( fyxyyyxyxyxxx
0x̂'x̂for which
'x̂,x'x̂,x 2
i
2
ii
iiii dd
FT
RectificationWarp images to simplify epipolar geometry
Structure Computation
• Points: Linear, optimal, direct optimal
• Also lines and vanishing points
Planes and Homographies
TT )1,v(π Relation between plane and H given P and P’Relation between H and F, H from F, F from H
The infinity homography H∞
Three-View Geometry
The Birth of Venus (detail), c. 1485, Sandro Botticelli
Trifocal Tensor
Three View Reconstruction
• (normalized) linear• minimal (6 points)• MLE (Gold Standard)
Multiple-View Geometry
The Birth of Venus (detail), c. 1485, Sandro Botticelli
Multiple View Geometry
wxyzpqrs
lszkryjqxipwiiii Qxxxx 0
Quadrifocal tensor
wxyzpqrs
srqp Qllll 081 parameters, but only 29 DOF!
Multiple View Reconstruction
• Affine factorization• Projective factorization
n
mn
mm
n
n
XXX
P
P
P
xxx
xxx
xxx
21
m
2
1
21
222
21
112
11
Multiple View Reconstruction
• Sequential reconstruction
Bundle AdjustmentMaximum Likelyhood Estimation for complete structure and motion
U1
U2
U3
WT
W
V
P1 P2 P3 M
J JJN T
12xm 3xn(in general
much larger)
m
k
n
iikd
1 1
2
ki X̂P̂,x
Bundle AdjustmentMaximum Likelyhood Estimation for complete structure and motion
m
k
n
iikd
1 1
2
ki X̂P̂,x
WT V
U-WV-1WT
NI0
WVI 1
11xm 3xn
Bundle adjustment
No bundle adjustment
Bundle adjustment needed to avoid drift of virtual objectthroughout sequence
Bundle adjustment (including radial distortion)
*
*
projection
constraints
Tii
Tiii Ωω KKPP
Auto-calibration
Tijiijj
HH ωω
Dynamic Structure from Motion
3
2
1
321
223
222
221
113
112
111
21
222
21
112
11
S
S
S
PPP
PPP
PPP
xxx
xxx
xxx
mmmmmmmn
mm
n
n
lll
lll
lll
Cheirality
iiiiii
Xof hullconvex preserves
0λ allor 0λ allwith 'XλX
T
T
Oriented projective geometry
Allows to use fact that points are in front of camera• to recover quasi-affine reconstruction• to determine order for image warping• to determine orientation for rectification
with epipoles in images• etc.
Duality
Gives possibility to interchange role of P and X in algorithms
dc
db
da
P
d
c
b
a
WZ
WY
WX
W
Z
Y
X
dc
db
da
XP
TTTT
TTTT
)1,1,1(e,)1,0,0(e,)0,1,0(e,)0,0,1(e
)1,0,0,0(E,)0,1,0,0(E,)0,0,1,0(E,)0,0,0,1(E
4321
4321
Recommended
