Embed Size (px)
Citation preview
ComputerVision
Multiple View Geometry
Marc PollefeysCOMP 256
ComputerVision
2
Last class
Gaussian pyramid
Laplacian pyramid
Gaborfilters
Fouriertransform
ComputerVision
3
Histograms : co-occurrence matrix Not last class …
ComputerVision
4
Texture synthesis [Zalesny & Van Gool 2000]
2 analysis iterations
6 analysis iterations
9 analysis iterations
ComputerVision
5
View-dependent texture synthesis[Zalesny & Van Gool 2000]
ComputerVision
6
Efros & Leung ’99
• Assuming Markov property, compute P(p|N(p))– Building explicit probability tables infeasible
– Instead, let’s search the input image for all similar neighborhoods — that’s our histogram for p
• To synthesize p, just pick one match at random
pp
non-parametricsampling
Input image Synthesizing a pixel
ComputerVision
7
pp
Efros & Leung ’99 extended
• Observation: neighbor pixels are highly correlated
Input image
non-parametricsampling
BB
Idea:Idea: unit of synthesis = block unit of synthesis = block• Exactly the same but now we want P(B|N(B))
• Much faster: synthesize all pixels in a block at once
• Not the same as multi-scale!
Synthesizing a block
ComputerVision
8
Input texture
B1 B2
Random placement of blocks
block
B1 B2
Neighboring blocksconstrained by overlap
B1 B2
Minimal errorboundary cut
ComputerVision
9min. error boundary
Minimal error boundary
overlapping blocks vertical boundary
__ ==22
overlap error
ComputerVision
10
ComputerVision
11
ComputerVision
12
ComputerVision
13
Why do we see more flowers in the distance?
Perpendicular textures
[Leung & Malik CVPR97]
ComputerVision
14
Shape-from-texture
ComputerVision
15
Jan 16/18 - Introduction
Jan 23/25 Cameras Radiometry
Jan 30/Feb1 Sources & Shadows Color
Feb 6/8 Linear filters & edges Texture
Feb 13/15 Multi-View Geometry Stereo
Feb 20/22 Optical flow Project proposals
Feb27/Mar1 Affine SfM Projective SfM
Mar 6/8 Camera Calibration Silhouettes and Photoconsistency
Mar 13/15 Springbreak Springbreak
Mar 20/22 Segmentation Fitting
Mar 27/29 Prob. Segmentation Project Update
Apr 3/5 Tracking Tracking
Apr 10/12 Object Recognition Object Recognition
Apr 17/19 Range data Range data
Apr 24/26 Final project Final project
Tentative class schedule
ComputerVision
16
THE GEOMETRY OF MULTIPLE VIEWS
Reading: Chapter 10.
• Epipolar Geometry
• The Essential Matrix
• The Fundamental Matrix
• The Trifocal Tensor
• The Quadrifocal Tensor
ComputerVision
17
Epipolar Geometry
• Epipolar Plane
• Epipoles
• Epipolar Lines
• Baseline
ComputerVision
18
Epipolar Constraint
• Potential matches for p have to lie on the corresponding epipolar line l’.
• Potential matches for p’ have to lie on the corresponding epipolar line l.
ComputerVision
19
Epipolar Constraint: Calibrated Case
Essential Matrix(Longuet-Higgins, 1981)
ComputerVision
20
Properties of the Essential Matrix
• E p’ is the epipolar line associated with p’.
• ETp is the epipolar line associated with p.
• E e’=0 and ETe=0.
• E is singular.
• E has two equal non-zero singular values (Huang and Faugeras, 1989).
T
T
ComputerVision
21
Epipolar Constraint: Small MotionsTo First-Order:
Pure translation:Focus of Expansion
ComputerVision
22
Epipolar Constraint: Uncalibrated Case
Fundamental Matrix(Faugeras and Luong, 1992)
ComputerVision
23
Properties of the Fundamental Matrix
• F p’ is the epipolar line associated with p’.
• FT p is the epipolar line associated with p.
• F e’=0 and FT e=0.
• F is singular.
T
T
ComputerVision
24
The Eight-Point Algorithm (Longuet-Higgins, 1981)
|F| =1.
Minimize:
under the constraint
2
ComputerVision
25
Non-Linear Least-Squares Approach (Luong et al., 1993)
Minimize
with respect to the coefficients of F , using an appropriate rank-2 parameterization.
ComputerVision
26
Problem with eight-point algorithm
linear least-squares: unit norm vector F yielding smallest residual
What happens when there is noise?
ComputerVision
27
The Normalized Eight-Point Algorithm (Hartley, 1995)
• Center the image data at the origin, and scale it so themean squared distance between the origin and the data points is 2 pixels: q = T p , q’ = T’ p’.
• Use the eight-point algorithm to compute F from thepoints q and q’ .
• Enforce the rank-2 constraint.
• Output T F T’.T
i i i i
i i
ComputerVision
28
Epipolar geometry example
ComputerVision
29
Example: converging cameras
courtesy of Andrew Zisserman
ComputerVision
30
Example: motion parallel with image plane
(simple for stereo rectification)courtesy of Andrew Zisserman
ComputerVision
31
Example: forward motion
e
e’
courtesy of Andrew Zisserman
ComputerVision
32
Fundamental matrix for pure translation
courtesy of Andrew Zisserman
auto-epipolar
ComputerVision
33
Fundamental matrix for pure translation
courtesy of Andrew Zisserman
ComputerVision
34
Trinocular Epipolar Constraints
These constraints are not independent!
ComputerVision
35
Trinocular Epipolar Constraints: Transfer
Given p and p , p can be computed
as the solution of linear equations.
1 2 3
ComputerVision
36
• problem for epipolar transfer in trifocal plane!
image from Hartley and Zisserman
Trinocular Epipolar Constraints: Transfer
There must be more to trifocal geometry…
ComputerVision
37
Trifocal Constraints
ComputerVision
38
Trifocal Constraints
All 3x3 minorsmust be zero!
Calibrated Case
Trifocal Tensor
ComputerVision
39
Trifocal Constraints
Uncalibrated Case
Trifocal Tensor
ComputerVision
40
Trifocal Constraints: 3 Points
Pick any two lines l and l through p and p .
Do it again.2 3 2 3
T( p , p , p )=01 2 3
ComputerVision
41
Properties of the Trifocal Tensor
Estimating the Trifocal Tensor
• Ignore the non-linear constraints and use linear least-squares
• Impose the constraints a posteriori.
• For any matching epipolar lines, l G l = 0.
• The matrices G are singular.
• They satisfy 8 independent constraints in theuncalibrated case (Faugeras and Mourrain, 1995).
2 1 3T i
1i
ComputerVision
42
For any matching epipolar lines, l G l = 0. 2 1 3T i
The backprojections of the two lines do not define a line!
ComputerVision
43
108 putative matches 18 outliers
88 inliers 95 final inliers
(26 samples)
(0.43)(0.23)
(0.19)
TrifocalTensor
Example
courtesy of Andrew Zisserman
ComputerVision
44
additional line matches
TrifocalTensor
Example
images courtesy of Andrew Zisserman
ComputerVision
45
Transfer: trifocal transfer
doesn’t work if l’=epipolar line
image courtesy of Hartley and Zisserman
(using tensor notation)
ComputerVision
46
Image warping using T(1,2,N)(Avidan and Shashua `97)
ComputerVision
47
Multiple Views (Faugeras and Mourrain, 1995)
ComputerVision
48
Two Views
Epipolar Constraint
ComputerVision
49
Three Views
Trifocal Constraint
ComputerVision
50
Four Views
Quadrifocal Constraint(Triggs, 1995)
ComputerVision
51
Geometrically, the four rays must intersect in P..
ComputerVision
52
Quadrifocal Tensorand Lines
ComputerVision
53
Quadrifocal tensor
• determinant is multilinear
thus linear in coefficients of lines !
• There must exist a tensor with 81 coefficients containing all possible combination of x,y,w coefficients for all 4 images: the quadrifocal tensor
wi
wi
yi
yi
xi
xii
Ti MlMlMl Ml
Twi
yi
xi lll ][
s
r
q
p
pqrs
MMMM
Q
4
3
2
1
det
ComputerVision
55
from perspective to omnidirectional cameras
perspective camera(2 constraints / feature)
radial camera (uncalibrated)(1 constraints / feature)
3 constraints allow to reconstruct 3D point
more constraints also tell something about cameras
multilinear constraints known as epipolar, trifocal and quadrifocal constraints
(0,0)
l=(y,-x)
(x,y)
ComputerVision
56
Radial quadrifocal tensor
• Linearly compute radial quadrifocal tensor Qijkl from 15 pts in 4 views
• Reconstruct 3D scene and use it for calibration
(2x2x2x2 tensor)
(2x2x2 tensor)
Not easy for real data, hard to avoid degenerate cases (e.g. 3 optical axes intersect in single point). However, degenerate case leads to simpler 3 view algorithm for pure rotation• Radial trifocal tensor Tijk from 7 points in 3 views
• Reconstruct 2D panorama and use it for calibration
(x,y)
ComputerVision
57
Non-parametric distortion calibration(Thirthala and Pollefeys, ICCV’05)
normalized radius
angle
• Models fish-eye lenses, cata-dioptric systems, etc.
ComputerVision
58
Non-parametric distortion calibration(Thirthala and Pollefeys, ICCV’05)
normalized radiusangle
90o
• Models fish-eye lenses, cata-dioptric systems, etc.
ComputerVision
59
Next class:Stereo
F&PChapter 11
image I(x,y) image I´(x´,y´)Disparity map D(x,y)
(x´,y´)=(x+D(x,y),y)
