View
221
Download
1
Tags:
Embed Size (px)
Citation preview
Robot Vision SS 2008 Matthias Rüther 1
ROBOT VISION Lesson 6: Shape from Stereo
Matthias Rüther
Slides partial courtesy of Marc Pollefeys Department of Computer Science
University of North Carolina, Chapel Hill
Robot Vision SS 2008 Matthias Rüther 2
Content
Two View Geometry– Epipolar Geometry
– 3D reconstruction
Computing F– Point Correspondences
– Interest Points
– Matching
Robot Vision SS 2008 Matthias Rüther 3
C, C’, x, x’ and X are coplanar
Epipolar Geometry
Robot Vision SS 2008 Matthias Rüther 4
(i) Correspondence geometry: Given an image point x in the first view, how does this constrain the position of the
corresponding point x’ in the second image?
(ii) Camera geometry (motion): Given a set of corresponding image points {xi ↔x’i}, i=1,…,n, what are the cameras P and P’ for the two views?
(iii) Scene geometry (structure): Given corresponding image points xi ↔x’i and cameras P, P’, what is the position of (their pre-image) X in space?
Three Questions
Robot Vision SS 2008 Matthias Rüther 5
What if only C, C’, x are known?
Epipolar Geometry
Robot Vision SS 2008 Matthias Rüther 6
All points on project on l and l’
Epipolar Geometry
Robot Vision SS 2008 Matthias Rüther 7
Family of planes and lines l and l’ Intersection in e and e’
Epipolar Geometry
Robot Vision SS 2008 Matthias Rüther 8
epipoles e,e’= intersection of baseline with image plane = projection of projection center in other image= vanishing point of camera motion direction
an epipolar plane = plane containing baseline (1-D family)
an epipolar line = intersection of epipolar plane with image(always come in corresponding pairs)
Epipolar Geometry
Robot Vision SS 2008 Matthias Rüther 9
Example: Converging Cameras
Robot Vision SS 2008 Matthias Rüther 10
Example: motion parallel with image plane
Robot Vision SS 2008 Matthias Rüther 11
e
e’
Example: forward motion
Robot Vision SS 2008 Matthias Rüther 12
algebraic representation of epipolar geometry
l'x
we will see that mapping is (singular) correlation (i.e. projective mapping from points to lines) represented by the fundamental matrix F
The Fundamental Matrix (F)
Robot Vision SS 2008 Matthias Rüther 13
geometric derivation
xHx' π
x'e'l' FxxHe' π
mapping from 2-D to 1-D family (rank 2)
The Fundamental Matrix (F)
0
0
0
12
13
23
aa
aa
aa
a
Robot Vision SS 2008 Matthias Rüther 14
algebraic derivation
λCxPλX IPP
PP'e'F
xPP'CP'l
(note: doesn’t work for C=C’ F=0)
xP
λX
The Fundamental Matrix (F)
Robot Vision SS 2008 Matthias Rüther 15
correspondence condition
0Fxx'T
The fundamental matrix satisfies the condition that for any pair of corresponding points x↔x’ in the two images
0l'x'T
The Fundamental Matrix (F)
Robot Vision SS 2008 Matthias Rüther 16
F is the unique 3x3 rank 2 matrix that satisfies x’TFx=0 for all x↔x’
(i) Transpose: if F is fundamental matrix for (P,P’), then FT is fundamental matrix for (P’,P)
(ii) Epipolar lines: l’=Fx & l=FTx’
(iii) Epipoles: on all epipolar lines, thus e’TFx=0, x e’TF=0, similarly Fe=0
(iv) F has 7 d.o.f. , i.e. 3x3-1(homogeneous)-1(rank2)
(v) F is a correlation, projective mapping from a point x to a line l’=Fx (not a proper correlation, i.e. not invertible)
The Fundamental Matrix (F)
Robot Vision SS 2008 Matthias Rüther 17
l,l’ epipolar lines, k line not through e l’=F[k]xl and symmetrically l=FT[k’]xl’
lke
kl lFk
e'
(pick k=e, since eTe≠0)
leFl' l'e'Fl T
Epipolar Line Geometry
Robot Vision SS 2008 Matthias Rüther 18
-1-T FHH'F̂ x'H''x̂ Hx,x̂
Derivation based purely on projective concepts
X̂P̂XHPHPXx -1
F invariant to transformations of projective 3-space
X̂'P̂XHHP'XP'x' -1
FP'P,
P'P,F
unique
not unique
canonical form
m]|[MP'0]|[IP
MmF
Invariance under projective transformation
Robot Vision SS 2008 Matthias Rüther 19
F matrix corresponds to P,P’ iff P’TFP is skew-symmetric
X0,FPXP'X TT
Possible choice:
]e'|F][[e'P' 0]|[IP
Canonical representation:
]λe'|ve'F][[e'P' 0]|[IP T
Canonical cameras given F
Robot Vision SS 2008 Matthias Rüther 20
~fundamental matrix for calibrated cameras (remove K)
RtE
0x̂E'x̂ T
FKK'E T
x'K'x̂ x;Kx̂ -1-1
5 d.o.f. (3 for R; 2 for t up to scale)
E is essential matrix if and only iftwo singular values are equal (and third=0)
T0)VUdiag(1,1,E
The Essential Matrix
t]|[RP''P̂0],|[IPˆ t]|[R'P'0],|[K 11 KKPKP
Robot Vision SS 2008 Matthias Rüther 22
C1
C2
l2
P
l1
e1
e2
0m m 1T2 F
Fundamental matrix (3x3 rank 2
matrix)
1. Computable from corresponding points
2. Simplifies matching3. Allows to detect wrong matches4. Related to calibration
Underlying structure in set of matches for rigid scenes
l2
C1m1
L1
m2
L2
M
C2
m1
m2
C1
C2
l2
P
l1
e1
e2
m1
L1
m2
L2
M
l2lT1
Canonical representation:
]λe'|ve'F][[e'P' 0]|[IP T
Epipolar Geometry
Robot Vision SS 2008 Matthias Rüther 23
given xi↔x‘i , compute P,P‘ and Xi
reconstruction problem:
ii PXx x PXi i for all i
without additional information possible up to projective ambiguity
3D reconstruction of cameras and structure
Robot Vision SS 2008 Matthias Rüther 24
(i) Compute F from correspondences(ii) Compute camera matrices from F(iii) Compute 3D point for each pair of
corresponding points
computation of Fuse x‘iFxi=0 equations, linear in coeff. F8 points (linear), 7 points (non-linear), 8+ (least-squares)
computation of camera matricesuse ]λe'|ve'F][[e'P' 0]|[IP T
triangulationcompute intersection of two backprojected rays
Outline of reconstruction
Robot Vision SS 2008 Matthias Rüther 25
iii XHPHPXx S-1S
λt]t'RR'|K[RR'λ0
t'R'-R' t]|K[RPH TTTT
1-S
Reconstruction ambiguity: similarity
Robot Vision SS 2008 Matthias Rüther 26
iii XHPHPXx P-1
P
Reconstruction ambiguity: projective
Robot Vision SS 2008 Matthias Rüther 27
xi↔x‘i
Original scene Xi
Projective, affine, similarity reconstruction = reconstruction that is identical to original up to projective, affine, similarity transformation
Literature: Metric and Euclidean reconstruction = similarity reconstruction
Terminology
Robot Vision SS 2008 Matthias Rüther 28
If a set of point correspondences in two views determine the fundamental matrix uniquely, then the scene and cameras may be reconstructed from these correspondences alone, and any two such reconstructions from these correspondences are projectively equivalent
i111 X,'P,P i222 X,'P,Pii xx -1
12 HPP -112 HPP
12 HXX 0FxFx :except ii
iiiii 22111-1
112 XPxXPHXHPHXP along same ray of P2, idem for P‘2
two possibilities: X2i=HX1i, or points along baseline
key result: allows reconstruction from pair of uncalibrated images
The projective reconstruction theorem
Robot Vision SS 2008 Matthias Rüther 29
Robot Vision SS 2008 Matthias Rüther 30
(i) Projective reconstruction
(ii) Affine reconstruction
(iii) Metric reconstruction
Stratified reconstruction
Robot Vision SS 2008 Matthias Rüther 31
ii HXXE
use control points XEi with known coordinatesto go from projective to metric
Eii XPHx -1(2 lin. eq. in H-1
per view, 3 for two views)
Direct metric reconstruction using ground truth
Robot Vision SS 2008 Matthias Rüther 32
C1
C2
l2
l1
e1
e20m m 1
T2 F
Fundamental matrix (3x3 rank 2
matrix)
1. Computable from corresponding points
2. Simplifies matching3. Allows to detect wrong matches4. Related to calibration
Underlying structure in set of matches for rigid scenes
l2
C1m1
L1
m2
L2
M
C2
m1
m2
C1
C2
l2
l1
e1
e2
m1
L1
m2
L2
M
l2lT1
Canonical representation:
]λe'|ve'F][[e'P' 0]|[IP T
1T
1 lPπ
1T
1T
22 lPPl
0lm 2T
2
111 mel
0m][ePPm 1x1T
1T
2T2
Epipolar Geometry: computation of F
Robot Vision SS 2008 Matthias Rüther 33
0Fxx'T
separate known from unknown
0'''''' 333231232221131211 fyfxffyyfyxfyfxyfxxfx
0,,,,,,,,1,,,',',',',',' T333231232221131211 fffffffffyxyyyxyxyxxx
(data) (unknowns)(linear)
0Af
0f1''''''
1'''''' 111111111111
nnnnnnnnnnnn yxyyyxyxyxxx
yxyyyxyxyxxx
Computation of F: basic equation
Robot Vision SS 2008 Matthias Rüther 34
0Fe'T 0Fe 0detF 2Frank
T333
T222
T111
T
3
2
1
VσUVσUVσUVσ
σσ
UF
SVD from linearly computed F matrix (rank 3)
T222
T111
T2
1
VσUVσUV0
σσ
UF'
FF'-FminCompute closest rank-2 approximation
Imposing the singularity constraint
Robot Vision SS 2008 Matthias Rüther 35
Robot Vision SS 2008 Matthias Rüther 36
0
1´´´´´´
1´´´´´´
1´´´´´´
33
32
31
23
22
21
13
12
11
222222222222
111111111111
f
f
f
f
f
f
f
f
f
yxyyyyxxxyxx
yxyyyyxxxyxx
yxyyyyxxxyxx
nnnnnnnnnnnn
~10000 ~10000 ~10000 ~10000~100 ~100 1~100 ~100
!Orders of magnitude differenceBetween column of data matrix least-squares yields poor results
The NOT normalized 8-point algorithm
Robot Vision SS 2008 Matthias Rüther 37
Transform image to ~[-1,1]x[-1,1]
(0,0)
(700,500)
(700,0)
(0,500)
(1,-1)
(0,0)
(1,1)(-1,1)
(-1,-1)
1
1500
2
10700
2
Least squares yields good results (Hartley, PAMI´97)
The normalized 8-point algorithm
Robot Vision SS 2008 Matthias Rüther 38
Maximum Likelihood Estimation
i
iiii dd 22 'x̂,x'x̂,x
(= least-squares for Gaussian noise)
0x̂F'x̂ subject to T
iXt],|[MP'0],|[IP Parameterize:
Initialize: normalized 8-point, (P,P‘) from F, reconstruct Xi
iiii XP'x̂,PXx̂
Minimize cost using Levenberg-Marquardt(preferably sparse LM, see book)
(overparametrized)
The Gold Standard Algorithm
Robot Vision SS 2008 Matthias Rüther 39
Examples
Robot Vision SS 2008 Matthias Rüther 40
Examples
Robot Vision SS 2008 Matthias Rüther 41
Examples
Robot Vision SS 2008 Matthias Rüther 42
1. Do not use unnormalized algorithms
2. Quick and easy to implement: 8-point normalized
3. Better: enforce rank-2 constraint during minimization
4. Best: Maximum Likelihood Estimation (minimal parameterization, sparse implementation)
Recommendations
Robot Vision SS 2008 Matthias Rüther 43
Extract feature points to relate images
Required properties:– Well-defined
(i.e. neigboring points should all be different)
– Stable across views
(i.e. same 3D point should be extracted as feature for neighboring viewpoints)
The correspondence problem: feature points
Robot Vision SS 2008 Matthias Rüther 44
homogeneous
edge
corner
M should have large eigenvalues
(e.g.Harris&Stephens´88; Shi&Tomasi´94)
Find points that differ as much as possible from all neighboring points
Feature = local maxima (subpixel) of F(1, 2)
Feature points
Robot Vision SS 2008 Matthias Rüther 45
Select strongest features (e.g. 1000/image)
Feature points
Robot Vision SS 2008 Matthias Rüther 46
Evaluate NCC for all features with
similar coordinates
Keep mutual best matchesKeep mutual best matches
Still many wrong matches!Still many wrong matches!
10101010 ,,´´, e.g. hhww yyxxyx
?
Feature matching
Robot Vision SS 2008 Matthias Rüther 47
0.96 -0.40 -0.16 -0.39 0.19
-0.05 0.75 -0.47 0.51 0.72
-0.18 -0.39 0.73 0.15 -0.75
-0.27 0.49 0.16 0.79 0.21
0.08 0.50 -0.45 0.28 0.99
1 5
24
3
1 5
24
3
Gives satisfying results for small image motions
Similarity Example
Robot Vision SS 2008 Matthias Rüther 48
Step 1. Extract features
Step 2. Compute a set of potential matches
Step 3. doStep 3.1 select minimal sample (i.e. 8 matches)
Step 3.2 compute F
Step 3.3 determine inliers
until (#inliers,#samples)<95%
samples#7)1(1
matches#inliers#
#inliers 90% 80% 70% 60% 50%
#samples 5 13 35 106 382
Step 4. Compute F based on all inliersStep 5. Look for additional matchesStep 6. Refine F based on all correct matches
(generate hypothesis)
(verify hypothesis)
RANSAC
Robot Vision SS 2008 Matthias Rüther 49
restrict search range to neighborhood of epipolar line (1.5 pixels)
relax disparity restriction (along epipolar line)
Finding more matches
Robot Vision SS 2008 Matthias Rüther 50
Degenerate cases– Planar scene
– Pure rotation
No unique solution– Remaining DOF filled by noise
– Use simpler model (e.g. homography)
Model selection (Torr et al., ICCV´98, Kanatani, Akaike)
– Compare H and F according to expected residual error (compensate for model complexity)
Degenerate Cases
Robot Vision SS 2008 Matthias Rüther 51
Absence of sufficient features (no texture)
Repeated structure ambiguity
(Schaffalitzky and Zisserman, BMVC‘98)
• Robust matcher also finds Robust matcher also finds support for wrong hypothesissupport for wrong hypothesis• solution: detect repetition solution: detect repetition
More problems