Embed Size (px)
Citation preview
Lecture 4 - Silvio Savarese 16-Jan-15
Professor Silvio Savarese Computational Vision and Geometry
Lab
Lecture 4 Single View Metrology
Lecture 4 - Silvio Savarese 16-Jan-15
• Review calibration and 2D transformations • Vanishing points and lines • Estimating geometry from a single image • Extensions
Reading: [HZ] Chapter 2 “Projective Geometry and Transformation in 2D” [HZ] Chapter 3 “Projective Geometry and Transformation in 3D” [HZ] Chapter 8 “More Single View Geometry” [Hoiem & Savarese] Chapter 2
Lecture 4 Single View Metrology
Calibration Problem
jC
ii PMP → !"
#$%
&=→
i
ii v
up
In pixels
[ ]TRKM =
!!!
"
#
$$$
%
& −
=
100
0
cot
sin o
o
v
u
Kθ
β
θαα
World ref. system
Calibration Problem
jC
ii PMP → !"
#$%
&=→
i
ii v
up
Need at least 6 correspondences11 unknown
[ ]TRKM =
In pixelsWorld ref. system
Pinhole perspective projectionOnce the camera is calibrated...
[ ]TRKM =
COw
-Internal parameters K are known-R, T are known – but these can only relate C to the calibration rig
Pp
Can I estimate P from the measurement p from a single image?
No - in general ☹ (P can be anywhere along the line defined by C and p)
Line of sight
C
p
Line of sight
Pinhole perspective projectionRecovering structure from a single view
Ow
P
unknown known Known/Partially known/unknown
http://www.robots.ox.ac.uk/~vgg/projects/SingleView/models/hut/hutme.wrl
Recovering structure from a single view
Transformation in 2D
-Isometries
-Similarities
-Affinity
-Projective
Transformation in 2D
Isometries:!!!
"
#
$$$
%
&
=
!!!
"
#
$$$
%
&
!"
#$%
&=
!!!
"
#
$$$
%
&
1yx
H1yx
10tR
1'y'x
e
- Preserve distance (areas)- 3 DOF- Regulate motion of rigid object
[Euclidean]
[Eq. 4]
Transformation in 2D
Similarities:x 'y'1
!
"
###
$
%
&&&=
SR t0 1
!
"##
$
%&&
xy1
!
"
###
$
%
&&&= Hs
xy1
!
"
###
$
%
&&&
- Preserve- ratio of lengths- angles
-4 DOF
[Eq. 5]S= s 0
0 s!
"#
$
%&
Transformation in 2D
Affinities:!!!
"
#
$$$
%
&
=
!!!
"
#
$$$
%
&
!"
#$%
&=
!!!
"
#
$$$
%
&
1yx
H1yx
10tA
1'y'x
a
!"
#$%
&=
2221
1211
aaaa
A )(RD)(R)(R φφθ ⋅⋅−⋅= !"
#$%
&=
y
x
s00s
D
[Eq. 6]
[Eq. 7]
Transformation in 2D
Affinities:!!!
"
#
$$$
%
&
=
!!!
"
#
$$$
%
&
!"
#$%
&=
!!!
"
#
$$$
%
&
1yx
H1yx
10tA
1'y'x
a
!"
#$%
&=
2221
1211
aaaa
A )(RD)(R)(R φφθ ⋅⋅−⋅= !"
#$%
&=
y
x
s00s
D
-Preserve:- Parallel lines- Ratio of areas- Ratio of lengths on collinear lines- others…
- 6 DOF
[Eq. 6]
[Eq. 7]
Transformation in 2D
Projective:!!!
"
#
$$$
%
&
=
!!!
"
#
$$$
%
&
!"
#$%
&=
!!!
"
#
$$$
%
&
1yx
H1yx
bvtA
1'y'x
p
- 8 DOF- Preserve:
- cross ratio of 4 collinear points- collinearity - and a few others…
[Eq. 8]
The cross ratio
1423
2413
PPPPPPPP
−−
−−
The cross-ratio of 4 collinear points
Can permute the point ordering
!!!!
"
#
$$$$
%
&
=
1i
i
i
i ZYX
P
3421
2431
PPPPPPPP
−−
−−
P1P2
P3P4
[Eq. 9]
Lecture 4 - Silvio Savarese 16-Jan-15
• Review calibration and 2D transformations • Vanishing points and lines • Estimating geometry from a single image • Extensions
Reading: [HZ] Chapter 2 “Projective Geometry and Transformation in 2D” [HZ] Chapter 3 “Projective Geometry and Transformation in 3D” [HZ] Chapter 8 “More Single View Geometry” [Hoiem & Savarese] Chapter 2
Lecture 4 Single View Metrology
Lines in a 2D plane
0cbyax =++
-c/b
-a/b
!!!
"
#
$$$
%
&
=
cba
l
If x = [ x1, x2]T ∈ l 0cba
1xx T
2
1
=
!!!
"
#
$$$
%
&
!!!
"
#
$$$
%
&
l
x
y
[Eq. 10]
Lines in a 2D plane
Intersecting lines
llx !×= l
l!
Proof
lll ⊥"×lll !⊥!×
0l)ll( =⋅"×→0l)ll( =!⋅!×→
lx∈→
x
lx !∈→
→ x is the intersecting point
x
y
x
[Eq. 12][Eq. 13]
[Eq. 11]
2D Points at infinity (ideal points)
0x,xxx
x 3
3
2
1
≠
"""
#
$
%%%
&
'
=!!!
"
#
$$$
%
&
=
cba
l
!!!
"
#
$$$
%
&
'
='
cba
l ''
!!!
"
#
$$$
%
&
−∝)×→
0ab
llLet’s intersect two parallel lines:
• In Euclidian coordinates this point is at infinity• Agree with the general idea of two lines intersecting at infinity
l
l!!!!
"
#
$$$
%
&
=∞
0''
2
1
xx
x
'/'/ baba −=−
∞= x Eq.13]
2D Points at infinity (ideal points)
0x,xxx
x 3
3
2
1
≠
"""
#
$
%%%
&
'
=!!!
"
#
$$$
%
&
=
cba
l
!!!
"
#
$$$
%
&
'
='
cba
l ''
[ ] 00
lT =
!!!
"
#
$$$
%
&
−=∞ ab
cbax
Note: the line l = [a b c]T pass trough the ideal point
So does the line l’ since a b’ = a’ b
l
l!
'/'/ baba −=−
∞x
[Eq. 15]
Lines infinity ∞lSet of ideal points lies on a line called the line at infinity.How does it look like?
∞l
!!!
"
#
$$$
%
&
=∞
100
l
T
2
1
0xx
!!!
"
#
$$$
%
&
0100=
!!!
"
#
$$$
%
&Indeed:
A line at infinity can be thought of the set of “directions” of lines in the plane
'∞x
!!!
"
#
$$$
%
&
−=
0''ab
!!!
"
#
$$$
%
&
−=
0''''ab
''∞x
Projective transformation of a point at infinity
pHp ='
!"
#$%
&=
bvtA
H
?=∞pH!!!
"
#
$$$
%
&
=
!!!
"
#
$$$
%
&
!"
#$%
&=
z
y
x
ppp
bvtA
'''
011
is it a point at infinity?
…no!
?=∞pHA!!!
"
#
$$$
%
&
=
!!!
"
#
$$$
%
&
!"
#$%
&=
0''
011
0 y
x
pp
btA
An affine transformation of a point at infinity is still a point at infinity
[Eq. 17]
[Eq. 18]
Projective transformation of a line (in 2D)
lHl T−="
!"
#$%
&=
bvtA
H
?lH T =∞−
!!!
"
#
$$$
%
&
=
!!!
"
#
$$$
%
&
!"
#$%
&=
−
btt
100
bvtA
y
xT
is it a line at infinity?
…no!
?lH TA =∞−
!!!
"
#
$$$
%
&
=
!!!
"
#
$$$
%
&
!"
#$%
&
−=
!!!
"
#
$$$
%
&
!"
#$%
&=
−
−−
100
100
10
100
10 TT
TT
AtAtA
[Eq. 19]
[Eq. 20]
[Eq. 21]
Points and planes in 3D
!!!!
"
#
$$$$
%
&
=
1xxx
x3
2
1
!!!!
"
#
$$$$
%
&
=Π
dcba
0dczbyax =+++
x
y
z
x∈ Π↔ xTΠ = 0
How about lines in 3D? • Lines have 4 degrees of freedom - hard to represent in 3D-space• Can be defined as intersection of 2 planes
[Eq. 23][Eq. 22]
Points at infinity in 3DPoints where parallel lines intersect in 3D
!!!!
"
#
$$$$
%
&
=∞
03
2
1
xxx
xworld
Parallel lines
point at infinity
Vanishing pointsThe projective projection of a point at infinity into the image plane defines a vanishing point.
M
!!!!
"
#
$$$$
%
&
=∞
03
2
1
xxx
x
!!!
"
#
$$$
%
&
=∞
3
2
1
ppp
p
world
p∞
Parallel lines
point at infinity
= direction of the line in 3D
v=M X∞ =K I 0[ ]
abc0
"
#
$$$$
%
&
''''
=Kabc
"
#
$$$
%
&
'''
dv K=
d = direction of the line = [a, b, c]T
d
C
v
X∞ =
abc0
"
#
$$$$
%
&
''''
M
vvd
1
1
−
−
=KK
[Eq. 24] [Eq. 25]
Proof:
Vanishing points and directions
Vanishing (horizon) line
horizon
∞
−
= lHlT
Phor
∞l
π
Projective transformation M
Image
[Eq. 26]
Are these two lines parallel or not?
- Recognize the horizon line- Measure if the 2 lines meet at the horizon- if yes, these 2 lines are // in 3D
•Recognition helps reconstruction!•Humans have learnt this
Vanishing points and planes
C
n
horizTK ln =
…
horizl
πl∞
[Eq. 27]
Planes at infinity
x
y
z
• A set of 2 or more lines at infinity defines the plane at infinity
∞Π…
!!!!
"
#
$$$$
%
&
=Π∞
1000
plane at infinity …
∞Π
• Parallel planes intersect at infinity in a common line – the line at infinity
Angle between 2 vanishing points
2T21
T1
2T1
vvvv
vvcos
ωω
ωθ =
C
d1v2
v1d2
θ
If 90=θ 0vv 2T1 =→ ω
ω = (K KT )−1[Eq. 28]
x1∞
x2∞
[Eq. 29]Scalar equation
Projective transformation of ∞Ω
= (K KT )−1
It is not function of R, T
!!!
"
#
$$$
%
&
=
654
532
421
ωωω
ωωω
ωωω
ω symmetric and known up scale
02 =ω zero-skew 31 ωω = square pixel02 =ω
1.
2.
3. 4.
M = K R T!"
#$=M −TΩ∞M
−1
Absolute conic
[Eq. 30]
ω
Why is this useful?
• To calibrate the camera• To estimate the geometry of the 3D world
Lecture 4 - Silvio Savarese 16-Jan-15
• Review calibration • Vanishing points and line • Estimating geometry from a single image • Extensions
Reading: [HZ] Chapter 2 “Projective Geometry and Transformation in 3D” [HZ] Chapter 3 “Projective Geometry and Transformation in 3D” [HZ] Chapter 8 “More Single View Geometry” [Hoiem & Savarese] Chapter 2
Lecture 4 Single View Metrology
θ = ?
0vv 2T1 =ω
v1v2
ω = (K KT )−1Do we have enough constraints to estimate K?K has 5 degrees of freedom and Eq.29 is a scalar equation ☹
Single view calibration - example
2T21
T1
2T1
vvvv
vvcos
ωω
ωθ =
[Eq.
28]
= 90o
v1v2
Single view calibration - example
2T21
T1
2T1
vvvv
vvcos
ωω
ωθ =
[Eq.
28]
v3
0vv 2T1 =ω
0vv 3T1 =ω
0vv 3T2 =ω
[Eqs
. 31]
Single view calibration - example
0vv 2T1 =ω
0vv 3T1 =ω
0vv 3T2 =ω
!!!
"
#
$$$
%
&
=
654
532
421
ωωω
ωωω
ωωω
ω
31 ωω =02 =ω à Compute :ω
known up to scale
ω = (K KT )−1 K(Cholesky factorization; HZ pag 582)
Once ω is calculated, we get K:
[Eqs
. 31]
Single view reconstruction - example
lh
knownK horizTK ln =→ = Scene plane orientation in
the camera reference system
Select orientation discontinuities
Single view reconstruction - example
Recover the structure within the camera reference systemNotice: the actual scale of the scene is NOT recovered
C
•Recognition helps reconstruction!•Humans have learnt this
Lecture 4 - Silvio Savarese 16-Jan-15
• Review calibration • Vanishing points and lines • Estimating geometry from a single image • Extensions
Reading: [HZ] Chapter 2 “Projective Geometry and Transformation in 3D” [HZ] Chapter 3 “Projective Geometry and Transformation in 3D” [HZ] Chapter 8 “More Single View Geometry” [Hoiem & Savarese] Chapter 2
Lecture 4 Single View Metrology
Criminisi & Zisserman, 99
http://www.robots.ox.ac.uk/~vgg/projects/SingleView/models/merton/merton.wrl
Criminisi & Zisserman, 99
http://www.robots.ox.ac.uk/~vgg/projects/SingleView/models/merton/merton.wrl
La Trinita' (1426) Firenze, Santa Maria Novella; by Masaccio (1401-1428)
La Trinita' (1426) Firenze, Santa Maria Novella; by Masaccio (1401-1428)
http://www.robots.ox.ac.uk/~vgg/projects/SingleView/models/hut/hutme.wrl
Manually select:• Vanishing points and lines;• Planar surfaces;• Occluding boundaries;• Etc..
Single view reconstruction - drawbacks
Automatic Photo Pop-upHoiem et al, 05
Automatic Photo Pop-upHoiem et al, 05…
Automatic Photo Pop-upHoiem et al, 05…
http://www.cs.uiuc.edu/homes/dhoiem/projects/software.html
Software:
Training
Image Depth
Prediction
Planar Surface Segmentation
Plane Parameter MRF
Connectivity Co-Planarity
Make3D
youtube
Saxena, Sun, Ng, 05…
A software: Make3D“Convert your image into 3d model”
Single Image Depth ReconstructionSaxena, Sun, Ng, 05…
http://make3d.stanford.edu/http://make3d.cs.cornell.edu/
Coherent object detection and scene layout estimation from a single image
Y. Bao, M. Sun, S. Savarese, CVPR 2010, BMVC 2010
Y. BaoM. Sun
Next lecture:
Multi-view geometry (epipolar geometry)
Appendix
