3D Urban Modeling

Marc PollefeysJan-Michael Frahm, Philippos Mordohai

Fall 2006 / Comp 790-089

Tue & Thu 14:00-15:15

3D Urban Modeling

• Basics: sensor, techniques, …• Video, LIDAR, GPS, IMU, …• SfM, Stereo, …• Robot Mapping, GIS, …

• Review state-of-the-art• Video and LIDAR-based systems

• Projects to experiment• Virtual UNC-Chapel Hill• DARPA Urban Challenge• …

3D Urban Modeling course schedule(tentative)

Sep. 5, 7 Introduction Video-based Urban 3D Capture

Sep. 12, 14 Cameras

Sep. 19, 21

Sep. 26, 28 Project proposals

Oct. 3, 5

Oct. 10, 12

Oct. 17, 19 (fall break)

Oct. 24, 26 Project Status Update

Oct. 31, Nov. 2

Nov. 7, 9

Nov. 14, 16

Nov. 21, 23 (Thanksgiving)

Nov. 28, 30

Dec. 5 Project demonstrations (classes ended)Note: Dec. 3 is CVPR deadline

Course material

Slides, notes, papers and references

see course webpage/wiki (later)

On-line “shape-from-video” tutorial:http://www.cs.unc.edu/~marc/tutorial.pdfhttp://www.cs.unc.edu/~marc/tutorial/

Geometry primer

l'lx

Incidence of point and line

x'xl

0xl T 0Xπ T

0ll * CT

1* CC

0QXX T 0πQπ * T

-1* QQ

Joins

Conics and dual conics

0xx CT

Incidence of point and plane

Joins

Conics and dual conics

0Xπππ

T

3

T

2

T

1

0πXXX

T

3

T

2

T

1

Geometry primer

Transformation for lines

ll' -TH

Transformation for conics-1-TCHHC '

Transformation for dual conics

THHCC **'

xx' HTransformation for points

XX' H

ππ' -TH

-1-TQHHQ'

THHQQ' **

Transformation for planes

Transformation for quadrics

Transformation for dual quadrics

Transformation for points

Geometry primer Hierarchy of 2D transformations

1002221

1211

y

x

taa

taa

1002221

1211

y

x

tsrsr

tsrsr

333231

232221

131211

hhh

hhh

hhh

1002221

1211

y

x

trr

trr

Projective8dof

Affine6dof

Similarity4dof

Euclidean3dof

Concurrency, collinearity, order of contact (intersection, tangency, inflection, etc.), cross ratio

Parallellism, ratio of areas, ratio of lengths on parallel lines (e.g midpoints), linear combinations of vectors (centroids). The line at infinity l∞

Ratios of lengths, angles.The circular points I,J

lengths, areas.

invariantstransformed squares

Geometry primer Hierarchy of 3D transformations

vTv

tAProjective15dof

Affine12dof

Similarity7dof

Euclidean6dof

Intersection and tangency

Parallellism of planes,Volume ratios, centroids,The plane at infinity π∞

Angles, ratios of length

The absolute conic Ω∞

Volume

10

tAT

10

tRT

s

10

tRT

Camera model

Relation between pixels and rays in space

?

Projector model

Relation between pixels and rays in space(dual of camera)

(main geometric difference is vertical principal point offset to reduce keystone effect)

?

Affine cameras

Pinhole camera model

TT ZfYZfXZYX )/,/(),,(

101

0

0

1

Z

Y

X

f

f

Z

fY

fX

Z

Y

X

linear projection in homogeneous coordinates!

homogeneous coordinates

non-homogeneous coordinates

Pinhole camera model

101

0

0

Z

Y

X

f

f

Z

fY

fX

101

01

01

1Z

Y

X

f

f

Z

fY

fX

PXx

0|I)1,,(diagP ff

Principal point offset

Tyx

T pZfYpZfXZYX )/,/(),,(

principal pointT

yx pp ),(

101

0

0

1

Z

Y

X

pf

pf

Z

ZpfY

ZpfX

Z

Y

X

y

x

x

x

Principal point offset

101

0

0

Z

Y

X

pf

pf

Z

ZpfY

ZpfX

y

x

x

x

camX0|IKx

1y

x

pf

pf

K calibration matrix

Object motion

Camera motion

CCD camera

1yy

xx

pp

K

11y

x

y

x

pfpf

mm

K

General projective camera

1yx

xx

p

ps

K

1yx

xx

p

p

K

t|IKRP

non-singular

11 dof (5+3+3)

t|RKP

intrinsic camera parametersextrinsic camera parameters

Camera matrix decomposition

Finding the camera center

0PC (use SVD to find null-space)

Finding the camera orientation and internal parameters

KR (use RQ decomposition ~QR)

Q R=( )-1= -1 -1QR

(if only QR, invert)

PXλC)P(Xx (for all X and λ C must be camera center)

t|RKP

Affine cameras

Radial distortion

• Due to spherical lenses (cheap)• Model:

R

yxyxKyxKyx ...))()(1(),( 222

2

22

1R

http://foto.hut.fi/opetus/260/luennot/11/atkinson_6-11_radial_distortion_zoom_lenses.jpgstraight lines are not straight anymore

pincushion dist.

barrel dist.

Radial distortion example

Meydenbauer camera

vertical lens shiftto allow direct ortho-photographs

Action of projective camera on points and lines

forward projection of line

μbaμPBPAμB)P(AμX

back-projection of line

lPT

PXlX TT PX x0;xlT

PXx projection of point

Action of projective camera on conics and quadricsback-projection to cone

CPPQ Tco 0CPXPXCxx TTT

PXx

projection of quadric

TPPQC ** 0lPPQlQ T*T*T

lPT

ii xX ? P

Resectioning

Direct Linear Transform (DLT)

ii PXx ii PXx

rank-2 matrix

Direct Linear Transform (DLT)

Minimal solution

Over-determined solution

5½ correspondences needed (say 6)

P has 11 dof, 2 independent eq./points

n 6 points

Apminimize subject to constraint

1p

use SVD

Singular Value Decomposition

XXVT XVΣ T XVUΣ T

Homogeneous least-squares

TVUΣA

1X AXmin subject to nVX solution

TVΣU

X

Degenerate configurations

(i) Points lie on plane or single line passing through projection center

(ii) Camera and points on a twisted cubic

Scale data to values of order 1

1. move center of mass to origin2. scale to yield order 1 values

Data normalization

D3

D2

Line correspondences

Extend DLT to lines

ilPT

ii 1TPXl

(back-project line)

ii 2TPXl (2 independent eq.)

Geometric error

Gold Standard algorithmObjective

Given n≥6 2D to 2D point correspondences {Xi↔xi’}, determine the Maximum Likelyhood Estimation of P

Algorithm

(i) Linear solution:

(a) Normalization:

(b) DLT

(ii) Minimization of geometric error: using the linear estimate as a starting point minimize the geometric error:

(iii) Denormalization:

ii UXX~ ii Txx~

UP~

TP -1

~ ~~

Calibration example

(i) Canny edge detection(ii) Straight line fitting to the detected edges(iii) Intersecting the lines to obtain the images corners

typically precision <1/10

(H&Z rule of thumb: 5n constraints for n unknowns)

Errors in the world

Errors in the image and in the world

ii XPx

iX

Errors in the image

iPXx̂

i

(standard case)

Restricted camera estimation

Minimize geometric error impose constraint through parametrization

Find best fit that satisfies• skew s is zero• pixels are square • principal point is known• complete camera matrix K is known

Minimize algebraic error assume map from param q P=K[R|-RC], i.e. p=g(q)minimize ||Ag(q)||

Restricted camera estimation

Initialization • Use general DLT• Clamp values to desired values, e.g. s=0, x= y

Note: can sometimes cause big jump in error

Alternative initialization• Use general DLT• Impose soft constraints

• gradually increase weights

Image of absolute conic

A simple calibration device

(i) compute H for each square (corners (0,0),(1,0),(0,1),(1,1))

(ii) compute the imaged circular points H(1,±i,0)T

(iii) fit a conic to 6 circular points(iv) compute K from through cholesky factorization

(≈ Zhang’s calibration method)

Some typical calibration algorithmsTsai calibration

Zhangs calibration

http://research.microsoft.com/~zhang/calib/

Z. Zhang. A flexible new technique for camera calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11):1330-1334, 2000.

Z. Zhang. Flexible Camera Calibration By Viewing a Plane From Unknown Orientations. International Conference on Computer Vision (ICCV'99), Corfu, Greece, pages 666-673, September 1999.

http://www.vision.caltech.edu/bouguetj/calib_doc/

*

*

projection

constraints

Tiii

Tii ωΩ KKPP

Self-calibration using absolute conic

Absolute conic projection:

Translate constraints on K through projection equation

to constraints on *

Euclidean projection matrix:

Upgrade from projective to metric

Transform structure and motion so that * diag(1,1,1,0)

some constraints, e.g. constant, no skew,...

(Faugeras ECCV’92; Triggs CVPR’97; Pollefeys et al. ICCV’98; etc.)

Practical linear self-calibration

0Ω

0Ω

0Ω

0ΩΩ

23T

13T

12T

22T

11T

PP

PP

PP

PPPP

(Pollefeys et al., ECCV‘02)

100

0ˆ0

00ˆ2

2

* f

fTPPKK

0ΩΩ

0ΩΩ

33T

22T

33T

11T

PPPP

PPPP

(only rough aproximation, but still usefull to avoid degenerate configurations)

(relatively accurate for most cameras)

9

1

9

1

1.0

11.0

101.0

12.0

1

1yy

xx

cf

csf

K

Don’t treat all constraints equal

after normalization!

when fixating point at image-center not only absolute quadric diag(1,1,1,0) satisfies ICCV98 eqs., but also diag(1,1,1,a), i.e. real or imaginary spheres!