Camera Modelsclass 8

Multiple View GeometryComp 290-089Marc Pollefeys

Multiple View Geometry course schedule(subject to change)

Jan. 7, 9 Intro & motivation Projective 2D Geometry

Jan. 14, 16

(no class) Projective 2D Geometry

Jan. 21, 23

Projective 3D Geometry (no class)

Jan. 28, 30

Parameter Estimation Parameter Estimation

Feb. 4, 6 Algorithm Evaluation Camera Models

Feb. 11, 13

Camera Calibration Single View Geometry

Feb. 18, 20

Epipolar Geometry 3D reconstruction

Feb. 25, 27

Fund. Matrix Comp. Structure Comp.

Mar. 4, 6 Planes & Homographies Trifocal Tensor

Mar. 18, 20

Three View Reconstruction

Multiple View Geometry

Mar. 25, 27

MultipleView Reconstruction

Bundle adjustment

Apr. 1, 3 Auto-Calibration Papers

Apr. 8, 10

Dynamic SfM Papers

Apr. 15, 17

Cheirality Papers

Apr. 22, 24

Duality Project Demos

N measurements (independent Gaussian noise ) model with d essential parameters(use s=d and s=(N-d))

(i) RMS residual error for ML estimator

(ii) RMS estimation error for ML estimator

2/12/12

/1/XX̂ NdNEeres

2/12/12

//XX̂ NdNEeest

nX

X

X

SM

Error in two images

2/1

2

4

n

neres

2/1

2

4

n

neest

nNnd 4 and 28

Backward propagation of covariance

X f -1P

X -11

xT

P JJ

Over-parameterizationA

JJ 1x

TP

Jf v

Forward propagation of covariance

JJ PT

X

Monte-Carlo estimation of covariance

=1 pixel =0.5cm (Crimisi’97)

Example:

Single view geometry

Camera model

Camera calibration

Single view geom.

Pinhole camera model

TT ZfYZfXZYX )/,/(),,(

101

0

0

1

Z

Y

X

f

f

Z

fY

fX

Z

Y

X

Pinhole camera model

101

0

0

Z

Y

X

f

f

Z

fY

fX

101

01

01

1Z

Y

X

f

f

Z

fY

fX

PXx

0|I)1,,(diagP ff

Principal point offset

Tyx

T pZfYpZfXZYX )/,/(),,(

principal pointT

yx pp ),(

101

0

0

1

Z

Y

X

pf

pf

Z

ZpfY

ZpfX

Z

Y

X

y

x

x

x

Principal point offset

101

0

0

Z

Y

X

pf

pf

Z

ZpfY

ZpfX

y

x

x

x

camX0|IKx

1y

x

pf

pf

K calibration matrix

Camera rotation and translation

C~

-X~

RX~

cam

X10

RCR

1

10

C~

RRXcam

Z

Y

X

camX0|IKx XC~

|IKRx

t|RKP C~

Rt PXx

CCD camera

1yx

xx

p

p

K

11y

x

x

x

pf

pf

m

m

K

Finite projective camera

1yx

xx

p

ps

K

1yx

xx

p

p

K

C~

|IKRP

non-singular

11 dof (5+3+3)

decompose P in K,R,C?

4p|MP 41pMC

~ MRK, RQ

{finite cameras}={P4x3 | det M≠0}

If rank P=3, but rank M<3, then cam at infinity

Camera anatomy

Camera centerColumn pointsPrincipal planeAxis planePrincipal pointPrincipal ray

Camera center

0PC

null-space camera projection matrix

λ)C(1λAX

λ)PC(1λPAPXx

For all A all points on AC project on image of A,

therefore C is camera center

Image of camera center is (0,0,0)T, i.e. undefined

Finite cameras:

1

pM 41

C

Infinite cameras: 0Md,0

d

C

Column vectors

0

0

1

0

ppppp 43212

Image points corresponding to X,Y,Z directions and origin

Row vectors

1p

p

p

0 3

2

1

Z

Y

X

y

x

T

T

T

1p

p

p0

3

2

1

Z

Y

X

w

yT

T

T

note: p1,p2 dependent on image reparametrization

The principal point

principal point

0,,,p̂ 3332313 ppp

330 Mmp̂Px

The principal axis vector

3m

camcamcam X0|IKXPx T1,0,0mMdetv 3

camcam PP k vv 4k

4p|MC~

|IKRP k

0)Rdet(

vector defining front side of camera

(direction unaffected)

vmMdetv 43 kkk camcam PP k

because

Action of projective camera on point

PXx

MdDp|MPDx 4

Forward projection

Back-projection

xPX 1PPPP

TT IPP

(pseudo-inverse)

0PC

λCxPλX

1

p-μxM

1

pM-

0

xMμλX 4

-14

-1-1

xMd -1

CD

Depth of points

C~

X~

mCXPXPT3T3T3 w

(dot product)(PC=0)

1m;0det 3 MIf , then m3 unit vector in positive direction

3m

)sign(detMPX;depth

T

w

TX X,Y,Z,T

Camera matrix decomposition

Finding the camera center

0PC (use SVD to find null-space)

432 p,p,pdetX 431 p,p,pdetY

421 p,p,pdetZ 321 p,p,pdetT

Finding the camera orientation and internal parameters

KRM (use RQ decomposition ~QR)

Q R=( )-1= -1 -1QR

(if only QR, invert)

When is skew non-zero?

1yx

xx

p

ps

K

1

arctan(1/s)

for CCD/CMOS, always s=0

Image from image, s≠0 possible(non coinciding principal axis)

HPresulting camera:

Euclidean vs. projective

homography 44

0100

0010

0001

homography 33P

general projective interpretation

Meaningfull decomposition in K,R,t requires Euclidean image and space

Camera center is still valid in projective space

Principal plane requires affine image and space

Principal ray requires affine image and Euclidean space

Cameras at infinity

00

dP

Camera center at infinity

0Mdet

Affine and non-affine cameras

Definition: affine camera has P3T=(0,0,0,1)

Affine cameras

Affine cameras

C~

rr

C~

rr

C~

rr

KC~

|IKRP3T3T

2T2T

1T1T

0

C~

r3T0 d

t

t

dt

t

t

3T

2T2T

1T1T

33T3T

32T2T

31T1T

r

C~

rr

C~

rr

K

r-C~

rr

r-C~

rr

r-C~

rr

KP

modifying p34 corresponds to moving along principal ray

Affine cameras

003T

2T2T

1T1T

0

3T

2T2T

1T1T

0

0

/r

C~

rr

C~

rr

K

r

C~

rr

C~

rr

1

/

/

KP

dddd

d

d

dd

dd

t

t

t

t

t

t

now adjust zoom to compensate

0

2T2T

1T1T

0

C~

rr

C~

rr

KPlimP

dt

t

Error in employing affine cameras

1

βrαrX

21

1

rβrαrX

321

XPXPXP t0

point on plane parallel with principal plane and through origin, then

general points

Δ

~

~

KXPx

0

0proj

d

y

x

0

affine~

~

KXPx

d

y

x

projxaffinex

0x

Affine imaging conditions

0proj0

projaffine x-xx-xd

Approximation should only cause small error

1. much smaller than d0

2. Points close to principal point (i.e. small field of view)

Decomposition of P∞

0

02x2

0

t~R~

10

x~KP

d

10

t~R~

10

x~K 02x2-1

0d

absorb d0 in K2x2

10

0R~

10

x~t~KK

10

x~Kt~R~

10

0K

10

x~t~KR~

K

02x22x2

0-12x22x202x22x2

10

0R~

10

x~K

10

t~R~

10

0KP 02x22x2

alternatives, because 8dof (3+3+2), not more

Summary parallel projection

1000

0010

0001

P canonical representation

10

0KK 22 calibration matrix

principal point is not defined

A hierarchy of affine cameras

Orthographic projection

Scaled orthographic projection

1000

0010

0001

P

10

tRH

10rr

P 21T

11T

tt

ktt

/10rr

P 21T

11T

(5dof)

(6dof)

A hierarchy of affine cameras

Weak perspective projection

ktt

y

x

/10rr

1α

αP 2

1T1

1T

(7dof)

1. Affine camera=camera with principal plane coinciding with ∞

2. Affine camera maps parallel lines to parallel lines

3. No center of projection, but direction of projection PAD=0(point on ∞)

A hierarchy of affine camerasAffine camera

ktts

y

x

A

/10rr

1α

αP 2

1T1

1T

(8dof)

1000P 2232221

1131211

tmmmtmmm

A

affine 44100000100001

affine 33P

A

Pushbroom cameras

T)(X X,Y,X,T T),,(PX wyx T)/,( wyx

Straight lines are not mapped to straight lines!(otherwise it would be a projective camera)

(11dof)

Line cameras

ZYX

pppppp

yx

232221

131211(5dof)

Null-space PC=0 yields camera center

Also decomposition c~|IRKP 22222232

Next class: Camera calibration