3Dcamerapose

⎡ ⎤⎢ ⎥⎣ ⎦

x =uv1

11

P1 ∈ !3×4 P2

PF2Dcorrespondences

⎡ ⎤⎢ ⎥⎢ ⎥⎢ ⎥⎣ ⎦

X=XYZ

Optical Flow: 2D point correspondences

Optical Flow: 2D point correspondences

Optical Flow: 2D point correspondences

I(x) J(x)

t = 0 t = 1

d

I(x) J(x)

I(x) = J(x+ d)

Whend=0

=-

mindE(d) = ||J(x+ d)� I(x)||2

E(d=0)

Three steps for solving this problem

1: Solve for

�E

�d

����d⇤

= 0

3: Solve for d, warp image, iterate

2: Taylor expansion on J(x+ d)

Step1:Solvefor

E(d) = ||J(x+ d)� I(x)||2

�E

�d

����d⇤

= 0

E(d⇤ = 0)

Step1:Solvefor

E(d) = ||J(x+ d)� I(x)||2

�E

�d

����d⇤

= 0

E(d⇤ = 0)

�E

�d

����d⇤

= 2�J(x+ d)

�d

T

(J(x+ d)� I(x)) = 0

Step1:Solvefor

E(d) = ||J(x+ d)� I(x)||2

�E

�d

����d⇤

= 0

E(d⇤ = 0)

�E

�d

����d⇤

= 2�J(x+ d)

�d

T

(J(x+ d)� I(x)) = 0

�E

�d

����d⇤

= 2�J(x)

�x

T

(J(x+ d)� I(x)) = 0

E(d⇤ = 0)

�E

�d

����d⇤

= 2�J(x)

�x

T

(J(x+ d)� I(x)) = 0

�J(x)

�x

=

�J(x)

�y=

= +

Step2:TaylorexpansionJ(x+ d)

J(x+ d) = J(x) +�J(x)

�xd

dy⇤dx

⇤ +

Putting all together

J(x+ d) = J(x) +�J(x)

�xd

�E

�d

����d⇤

= 2�J(x)

�x

T

(J(x+ d)� I(x)) = 0

�J(x)

�x

T �J(x)

�xd =

�J(x)

�x

T

(I(x)� J(x))

�J(x)

�x

T �J(x)

�xd =

�J(x)

�x

T

(I(x)� J(x))

�J(x)

�x

T �J(x)

�xd =

�J(x)

�x

T

(I(x)� J(x))

2Dunknownsflowvectorperpixel2equations

�J(x)

�x

T �J(x)

�xd =

�J(x)

�x

T

(I(x)� J(x))

Alsoknownassecondmomentmatrix

�J(x)

�x

T �J(x)

�x

�J(x)

�x

2

�J(x)

�x

�J(x)

�y

�J(x)

�x

�J(x)

�y

�J(x)

�y

2

�J(x)

�x

T �J(x)

�xd =

�J(x)

�x

T

(I(x)� J(x))

�J(x)

�x

T �J(x)

�xd =

�J(x)

�x

T

(I(x)� J(x))

�J(x)

�x

T �J(x)

�xd =

�J(x)

�x

T

(I(x)� J(x))

�J(x)

�x

T �J(x)

�xd =

�J(x)

�x

T

(I(x)� J(x))

dx

dy

=

�J(x)

�x

T �J(x)

�xd =

�J(x)

�x

T

(I(x)� J(x))

dx

dy =+

min

d

E(d) =X

x

||J(x+ d)� I(x)||2

X

x

�J(x)

�x

T �J(x)

�xd =

X

x

�J(x)

�x

T

(I(x)� J(x))

min

d

E(d) =X

x

||J(x+ d)� I(x)||2

2×2matrix

Summingoverpixels

=

2×1matrix

Summingoverpixels

=

dx

dy =+

X

x

�J(x)

�x

T �J(x)

�xd =

X

x

�J(x)

�x

T

(I(x)� J(x))

3: Solve for d, warp image, iterate

I(x)J(x) Error

d = (�7,�9)

3: Solve for d, warp image, iterate

I(x)J(x) Error

d = (�7,�9)

J

t=1(x) = J(x+ d)

Error

d=(-7.1,-8.8)

d=(-6.8,-8.9)

d=(-1.4,-3.0)

I(x) J(x)

t = 0 t = 1

�J(x)

�x

T �J(x)

�xd =

�J(x)

�x

T

(I(x)� J(x))

�J(x)

�x

T �J(x)

�xd =

�J(x)

�x

T

(I(x)� J(x))

�J(x)

�x

=

�J(x)

�y=

�J(x)

�x

T �J(x)

�xd =

�J(x)

�x

T

(I(x)� J(x))

dx

dy

=

3: Solve for d, warp image, iterate

I(x)J(x) Error

J

t=1(x) = J(x+ d)

d=(-4.9,-0.4)

Error

d=(-4.9,-0.4)

d=(-0.1,-5.8)

d=(0, -3.7)