TUM - Joint Optimization of Pose And Depth Using a Prox-Linear … · 2019. 12. 28. · Master...

Problem Formulation Optimization Results

Joint Optimization of Pose And Depth Using aProx-Linear ApproachMaster Thesis Presentation

Florian Hofherr

20. Dezember 2019

Florian Hofherr

Joint Optimization of Pose And Depth Using a Prox-Linear Approach

Problem Formulation Optimization Results

1 Problem Formulation

2 Optimization

3 Results

Florian Hofherr

Joint Optimization of Pose And Depth Using a Prox-Linear Approach

Problem Formulation Optimization Results

1 Problem Formulation

2 Optimization

3 Results

Florian Hofherr

Joint Optimization of Pose And Depth Using a Prox-Linear Approach

Problem Formulation Optimization Results

SLAM Overview

X0

Y0

Z0

Scene

CameraTrajectory

Florian Hofherr

Joint Optimization of Pose And Depth Using a Prox-Linear Approach

Problem Formulation Optimization Results

Warping Between two Frames

X

X0

Y0

Z0

x0y0

0xX1

Y1

Z1

x1y1

1x

R (w) ,T

w: Exponential CoordinatesPinhole Camera: hx̃ = KX

1x = ω0x (R,T , 0h)

Florian Hofherr

Joint Optimization of Pose And Depth Using a Prox-Linear Approach

Problem Formulation Optimization Results

Photometric Data Term

X

I0

x0y0

0x

X0

Y0

Z0

I1

x1y11x

X1

Y1

Z1

Photometric → Compare image intensities

Bilinear/Bicubic interpolation in second image

Assume Lambertian surfaces

Dense → All valid pixels

Florian Hofherr

Joint Optimization of Pose And Depth Using a Prox-Linear Approach

Problem Formulation Optimization Results

Photometric Data Term

Data term for two images

Edata (w,T ,h) =∑x∈VI1

` (J I1 (ωx (R (w) ,T ,h))− I0 (x))

VI1 : Set of valid pixels

`: Loss function

J : Interpolation operator

Extendable for more images

Florian Hofherr

Joint Optimization of Pose And Depth Using a Prox-Linear Approach

Problem Formulation Optimization Results

Regularization

Isotropic Huber Regularization with image-driven weights

Ereg (h) =∑x∈P

γ (x) ‖Dh (x)‖h

P: Set of all pixels

γ (x) = e−α‖DI0(x)‖β2

D: Discrete gradient operator (forward differences)

Huber norm: ‖x‖h =

{12h ‖x‖

22 if ‖x‖2 ≤ h

‖x‖2 −h2 else

Florian Hofherr

Joint Optimization of Pose And Depth Using a Prox-Linear Approach

Problem Formulation Optimization Results

Joint Optimization Problem

Joint Optimization Problem

minw,T,h

{Edata (w,T,h) + λregEreg (h)}

Advantages

Only one optimization for pose and depth

No keypoint selection, all pixels for pose

Difficulties

Non-linear, Non-Convex

High-Dimensional

Florian Hofherr

Joint Optimization of Pose And Depth Using a Prox-Linear Approach

Problem Formulation Optimization Results

1 Problem Formulation

2 Optimization

3 Results

Florian Hofherr

Joint Optimization of Pose And Depth Using a Prox-Linear Approach

Problem Formulation Optimization Results

Composite Optimization Problem

Composite Optimization Problem

minx

F (x) := g (x) + h (c (x))

g : Rd → R and h : Rm → R proper, closed and convex

c : Rd → Rm be a C 1-smooth

Florian Hofherr

Joint Optimization of Pose And Depth Using a Prox-Linear Approach

Problem Formulation Optimization Results

Composite Optimization Problem

Composite Optimization Problem

minx

F (x) := g (x) + h (c (x))

g : Rd → R and h : Rm → R proper, closed and convex

c : Rd → Rm be a C 1-smooth

g (·) ↔ λregEreg (h) = λreg∑x∈P

γ (x) ‖Dh (x)‖h

Florian Hofherr

Joint Optimization of Pose And Depth Using a Prox-Linear Approach

Problem Formulation Optimization Results

Composite Optimization Problem

Composite Optimization Problem

minx

F (x) := g (x) + h (c (x))

g : Rd → R and h : Rm → R proper, closed and convex

c : Rd → Rm be a C 1-smooth

h (c (·)) ↔ Edata (w,T ,h) =∑x∈VI1

` (J I1 (ωx (R (w) ,T ,h))− I0 (x))

Florian Hofherr

Joint Optimization of Pose And Depth Using a Prox-Linear Approach

Problem Formulation Optimization Results

The Standard Prox-Linear Algorithm

Composite Optimization Problem

minx

F (x) := g (x) + h (c (x))

Iterative update

xk+1 = arg minx∈Rd

{g (x) + h

(c(xk)

+ Jc(xk) (

x − xk))

+1

2t

∥∥x − xk∥∥22

}

Jc(xk): Jacobian Matrix of c

Florian Hofherr

Joint Optimization of Pose And Depth Using a Prox-Linear Approach

Problem Formulation Optimization Results

The Standard Prox-Linear Algorithm

Composite Optimization Problem

minx

F (x) := g (x) + h (c (x))

Iterative update

xk+1 = arg minx∈Rd

{g (x) + h

(c(xk)

+ Jc(xk) (

x − xk))

+1

2t

∥∥x − xk∥∥22

}

Jc(xk): Jacobian Matrix of c

g ≡ 0, h = 12

∑x2i ⇒ Levenberg-Marquardt

h (x) = x ⇒ Prox-gradient algorithm

Florian Hofherr

Joint Optimization of Pose And Depth Using a Prox-Linear Approach

Problem Formulation Optimization Results

Weighted Proximal Mapping

f : Rn → R closed, proper and convex

M ∈ Rn×n symmetric positive definite matrix (→ diagonal)

Weighted norm: ‖u‖2M = 〈M−1u, u〉

proxMf (u) := arg minv∈Rn

{f (v) +

1

2‖v − u‖2M

}

Florian Hofherr

Joint Optimization of Pose And Depth Using a Prox-Linear Approach

Problem Formulation Optimization Results

The Weighted Prox-Linear Algorithm

Composite Optimization Problem

minx

F (x) := g (x) + h (c (x))

Iterative update

xk+1 = arg minx∈Rd

{g (x) + h

(c(xk)

+ Jc(xk) (

x − xk))

+1

2

∥∥x − xk∥∥2Mk

}

Step widths:(Mk)−1

= 1ζkstep

M−10 + diag(Jk

TJk)

Florian Hofherr

Joint Optimization of Pose And Depth Using a Prox-Linear Approach

Problem Formulation Optimization Results

The Weighted Prox-Linear Algorithm

Composite Optimization Problem

minx

F (x) := g (x) + h (c (x))

Iterative update

xk+1 = arg minx∈Rd

{g (x) + h

(c(xk)

+ Jc(xk) (

x − xk))

+1

2

∥∥x − xk∥∥2Mk

}

Step widths:(Mk)−1

= 1ζkstep

M−10 + diag(Jk

TJk)

Florian Hofherr

Joint Optimization of Pose And Depth Using a Prox-Linear Approach

Problem Formulation Optimization Results

Solution of the Sub-Problems

Sub-Problem

uk+1 = arg minu

{h(Jku − bk

)+ λregEreg (u) +

1

2

∥∥∥u − uk∥∥∥2Mk

}

General Case

h: Absolute loss, Huber loss

Standard TV regularization possible

Solve using preconditioned PDHG

Florian Hofherr

Joint Optimization of Pose And Depth Using a Prox-Linear Approach

Problem Formulation Optimization Results

Solution of the Sub-Problems

Sub-Problem

uk+1 = arg minu

{h(Jku − bk

)+ λregEreg (u) +

1

2

∥∥∥u − uk∥∥∥2Mk

}

Special Case: h (x) = 12 ‖x‖

2

Linearize Regularization

⇒ Analytic Solution

Mixture: Levenberg-Marquardt on data term, gradient descenton regularization

Florian Hofherr

Joint Optimization of Pose And Depth Using a Prox-Linear Approach

Problem Formulation Optimization Results

1 Problem Formulation

2 Optimization

3 Results

Florian Hofherr

Joint Optimization of Pose And Depth Using a Prox-Linear Approach

Problem Formulation Optimization Results

Experiment 1

Compare joint estimation to pure depth/pose estimation

Absolute data loss

Pure estimation: Use ground truth

New Tsukuba data set

Florian Hofherr

Joint Optimization of Pose And Depth Using a Prox-Linear Approach

Problem Formulation Optimization Results

Comparison Joint Approach vs. Pure Pose/Depth

Image Pair

Florian Hofherr

Joint Optimization of Pose And Depth Using a Prox-Linear Approach

Problem Formulation Optimization Results

Comparison Joint Approach vs. Pure Pose/Depth

True depth and initial value

100

200

300

Florian Hofherr

Joint Optimization of Pose And Depth Using a Prox-Linear Approach

Problem Formulation Optimization Results

Comparison Joint Approach vs. Pure Pose/Depth

Result Joint Optimization

100

200

300

−1

−0.5

0

0.5

1

Florian Hofherr

Joint Optimization of Pose And Depth Using a Prox-Linear Approach

Problem Formulation Optimization Results

Comparison Joint Approach vs. Pure Pose/Depth

Result Pure Depth Estimation

100

200

300

−1

−0.5

0

0.5

1

Florian Hofherr

Joint Optimization of Pose And Depth Using a Prox-Linear Approach

Problem Formulation Optimization Results

Comparison Joint Approach vs. Pure Pose/Depth

Comparison Results Pose

blue: joint optimization

orange: pure pose optimization

0 10 20 300

1

2

3

Iterations

δ win

[◦]

0 10 20 300

0.05

0.1

0.15

Iterations

δ T

Florian Hofherr

Joint Optimization of Pose And Depth Using a Prox-Linear Approach

Problem Formulation Optimization Results

Experiment 2

Compare data loss functions

Absolute loss and special case (quadratic data loss)

Reduced step widths for special case

Florian Hofherr

Joint Optimization of Pose And Depth Using a Prox-Linear Approach

Problem Formulation Optimization Results

Comparison Absolute Loss vs. Special Case

Image Pair

Florian Hofherr

Joint Optimization of Pose And Depth Using a Prox-Linear Approach

Problem Formulation Optimization Results

Comparison Absolute Loss vs. Special Case

Result Absolute Loss

100

200

300

−1

−0.5

0

0.5

1

Florian Hofherr

Joint Optimization of Pose And Depth Using a Prox-Linear Approach

Problem Formulation Optimization Results

Comparison Absolute Loss vs. Special Case

Result Special Case

100

200

300

−1

−0.5

0

0.5

1

Florian Hofherr

Joint Optimization of Pose And Depth Using a Prox-Linear Approach

Problem Formulation Optimization Results

Comparison Absolute Loss vs. Special Case

Comparison Results Pose

blue: absolute loss

yellow: special case (quadratic loss)

0 50 100 150 2000

1

2

Iterations

δ win

[◦]

0 50 100 150 2000

0.02

0.04

0.06

0.08

0.1

Iterations

δ T

Florian Hofherr

Joint Optimization of Pose And Depth Using a Prox-Linear Approach

Problem Formulation Optimization Results

Comparison Absolute Loss vs. Special Case

Energy absolute loss

green: data part

cyan: Regularization part

0 20 40 60 80 100 120 140 160 180 200 220

105

106

107

Iterations

En

ergy

Florian Hofherr

Joint Optimization of Pose And Depth Using a Prox-Linear Approach

Problem Formulation Optimization Results

Comparison Absolute Loss vs. Special Case

Energy special case (quadratic loss)

green: data part

cyan: Regularization part

0 20 40 60 80 100 120 140 160 180 200 220106

107

108

Iterations

En

ergy

Florian Hofherr

Joint Optimization of Pose And Depth Using a Prox-Linear Approach

Problem Formulation Optimization Results

Questions?

Florian Hofherr

Joint Optimization of Pose And Depth Using a Prox-Linear Approach

Problem Formulation Optimization Results

Thank YouAnd

Merry Christmas

Florian Hofherr

Joint Optimization of Pose And Depth Using a Prox-Linear Approach