Accurate Depth and Normal Maps from Occlusion-Aware Focal Stack Symmetry

Michael Strecke, Anna Alperovich, and Bastian Goldluecke

Computer Vision and Image AnalysisUniversity of Konstanz

Contributions

• Novel way to handle occlusions when constructing cost volumes based on focalstack symmetry

• Joint regularization of depth and normals for smooth normal maps consistentwith depth estimate

Light field structure and focal stacksLight fields are defined as 4D function L : Π× Ω→ R on ray space,

where rays are given by intersection points with the focal plane Π and the image plane Ω.

Refocusing to disparity α: aperture filter σ over subaperture views v = (s, t),

ϕp(α) =

∫Π

σ(v)L(p + αv,v) dv. (1)

x

y

t

s

Lin et al. [6]: in absence of occlusion, the focal stack is symmetric around the ground

truth disparity. Assignment cost for disparities measures symmetry,

sϕp(α) =

∫ δmax

0

ρ(ϕp(α− δ)− ϕp(α + δ)) dδ. (2)

1 B B

B

d Bd+δ

Bd

Bd−δ

1 B B

B

d Bd+δ

Bd−δ

Occlusion-aware focal stack symmetryProblem: Focal stack at occlusions not symmetric around true disparity

Our assumption: occlusions only in one half-plane of view points

Then we can prove symmetry in partial focal stacks,

ϕ−e,p(d + δ) = ϕ+e,p(d− δ), where ϕ−e,p(α) =

∫ 0

−∞L(p + αse, se) ds

ϕ+e,p(α) =

∫ ∞0

L(p + αse, se) ds.

(3)

Our new disparity cost encourages symmetry for partial horizontal and vertical stacks:

sϕp(α) =

∫ δmax

0

min(ρ(ϕ−(1,0),p(α + δ)− ϕ+

(1,0),p(α− δ)),

ρ(ϕ−(0,1),p(α + δ)− ϕ+(0,1),p(α− δ))

)dδ.

(4)

(a) slice through focal stack ϕ from [6]

(b) slice through ϕ+

(c) slice through ϕ−

CV GT disp Lin et al. [6] Proposed

Box

es

28.1

2

20.8

7

Cot

ton

6.83

3.37

Din

o

10.0

1

3.52

Sid

eboa

rd

26.8

0

9.23

This work was supported by the ERC Starting Grant ”Light Field Imaging and Analysis” (LIA 336978, FP7-2014).

Presented at the Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, USA, July 2017.

Joint depth and normal map optimizationProblem: global optimal solution obtained with sublabel relaxation [7] locally flat.

Our approach: novel prior on normal maps which enforces correct relation to depth as

well as smoothness of the normal field:

E(ζ,n) = minα>0

∫Ω

ρ(ζ, x) + λ ‖Nζ − αn‖2 dx + R(n) dx (5)

where R(n) is a regularizer for the normal field given as

R(n) = supw∈C1c (Ω,Rn×m)

∫Ω

α ‖w −Dn‖ + γg ‖Dw‖F dx (6)

and reparametrized depth ζ := 12z

2 is related to normals by a linear operator N(ζ) [2].

Optimization for depth: Terms not dependent on ζ are removed, resulting in saddle

point problem:minζ,α>0

max‖p‖2≤λ,|ξ|≤1

(p, Nζ − αn) +

(ξ, ρ|ζ0 + (ζ − ζ0)∂ζρ|ζ0).

(7)

Solved using the primal-dual algorithm [1].

Optimization for normals: Removing all terms not depending on n: L1 denoising

problem

min‖n‖=1

∫Ω

λ ‖Nζ‖ ‖w − n‖ dx + R(n). (8)

Nonconvex due to ‖n‖ = 1: adoption of ideas from [10] for solution (local

parameterization of tangent space, effective linearization).

Comparison of normal maps for different methodsground truth normals sublabel relaxation [7] normal smoothing [10] depth smoothing [2] proposed method

Box

es

48.5

5

36.9

2

19.6

7

17.1

1

Cot

ton

33.6

4

31.5

4

12.8

7

9.78

Din

o

43.5

5

29.9

1

2.67

1.68

Sid

eboa

rd

49.5

9

65.4

8

12.4

9

8.58

Combining cost terms can improve robustnesscenter view focal stack symmetry focal stack + stereo

37.67 21.61

Accurate Depth and Normal Maps from Occlusion-Aware Focal Stack Symmetry

Michael Strecke, Anna Alperovich, and Bastian Goldluecke

Computer Vision and Image AnalysisUniversity of Konstanz

Experiments

Results on benchmark [3]EPI1 [5] EPI2 [9] LF [4] LF OCC [8] MV [3] Proposed

39.3

1

41.6

7

27.6

4

35.6

1

40.2

7

22.1

4

box

es

52.1

6

44.1

9

29.9

2

37.5

7

48.9

5

17.1

1

15.3

5

18.1

4

7.92

6.55

9.27

3.01

cott

on

57.9

1

23.1

4

23.2

6

30.3

5

38.5

0

9.78

10.4

9

15.9

1

19.0

5

14.7

7

6.50

3.43

dino

26.2

9

8.64

37.6

0

74.2

2

19.5

3

1.68

18.5

7

20.5

2

22.0

8

18.3

2

18.8

8

9.57

side

boa

rd

32.7

3

27.8

7

37.0

9

74.5

0

42.6

2

8.58

References[1] A. Chambolle and T. Pock, A first-order primal-dual algorithm

for convex problems with applications to imaging.

J. Math. Imaging Vis., 40(1):120–145, 2011.

[2] G. Graber, J. Balzer, S. Soatto, and T. Pock, Efficient

minimal-surface regularization of perspective depth maps in

variational stereo.

In Proc. International Conference on Computer Vision and

Pattern Recognition, 2015.

[3] K. Honauer, O. Johannsen, D. Kondermann, and

B. Goldluecke, A dataset and evaluation methodology for

depth estimation on 4d light fields.

In Asian Conf. on Computer Vision, 2016.

[4] H. Jeon, J. Park, G. Choe, J. Park, Y. Bok, Y. Tai, and

I. Kweon, Accurate depth map estimation from a lenslet light

field camera.

In Proc. International Conference on Computer Vision and

Pattern Recognition, 2015.

[5] O. Johannsen, A. Sulc, and B. Goldluecke, What sparse light

field coding reveals about scene structure.

In Proc. International Conference on Computer Vision and

Pattern Recognition, 2016.

[6] H. Lin, C. Chen, S.-B. Kang, and J. Yu, Depth recovery from

light field using focal stack symmetry.

In Proc. International Conference on Computer Vision, 2015.

[7] T. Moellenhoff, E. Laude, M. Moeller, J. Lellmann, and

D. Cremers, Sublabel-accurate relaxation of nonconvex

energies.

In Proc. International Conference on Computer Vision and

Pattern Recognition, 2016.

[8] T. Wang, A. Efros, and R. Ramamoorthi, Occlusion-aware

depth estimation using light-field cameras.

In Proceedings of the IEEE International Conference on

Computer Vision, pages 3487–3495, 2015.

[9] S. Wanner and B. Goldluecke, Variational light field analysis

for disparity estimation and super-resolution.

IEEE Transactions on Pattern Analysis and Machine

Intelligence, 36(3):606–619, 2014.

[10] B. Zeisl, C. Zach, and M. Pollefeys, Variational regularization

and fusion of surface normal maps.

In Proc. International Conference on 3D Vision (3DV), 2014.

Real-world results