Inferring Differential Structure from Defocused Imagesfmannan/766/shape_inference_dfd.pdf · Variational Relaxation Update Equation (Sander and Zucker [1]) D^ i(v) = 1 jN ij X j2N

Inferring Differential Structure from Defocused Images

Fahim Mannan

School of Computer ScienceMcGill University

[email protected]

April 8, 2014

Fahim Mannan (SOCS,McGill University) COMP 766 April 8, 2014 1 / 26

Overview

1 Introduction

2 Overview of ApproachDepth From DefocusQuadric Surface FittingVariational RelaxationSurface Re-estimation

3 Conclusion


Objective

Goal?

Estimate differential structure from a pair of defocused images.


Input

(a) Near Focused (b) Far Focused


Output

(c) Depthmap (d) 3D

along with Normal and Principal Curvatures at every pixel.


Darboux Frame

Local representation of a surface.

N: Normal

M: Principle curvature(max direction)

m: Principle curvature(min direction)


Darboux Frame

Local representation of a surface.

N: Normal

M: Principle curvature(max direction)

m: Principle curvature(min direction)


Overview of the Algorithm

1 Initial depth estimate using Depth from Defocus (DFD)

2 Quadric surface fitting at every pixel and compute initial Darbouxframe

3 Refine initial estimate using variational relaxation

4 Use the refined Darboux frame estimates to re-estimating the surface




















Related Work

1 Based on Sander and Zucker’s “Inferring Surface Trace andDifferential Structure” [1].

2 Ferrie et al. “Darboux Frames, Snakes and Super-Quadrics” [2]

3 Mathur et al. Curvature Consistency using Optimal EstimationTheory [3]

4 Savadjiev et al. Surface Recovery and Geometric Flow [4]

Note1 [1] considers volumetric images.

2 [2, 3, 4] 3D point cloud acquired using laser scanner.

3 This project uses two defocused images, and

4 Depth (noisy and low resolution) estimated using DFD.


Related Work










Related Work










Overview of Depth from DefocusModeling Defocus Blur

f : Focal length.

A: Aperture.

N: f-number f /A.

s: Sensor distance.

u: Object distance.

r : Blur width.

ρ: pixels/mm.

σ: Blur in pixels.

Image Plane

A

us

v

fr

Defocus Blur Equation

σ = ρr = ρfs

2N(

1

f− 1

u− 1

s) (1)



f : Focal length.

A: Aperture.

N: f-number f /A.

s: Sensor distance.

u: Object distance.

r : Blur width.

ρ: pixels/mm.

σ: Blur in pixels.

Image Plane

A

us

v

fr


σ = ρr = ρfs

2N(

1

f− 1

u− 1

s) (1)



0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 20

2

4

6

8

10

12

1/u m−1

blur

rad

ius

(pix

els)

focus = 1 m, f =50 mm, f/A = 22

0.5 1 1.5 2 2.5 3 3.5 4 4.5 50

2

4

6

8

10

12

u (meters)bl

ur r

adiu

s (p

ixel

s)

focus = 1 m, f =50 mm, f/A = 22


|σ| = ρfs

2N|1f− 1

u− 1

s|


Overview of Depth from DefocusModeling Defocused Images

Let,

I0 : True sharp image

I : Observed image

H(|σ|) : Defocus PSF

N (0, σ2N): Gaussian noise

(a) Pillbox

(b) Gaussian

Defocused Image Formation Equation

I = I0 ∗ H(|σ|) + η, where, η ∈ N (0, σ2N) (2)


Overview of Depth from DefocusDepth Estimation

Let,

IS : Sharper image

IB : Blurrier image

HS & HB : PSFs for the sharper and blurrier images

ηS & ηB : Gaussian noise from N (0, σ2N)

Pair of Defocused Images

IS = I0 ∗ HS + ηS (3)

IB = I0 ∗ HB + ηB (4)



Let,

IS : Sharper image

IB : Blurrier image




IS = I0 ∗ HS + ηS (3)

IB = I0 ∗ HB + ηB (4)



Let,

IS : Sharper image

IB : Blurrier image




IS = I0 ∗ HS + ηS (3)

IB = I0 ∗ HB + ηB (4)



Relative Blur

HB = HS ∗ HR (5)

Relative Blur Radius

σS = min{|σ1|, |σ2|}σB = max{|σ1|, |σ2|}

σR =√σ2B − σ2S (6)



Relative Blur

HB = HS ∗ HR (5)

Relative Blur Radius

σS = min{|σ1|, |σ2|}σB = max{|σ1|, |σ2|}

σR =√σ2B − σ2S (6)



A Simplification

Recall Eqs. 3 and 4,

IS = I0 ∗ HS + ηS

IB = I0 ∗ HB + ηB

Assume,

IB ≈ IS ∗ HR + ηB (7)

Relative Blur Estimation Problem

arg minHR

‖IB − IS ∗ HR‖ (8)



A Simplification

Recall Eqs. 3 and 4,

IS = I0 ∗ HS + ηS

IB = I0 ∗ HB + ηB

Assume,

IB ≈ IS ∗ HR + ηB (7)

Relative Blur Estimation Problem

arg minHR

‖IB − IS ∗ HR‖ (8)


Overview of Depth from DefocusImplementation

1 Apply isotropic diffusion to the sharper image i.e. IS ∗ HR

2 For every pixel choose HR that minimizes ‖IB − IS ∗ HR‖3 Compute depth corresponding to HR




2 For every pixel choose HR that minimizes ‖IB − IS ∗ HR‖

3 Compute depth corresponding to HR




2 For every pixel choose HR that minimizes ‖IB − IS ∗ HR‖3 Compute depth corresponding to HR


Initial Quadric Surface Fitting

Quadric Surface Model

φ(u, v) = (u, v , h(u, v))

where, h(u, v) =1

2au2 + buv +

1

2cv2 + eu + fv + g

Quadric Fitting for a Patch

1 For every 7× 7 patch.

2 Fit the quadric in the perspective projection space usingLeast-Squares.


Variational RelaxationExample

Figure: Neighborhood Support








Variational Relaxation

Variational Relaxation Update Equation (Sander and Zucker [1])

D̂i (v) =1

|Ni |∑j∈Ni

Dj(v) (9)

Information Theoretic Based Update (Mathur et al. [3])

D̂i (v) =1

|Ni |∑j∈Ni

λjDj(v) (10)

where λj is a measure of confidence in node j .

Proposed Weight

λ′j = λj f (‖IB − IS ∗ HR‖). (11)




D̂i (v) =1

|Ni |∑j∈Ni

Dj(v) (9)


D̂i (v) =1

|Ni |∑j∈Ni

λjDj(v) (10)


Proposed Weight

λ′j = λj f (‖IB − IS ∗ HR‖). (11)




D̂i (v) =1

|Ni |∑j∈Ni

Dj(v) (9)


D̂i (v) =1

|Ni |∑j∈Ni

λjDj(v) (10)


Proposed Weight

λ′j = λj f (‖IB − IS ∗ HR‖). (11)


Surface Re-estimation

Energy Minimization Model

E (s) = arg mins

∑‖IB − IS ∗ HR(s)‖+ α

∑(r ,θ)

(‖∇2

(r ,θ)s‖ − κ(r ,θ))

(12)

Where, κ(r ,θ) is the estimated curvature in a direction (r , θ).


Results

Figure: 3D point cloud with normal


Results

(a) Initial (b) Iteration 10

Figure: XY-plane projection of normals. Starting with noisy depthmap.


Results

(a) Initial (b) Iteration 10

Figure: XY-plane projection of normals. Starting with MAP-MRF depthmap.


Observations/Limitations

Noise needs to be low to get meaningful estimates.

Low resolution depthmap =⇒ mostly planar.

Boundary needs to be handled carefully.


Next Steps

Use λ′ weighting in variational relaxation.

Re-estimate surface using the refined estimates.


References

[1] P. T. Sander and S. W. Zucker, “Inferring surface trace and differential structurefrom 3-d images,” IEEE PAMITC, vol. 12, no. 9, pp. 833–854, 1990.

[2] F. Ferrie, J. Lagarde, and P. Whaite, “Darboux frames, snakes, and super-quadrics:geometry from the bottom up,” PAMI, IEEE Transactions on, vol. 15, no. 8, pp.771–784, Aug 1993.

[3] S. Mathur and F. Ferrie, “Edge localization in surface reconstruction using optimalestimation theory,” in CVPR, 1997., Jun 1997, pp. 833–838.

[4] P. Savadjiev, F. Ferrie, and K. Siddiqi, “Surface recovery from 3d point data using acombined parametric and geometric flow approach,” in EMMCVPR, 2003, vol.2683, pp. 325–340.


The End


Documents

Inferring Differential Structure from Defocused Imagesfmannan/766/shape_inference_dfd.pdf · Variational Relaxation Update Equation (Sander and Zucker [1]) D^ i(v) = 1 jN ij X j2N