1

Linear View Synthesis Using a Dimensionality Gap Light Field Prior

Anat Levin and Fredo Durand

Weizmann Institute of Science & MIT CSAIL

22Light fields

Light field: the set of rays emitted from a scene in all possible directions

33Light fields

Novel view rendering

(Animation by Marc Levoy)

44Light fields

Novel view rendering

(Animation by Marc Levoy)

55Light fields

Novel view rendering Synthetic refocusing

(Animation by Marc Levoy)

66

u

v

4D light field

The set of light rays hitting the camera aperture plane is 4D:

• Ray hitting point- 2D

• Ray orientation- 2D

(In general: a 7D plenoptic space, including time and wavelength dimensions)

77Light field acquisition schemes and priors

Very different approaches to light field acquisition and manipulations exist in the literature.

The inherent difference between them is a different prior model on the light field space

88Light field acquisition schemes and priors

• 4D:

The light field is smooth, but involves 4 degrees of freedom

-Capture: 4D data (e.g. camera array)

-Inference: linear

99

• 4D:

-Capture: 4D data (e.g. camera array)

-Inference: linear

• 2D:

For Lambertian scenes all rays emerging from one point have same color. If depth is known, only 2 degrees of freedom

-Capture: 2D data (e.g. stereo camera)

-Inference: non linear depth estimation

Light field acquisition schemes and priors

1010In this talk: 3D light field prior

• 4D:

-Capture: 4D data (e.g. camera array

-Inference: linear

• 2D:-Capture: 2D data (e.g. stereo camera)

-Inference: non linear depth estimation

• 3D:

Depth is a 1D variable, hence the union of images at any depth covers no more than a 3D subset. Show that in the frequency domain there is only a 3D manifold of non zero entries.

-Capture: 3D data (e.g. focal stack)

-Inference: linear

uvy

x

1111Outline

• Linear view synthesis from a focal stack sequence

• The 3D light field prior

• Frequency derivation of synthesis algorithm

• Other applications of the 3D prior

1212Linear view synthesis with 3D prior

Output: Novel viewpoints (4D data)

2D Images x 2D set of novel viewpoints

Linear image processing

Input: Focal stack (3D data)

1D set of 2D images focused at different depth

1313Linear view synthesis algorithm

Average shifted images

Shift focal stack images by disparity

of desired view

Depth invariant deconvolution

1 2 3

No depth estimation!

1414

Shift invariant convolution~ focus sweep camera

Average shifted images

Ideal pinhole image

Depth invariant blur kernel

Inspiration: The focus sweep camera Hausler 72, Nagahara et al. 08

Captures a single image, average over all focus depths during exposure, provides

EDOF image from a single view

1515Linear view synthesis results

Video animation here

1616Disclaimers

• Novel viewpoints limited to the aperture area

• Convolution model breaks at occlusion boundaries

• Assume scene is Lambertian- in practice holds within the narrow range of angles of the aperture

1717Outline

• Linear view synthesis from a focal stack sequence

• The 3D light field prior

• Frequency derivation of synthesis algorithm

• Other applications of the 3D prior

18

y

xu

v

u

vy

x

• The set of light rays hitting the lens is 4D (x,y,u,v)

4D light field

(?,?,u0,0)

19

y

xu

v

u

v

y

x

• The set of light rays hitting the lens is 4D (x,y,u,v)

4D light field

(?,?,0,v0)

20

y

xu

v

u

v

y

x

• The set of light rays hitting the lens is 4D (x,y,u,v)

4D light field

21

y

xu

v

u

vy

x

• The set of light rays hitting the lens is 4D (x,y,u,v)

4D light field

22

uv

• The set of light rays hitting the lens is 4D

• Study the 4D Fourier domain L( , , , )x y u v(x,y,u,v)

4D light field spectrum

4D Fourier Transform

y

xu

v

y

x

4D Fourier Transform

23

uv

L( , , , )x y u v• The set of light rays hitting the lens is 4D

• Study the 4D Fourier domain

(x,y,u,v)

L( ,0,?,?)

uv

0x

y

xu

v

y

x

4D light field spectrum

24

y

xu

v uv

Frequency content only along 1D segments

4D Fourier Transform

y

x

4D light field spectrum

4D light field spectrum

Scene

4D Light field spectrum

Energy portion away from focal segments

26The slicing theorem

2D focused images at varying depths

y

xu

v

4D Fourier Transform

2D Fourier Transform

uvy

x

y

27The dimensionality gap

y

xu

v uvuv

4D Fourier Transformnear

far

depth color coding

uv

x

Light field spectrum: 4DImage spectrum: 2DDepth: 1D → Dimensionality gap

(Ng 05, Levin et al. 09)

Only the 3D manifold corresponding to physical focusing distance is useful

3D

28

uvGaussian prior: assigns non zero variance only to 3D set of entries on the focal segments

• Gaussian=> inference simple and linear

• Focal stack directly samples the manifold with non zero variance

y

x

3D Gaussian light field prior

29Outline

• Linear view synthesis from a focal stack sequence

• The 3D light field prior

• Frequency derivation of synthesis algorithm

• Other applications of the 3D prior

30

y

x

uv

View synthesis in the frequency domain4D spectrum of constant depth

scene

Spectra of focal stack images

1

Average focal stack spectra Spectra of

correct depthSample density

Deconvolution

(frequency domain)

31Outline

• Linear view synthesis from a focal stack sequence

• The 3D light field prior

• Frequency derivation of synthesis algorithm

• Other applications of the 3D prior

32Prior to infer light field from partial samples

In many other light field acquisition schemes we capture only a partial information on the light field- limited resolution, aliasing and each.

However, we capture linear measurements

On the other hand, we have a Gaussian prior, and we know the light field actually occupies only a low dimensional manifold of the 4D space.

Use the prior to “invert the rank deficient projection” and interpolate the measurements to get a light field with higher resolution, less aliasing.

33Improved viewpoints sample

4D Light field acquisition systems sample a 2D set of view points

• Can we do with sparser sample and 3D Gaussian prior for interpolation?

• How many samples needed? What is the right spacing?

• Shall we distribute samples on a grid? Better arrangement?

Grid: Standard sampling pattern

Circle: Sampling pattern with improved reconstruction

using 3D prior

34Superesolution of plenoptic camera

measurements

Plenoptic camera measurements are aliased

Replicas off the focal segments are high frequencies which we can re-bin and restore high frequency information

35Superesolution of plenoptic camera

measurements

Bicubic interpolation

Our result: applies for all depths simultaneously, no depth estimation

Lumsdaine and Georgiev: applies for a single known depth

36Summary

• Light field acquisition and synthesis strongly depends on light field prior

Existing priors:

• Linear view synthesis from the focal stack

• Other applications of 3D prior:

- viewpoints sample pattern

- depth invariant superesolution of plenoptic camera data

4D prior: capture- 4D data (e.g. camera array), inference- linear

2D prior: capture- 2D data (e.g. stereo), inference- non linear

Our new prior:

3D prior: capture- 3D data (e.g. focal stuck), inference linear