Raskar, Camera Culture, MIT Media Lab

C t ti l Ph t hComputational Photography:Epsilon to Coded Imaging

Camera Culture

p g g

Camera Culture

Ramesh Raskar

C C ltCamera CultureAssociate Professor, MIT Media Lab http://raskar.info

Tools

for

Visual Computing

Shadow

Refractive

Reflective

Fernald, Science [Sept 2006]

How can we create an entirely new class of imaging platforms

that have an understanding of the world that far exceeds human ability

and produce meaningful abstractions that are well ithi h h ibilit ?within human comprehensibility ?

Ramesh Raskar http://raskar.info

Mitsubishi Electric Research Laboratories Raskar 2006Spatial Augmented Reality

CurvedPlanar Non-planar Pocket-ProjObjects

Computational IlluminationComputational Illumination

CurvedPlanar Non planar

SingleProjector

?

Pocket ProjObjects1998 2002 20021997

Projector

jUser : T

?

1998 2002 1999 20031998

MultipleProjectors

Computational Camera and PhotographyComputational Camera and Photography

Motion Blurred Photo

Sh t T diti l MURAShort Exposure

Traditional MURAShutter

Captured SinglePh tPhoto

Deblurred Result

Banding Artifacts and some spatial frequencies

Dark d i some spatial frequencies

are lostand noisy

Blurring == Convolution

Sh Bl d

Fourier Transform

PSF == Sinc Function

Sharp Photo

Blurred Photo

Traditional Camera: Shutter is OPEN: Box Filter

ω

Sh Bl d

Fourier Transform

Sharp Photo

Blurred PhotoPSF == Broadband Function

Preserves High Spatial Frequencies

Flutter Shutter: Shutter is OPEN and CLOSED

Flutter Shutter CameraFlutter Shutter CameraRaskar, Agrawal, Tumblin [Siggraph2006]

LCD opacity switched in coded sequence

Traditional

Coded Exposu

rere

Deblurred I

Deblurred I ImageImage

Image of Static Object

Coded Exposure Coded Aperture

Temporal 1-D broadband code: Motion Deblurring

Spatial 2-D broadband mask: Focus Deblurring

Coded Aperture CameraCoded Aperture Camera

The aperture of a 100 mm lens is modified

Rest of the camera is unmodifiedInsert a coded mask with chosen binary pattern

LED

In Focus Photo

Out of Focus Photo: Open Aperture

Out of Focus Photo: Coded Aperture

Captured Blurred Photo

Refocused on Person

Raskar, Camera Culture, MIT Media Lab

Computational Photography

1. Epsilon Photography– Low-level Vision: Pixels– Multiphotos by bracketing (HDR, panorama)– ‘Ultimate camera’

2. Coded Photography– Mid-Level Cues:

• Regions, Edges, Motion, Direct/globalg , g , , g– Single/few snapshot

• Reversible encoding of data– Additional sensors/optics/illum

3. Essence Photography– Not mimic human eyeNot mimic human eye– Beyond single view/illum– ‘New artform’

Raskar, Camera Culture, MIT Media Lab

• Ramesh Raskar and J k T bliJack Tumblin

• Book Publishers: A K Peters

Less is MoreLess is More

Blocking Light == More InformationBlocking Light == More Information

Coding in Time Coding in Time Coding in SpaceCoding in Space

Larval Trematode WormLarval Trematode Worm Coded Aperture CameraCoded Aperture Camera

Shielding Light …Shielding Light …g gg g

Larval Trematode WormLarval Trematode Worm Turbellarian WormTurbellarian Worm

Mask?

Sensor

MaskSensorMask

?

SensorMask?

Sensor

Sensor

Mask

4D Light Field from 2D Photo:

d h ld

Full Resolution Digital Refocusing:

Heterodyne Light Field Camera

Coded Aperture Camera

Light Field Inside a CameraLight Field Inside a Camera

Light Field Inside a CameraLight Field Inside a Camera

LensletLenslet--based Light Field camerabased Light Field camera

[Adelson and Wang, 1992, Ng et al. 2005 ]

Stanford Plenoptic Camera Stanford Plenoptic Camera [Ng et al 2005][Ng et al 2005]

Contax medium format camera Kodak 16-megapixel sensor

4000 × 4000 pixels ÷ 292 × 292 lenses = 14 × 14 pixels per lens

Adaptive Optics microlens array 125μ square-sided microlenses

Digital RefocusingDigital Refocusingg gg g

[Ng et al 2005][Ng et al 2005]

Can we achieve this with a Can we achieve this with a MaskMask alone?alone?Can we achieve this with a Can we achieve this with a MaskMask alone?alone?

Mask based Light Field CameraSensor

MaskSensor

[Veeraraghavan, Raskar, Agrawal, Tumblin, Mohan, Siggraph 2007 ]

How to Capture How to Capture 4D Light Field with g

2D Sensor ?

Wh t h ld b th What should be the pattern of the mask ?pattern of the mask ?

Mask Tile

Cosine Mask Used

Mask Tile

1/f1/f0

Captured 2D Photo

Encoding due to Mask

Sensor Slice captures entire Light Field

fθfθ0

fxfx0

M d l iModulated Light Field

Modulation Function

Computing 4D Light Field

2D Sensor Photo, 1800*1800 2D Fourier Transform, 1800*1800

2D FFT

9*9=81 spectral copies

Rearrange 2D tiles into 4D planes200*200*9*94D IFFT

4D Light Field200*200*9*9

Full resolution 2D image of Focused Scene Parts

Captured 2D Photo

divide

Image of White Lambertian Plane

Wavefront Sensing in Any Wavelength !

MaskSensor

[Veeraraghavan, Raskar, Agrawal, Tumblin, Mohan, Siggraph 2007 ]

Lens Flare Reduction/Enhancement using Lens Flare Reduction/Enhancement using 4D Ray Sampling4D Ray Sampling4D Ray Sampling4D Ray Sampling

Captured Glare Glare Captured Glare Reduced

Glare Enhanced

Glare = low frequency noise in 2D

•But is high frequency noise in 4D

•Remove via simple outlier rejection

i

Sensor

j

xu xu

Rays = Waves for Propagation and Interface

Fresnel propagation Chirp (Lens) Fourier transform Fractional Fourier transform

x2 x3 x4x1x1

x0

u2u1 u3b

¡ ba x0

u4

x2x0

- x0

a

x1x0x0- a

b- a x0

x3x4- b

a x0

a

x4

I

Imaging via volume hologram (Depth-specific Imaging)

KVH (x4=0, u4=θs/λ; x3, u3) -20

-15

-100.6

0.8

1 u4u3

u 3 [mm

-1] -5

0

5

10

150

0.2

0.4

0.6

x3 x4L

ZZ ½ µ ¶ ¾

x3 [mm]

-0.4 -0.2 0 0.2 0.4

15

20 -0.2

K V H (x4; u4; x3; u3) =ZZ

dx03dx0

4e¡ i 2¼(u 04 x 4 ¡ u 0

3 x 3 ) exp½

¡ i2¼̧ zf (u03 + u0

4)µ

¡ u3 + u4 ¡µs

¸

¶ ¾

£ sinc½

L¸µ

¡ u3 + u4 +u0

3 + u04

2

¶ µu4 +

u04

2¡

µs

¸

¶ ¾sinc

½L¸

µ¡ u3 + u4 ¡

u03 + u0

42

¶ µu4 ¡

u04

2¡

µs

¸

¶ ¾

K V H I (x2; u2; x1; u1)

Derivation: h(x2; x1) = exp½

¡ i¼¸

zf

f 2 (x1 + x2 ¡ f µs)2¾

sinc½

L¸ f 2 (x1 + x2) (x2 ¡ f µs)

¾

Parameters:0 5 ¹

K V H (x4; u4; x3; u3)¸ = 0.5 ¹m

µs= 30°L = 1 mm

zf = 50 mm

Raskar, Camera Culture, MIT Media Lab

Computational Photography

Camera Culture Group Ramesh Raskar http://raskar.info

Computational Photography1. Epsilon Photography

– Low-level Vision: PixelsMask

Sensor

– Multiphotos by bracketing (HDR, panorama)– ‘Ultimate camera’

2. Coded Photography– Mid-Level Cues:Mid Level Cues:

• Regions, Edges, Motion, Direct/global

• Coded Exposure– Flutter Shutter Motion Deblurring

• Coded Aperture– Defocus

• Optical Heterodyning• Optical Heterodyning– Lightfield or Wavefront sensing

• Coded Glare• 6D Display u 3 [m

m-1

]

-20

-15

-10

-5

0

5

100.2

0.4

0.6

0.8

1

p y• Femto-second Imaging• Rays = Waves x3 [mm]

-0.4 -0.2 0 0.2 0.4

15

20 -0.2

0

1 2

11

2D 2D 2D

How can we create an entirely new class of imaging platforms

that have an understanding of the world that far exceeds human ability

and produce meaningful abstractions that are well ithi h h ibilit ?within human comprehensibility ?

Ramesh Raskar http://raskar.info