Computational photography research at Stanford - … Photography... · Computational photography...

Computational photography research at Stanford

Marc LevoyComputer Science DepartmentStanford University

January 2012

! Marc Levoy

Outline

! Light field photography

! FCam and the Stanford Frankencamera

! User interfaces for computational photography

! Ongoing projects• FrankenSLR• burst-mode photography• languages and architectures for programmable cameras• computational videography and cinematography• multi-bucket pixels

2

Light field photography using a handheld plenoptic camera

Ren Ng, Marc Levoy, Mathieu Brédif,Gene Duval, Mark Horowitz and Pat Hanrahan

(Proc. SIGGRAPH 2005and TR 2005-02)

! Marc Levoy

Light field photography[Ng SIGGRAPH 2005]

Prototype camera

4000 ! 4000 pixels ÷ 292 ! 292 lenses = 14 ! 14 pixels per lens

Contax medium format camera Kodak 16-megapixel sensor

Adaptive Optics microlens array 125µ square-sided microlenses

! 2010 Marc Levoy

Digital refocusing

!

!

! 2010 Marc Levoy

Example of digital refocusing

! 2010 Marc Levoy

Refocusing portraits

! 2010 Marc Levoy

Light field photography

(Flash demo)

• trades off (excess) spatial resolution forability to refocus and adjust the perspective

• sensor pixels should be made even smaller, subject to the diffraction limit, for example: 36mm ! 24mm ÷ 2.5µ pixels = 266 Mpix 20K ! 13K pixels 4000 ! 2666 pixels ! 20 ! 20 rays per pixelor 2000 ! 1500 pixels ! 3 ! 3 rays per pixel = 27 Mpix

! Marc Levoy

Commercialization

Camera 2.0Marc Levoy

Computer Science DepartmentStanford University

! 2010 Marc Levoy

! facilitate research in experimental computational photography

! for students in computational photography courses worldwide

! proving ground for plugins and apps for future cameras

The Stanford Frankencameras

14

Frankencamera F2 Nokia N900 “F”

! 2010 Marc Levoy

CS 448A - Computational Photography

15

! 2010 Marc Levoy

CS 448 projectsM. Culbreth, A. Davis!! ! ! A phone-to-phone 4D light field fax machine

J. Koberstein, A. Mattos!! ! Motion blur detection as a game controller N. Sakunkoo! ! ! ! ! ! ! Camera-assisted self portraits on mobile devices C. DuHadway!! ! ! ! ! ! Driving/parking assistive mosaics using car cameras C.-H. Chen, S.H. Park! ! ! High dynamic range (HDR) viewfinding V. Chandrasekhar, G. Takacs!High dynamic range imaging on the Nokia N95 L. Kim, S. Que ! ! ! ! User interfaces for adjusting focus and tone A.Y.L. Chan, S. Meador!! !Photography using circular/hemispherical ring flash P. Huang, Z. Zheng !! ! Image deblurring from multiple images M. Smith !! ! ! ! ! ! ! Representative (non-real) thumbnails for small screens A. Medearis ! ! ! ! ! ! Multi-shot noise reduction for sports photography D. Johnson !! ! ! ! ! ! Synthetic aperture photography using a painted aperture D. Chan !! ! ! ! ! ! Synthetic aperture rear view mirror for safer driving

16

! 2010 Marc Levoy

Frankencamera architecture

17

Image Sensor

Lens

Metadata

Actions

Flash

...

ApplicationProcessor

Con!gure1

Expose2

Readout3

Image Processing

StatisticsCollection

Imaging Processor

4

+

Devices

Requests

Images andStatistics

! 2010 Marc Levoy18

Sensor sensor;Flash flash;vector<Shot> burst(2);

burst[0].exposure = 1/200.;burst[1].exposure = 1/30.;

Flash::FireAction fire(&flash);fire.time = burst[0].exposure/2;burst[0].actions.insert(fire);

sensor.stream(burst);

while (1) { Frame flashFrame = sensor.getFrame(); Frame noflashFrame = sensor.getFrame();}

Frankencamera software: the FCAM API

! 2010 Marc Levoy

Demonstration applications

19

• Canon 430EX (smaller flash) strobed continuously

• Canon 580EX (larger flash) fired once at end of exposure

! 2010 Marc Levoy

Fcam is being actively ported to...

! Android (at least by NVIDIA)

! iOS?

! Nokia/Microsoft?

! Android point-and-shoot camera?

20

! Marc Levoy

Prospects for computational photography

! the megapixel wars are over (and it’s about time)

! computational photography is the next battlegroundin the camera industry (it’s already starting)

21

! Marc Levoy

! this “headroom” permits more computation per pixel,or more frames per second, or less custom hardware

22

Premise: available computing power in cameras is rising faster than megapixels

0

3

6

9

12

2005 2006 2007 2008 2009 20100

17.5

35

52.5

70

billi

ons o

f pix

els /

seco

nd

meg

apix

els

Avg megapixels in new cameras, CAGR = 1.2NVIDIA GTX texture fill rate, CAGR = 1.8(CAGR for Moore’s law = 1.5)

! Marc Levoy

Interfaces for computational photography

! the megapixel wars are over (long overdue)

! computational photography is the next battlegroundin the camera industry (it’s already starting)

! how will these features appear to consumers?• standard and invisible• standard and visible (and disable-able)• aftermarket plugins and apps for your camera

23

! Marc Levoy

Interfaces for computational photography

! the megapixel wars are over (long overdue)

! computational photography is the next battlegroundin the camera industry (it’s already starting)

! how will these features appear to consumers?• standard and invisible• standard and visible (and disable-able)• aftermarket plugins and apps for your camera

! but aren’t cameras too complicated already?• won’t computational photography make this worse?

24

! Marc Levoy

Sony NEX-5! manual is 159 pages long

• and full of grammatical mistakes

! scene modes are not explained well• Sunset: Shoots the red of the sunset beautifully. (page 43)

! user interface is difficult to navigate

25

1234567

89

10111213

! Marc Levoy

ProHDR for the iPhone 4S

! most popular HDR capture method on the market

! app developers are making photography easy again!26

" 1 " " 2 "

27

Free

Free

28

Free

29

30

Free

circa 500,000 downloads in

2011

31

iPhone 4, single HD video frame

32

SynthCam, align & average ~30 frames

SNR increases as sqrt(# of frames)

SynthCam

Canon 5DII, 1/400 sec

Canon 5DII, average of 15 shots of 1/6 sec

(Marc Levoy)

(Marc Levoy)

! 2010 Marc Levoy

Removing foreground objectsby translating the camera

• align the shots• match histograms• apply median filter

! 2010 Marc Levoy

Ongoing projects

! Frankencamera F3 (“FrankenSLR”)• SLR-quality sensor, but big, heavy, and expensive ($10K)

39

Micron MT9001• cell phone quality• $150

Cypress LUPA 4000• DSLR quality• $1500• arbitrary ROIs and non-destructive readout

! 2010 Marc Levoy

Ongoing projects

! Frankencamera F3 (“FrankenSLR”)• SLR-quality sensor, but big, heavy, and expensive ($10K)

! high-speed burst-mode photography• all still cameras should capture at (up to) 500 fps• capture while aiming the camera - no shutter half-press• frameless photography - ROIs, MOIs (“M” = Moment)

40

! 2010 Marc Levoy

Burst-mode photography

41

! 2010 Marc Levoy

Ongoing projects

! Frankencamera F3 (“FrankenSLR”)• SLR-quality sensor, but big, heavy, and expensive ($10K)

! high-speed burst-mode photography• all still cameras should capture at (up to) 500 fps• capture while aiming the camera - no shutter half-press• frameless photography - ROIs, MOIs (“M” = Moment)

! languages and architectures for programmable cameras• FCam was an API for camera control• now we need APIs for photographic processing and computer vision

42

! 2010 Marc Levoy

Ongoing projects

! Frankencamera F3 (“FrankenSLR”)• SLR-quality sensor, but big, heavy, and expensive ($10K)

! high-speed burst-mode photography• all still cameras should capture at (up to) 500 fps• capture while aiming the camera - no shutter half-press• frameless photography - ROIs, MOIs (“M” = Moment)

! languages and architectures for programmable cameras• FCam was an API for camera control• now we need APIs for photographic processing and computer vision

! computational videography & cinematography• stabilization, stereo, view interpolation, free-viewpoint video

43

! Marc Levoy

3D video stabilization[Agarwala 2011]

44 (http://web.cecs.pdx.edu/~fliu/project/subspace_stabilization/demo.mp4)

! 2009 Marc Levoy

Camera 3.0:multi-bucket pixels

! a CMOS pixel architecturewith multiple accumulation nodes per pixel

! can flip between TX1 and TX2 very fast• all pixels flip at once

45

storage nodes

photodetector

TX2TX1

transfer gates

! 2009 Marc Levoy

Flash/noflash photography[Agrawal SIGGRAPH 2005]

46

TX2TX1put flash exposure

in middle of ambient exposure

another app:HDR capture

(with no differences in object motion

between exposures)

!"#$%&'()*$+,-'.+"/

High Dynamic Range Photography

47

Short exposure Long exposure HDR Image

!"#$%&'()*$+,-'.+"/

Motion artifacts in HDR

48

!"#$%&'()*$+,-'.+"/

HDR Photography using 4-bucket pixels

49

Images with different exposures tightly interleaved

!"#$%&'()*$+,-'.+"/

4-Bucket Pixel Layout (3D View)

50

http://graphics.stanford.edu/projects/camera-2.0/

Image Sensor

Lens

Metadata

Actions

Flash

...

ApplicationProcessor

Con!gure1

Expose2

Readout3

Image Processing

StatisticsCollection

Imaging Processor

4

+

Devices

Requests

Images andStatistics

Sensor sensor;Shot low, med, high;

low.exposure = 1/80.;med.exposure = 1/20.;high.exposure = 1/5.;

sensor.capture(low);sensor.capture(med);sensor.capture(high);

Frame frames[3];frames[0] = sensor.getFrame();frames[1] = sensor.getFrame();frames[2] = sensor.getFrame();

fused = mergeHDR(frames);