51
Computational photography research at Stanford Marc Levoy Computer Science Department Stanford University January 2012

Computational photography research at Stanford - … Photography... · Computational photography research at Stanford Marc Levoy Computer Science Department Stanford University January

Embed Size (px)

Citation preview

Page 1: Computational photography research at Stanford - … Photography... · Computational photography research at Stanford Marc Levoy Computer Science Department Stanford University January

Computational photography research at Stanford

Marc LevoyComputer Science DepartmentStanford University

January 2012

Page 2: Computational photography research at Stanford - … Photography... · Computational photography research at Stanford Marc Levoy Computer Science Department Stanford University January

! Marc Levoy

Outline

! Light field photography

! FCam and the Stanford Frankencamera

! User interfaces for computational photography

! Ongoing projects• FrankenSLR• burst-mode photography• languages and architectures for programmable cameras• computational videography and cinematography• multi-bucket pixels

2

Page 3: Computational photography research at Stanford - … Photography... · Computational photography research at Stanford Marc Levoy Computer Science Department Stanford University January

Light field photography using a handheld plenoptic camera

Ren Ng, Marc Levoy, Mathieu Brédif,Gene Duval, Mark Horowitz and Pat Hanrahan

(Proc. SIGGRAPH 2005and TR 2005-02)

Page 4: Computational photography research at Stanford - … Photography... · Computational photography research at Stanford Marc Levoy Computer Science Department Stanford University January

! Marc Levoy

Light field photography[Ng SIGGRAPH 2005]

Page 5: Computational photography research at Stanford - … Photography... · Computational photography research at Stanford Marc Levoy Computer Science Department Stanford University January

Prototype camera

4000 ! 4000 pixels ÷ 292 ! 292 lenses = 14 ! 14 pixels per lens

Contax medium format camera Kodak 16-megapixel sensor

Adaptive Optics microlens array 125µ square-sided microlenses

Page 6: Computational photography research at Stanford - … Photography... · Computational photography research at Stanford Marc Levoy Computer Science Department Stanford University January
Page 7: Computational photography research at Stanford - … Photography... · Computational photography research at Stanford Marc Levoy Computer Science Department Stanford University January
Page 8: Computational photography research at Stanford - … Photography... · Computational photography research at Stanford Marc Levoy Computer Science Department Stanford University January

! 2010 Marc Levoy

Digital refocusing

!

!

Page 9: Computational photography research at Stanford - … Photography... · Computational photography research at Stanford Marc Levoy Computer Science Department Stanford University January

! 2010 Marc Levoy

Example of digital refocusing

Page 10: Computational photography research at Stanford - … Photography... · Computational photography research at Stanford Marc Levoy Computer Science Department Stanford University January

! 2010 Marc Levoy

Refocusing portraits

Page 11: Computational photography research at Stanford - … Photography... · Computational photography research at Stanford Marc Levoy Computer Science Department Stanford University January

! 2010 Marc Levoy

Light field photography

(Flash demo)

Page 12: Computational photography research at Stanford - … Photography... · Computational photography research at Stanford Marc Levoy Computer Science Department Stanford University January

• trades off (excess) spatial resolution forability to refocus and adjust the perspective

• sensor pixels should be made even smaller, subject to the diffraction limit, for example: 36mm ! 24mm ÷ 2.5µ pixels = 266 Mpix 20K ! 13K pixels 4000 ! 2666 pixels ! 20 ! 20 rays per pixelor 2000 ! 1500 pixels ! 3 ! 3 rays per pixel = 27 Mpix

! Marc Levoy

Commercialization

Page 13: Computational photography research at Stanford - … Photography... · Computational photography research at Stanford Marc Levoy Computer Science Department Stanford University January

Camera 2.0Marc Levoy

Computer Science DepartmentStanford University

Page 14: Computational photography research at Stanford - … Photography... · Computational photography research at Stanford Marc Levoy Computer Science Department Stanford University January

! 2010 Marc Levoy

! facilitate research in experimental computational photography

! for students in computational photography courses worldwide

! proving ground for plugins and apps for future cameras

The Stanford Frankencameras

14

Frankencamera F2 Nokia N900 “F”

Page 15: Computational photography research at Stanford - … Photography... · Computational photography research at Stanford Marc Levoy Computer Science Department Stanford University January

! 2010 Marc Levoy

CS 448A - Computational Photography

15

Page 16: Computational photography research at Stanford - … Photography... · Computational photography research at Stanford Marc Levoy Computer Science Department Stanford University January

! 2010 Marc Levoy

CS 448 projectsM. Culbreth, A. Davis!! ! ! A phone-to-phone 4D light field fax machine

J. Koberstein, A. Mattos!! ! Motion blur detection as a game controller N. Sakunkoo! ! ! ! ! ! ! Camera-assisted self portraits on mobile devices C. DuHadway!! ! ! ! ! ! Driving/parking assistive mosaics using car cameras C.-H. Chen, S.H. Park! ! ! High dynamic range (HDR) viewfinding V. Chandrasekhar, G. Takacs!High dynamic range imaging on the Nokia N95 L. Kim, S. Que ! ! ! ! User interfaces for adjusting focus and tone A.Y.L. Chan, S. Meador!! !Photography using circular/hemispherical ring flash P. Huang, Z. Zheng !! ! Image deblurring from multiple images M. Smith !! ! ! ! ! ! ! Representative (non-real) thumbnails for small screens A. Medearis ! ! ! ! ! ! Multi-shot noise reduction for sports photography D. Johnson !! ! ! ! ! ! Synthetic aperture photography using a painted aperture D. Chan !! ! ! ! ! ! Synthetic aperture rear view mirror for safer driving

16

Page 17: Computational photography research at Stanford - … Photography... · Computational photography research at Stanford Marc Levoy Computer Science Department Stanford University January

! 2010 Marc Levoy

Frankencamera architecture

17

Image Sensor

Lens

Metadata

Actions

Flash

...

ApplicationProcessor

Con!gure1

Expose2

Readout3

Image Processing

StatisticsCollection

Imaging Processor

4

+

Devices

Requests

Images andStatistics

Page 18: Computational photography research at Stanford - … Photography... · Computational photography research at Stanford Marc Levoy Computer Science Department Stanford University January

! 2010 Marc Levoy18

Sensor sensor;Flash flash;vector<Shot> burst(2);

burst[0].exposure = 1/200.;burst[1].exposure = 1/30.;

Flash::FireAction fire(&flash);fire.time = burst[0].exposure/2;burst[0].actions.insert(fire);

sensor.stream(burst);

while (1) { Frame flashFrame = sensor.getFrame(); Frame noflashFrame = sensor.getFrame();}

Frankencamera software: the FCAM API

Page 19: Computational photography research at Stanford - … Photography... · Computational photography research at Stanford Marc Levoy Computer Science Department Stanford University January

! 2010 Marc Levoy

Demonstration applications

19

• Canon 430EX (smaller flash) strobed continuously

• Canon 580EX (larger flash) fired once at end of exposure

Page 20: Computational photography research at Stanford - … Photography... · Computational photography research at Stanford Marc Levoy Computer Science Department Stanford University January

! 2010 Marc Levoy

Fcam is being actively ported to...

! Android (at least by NVIDIA)

! iOS?

! Nokia/Microsoft?

! Android point-and-shoot camera?

20

Page 21: Computational photography research at Stanford - … Photography... · Computational photography research at Stanford Marc Levoy Computer Science Department Stanford University January

! Marc Levoy

Prospects for computational photography

! the megapixel wars are over (and it’s about time)

! computational photography is the next battlegroundin the camera industry (it’s already starting)

21

Page 22: Computational photography research at Stanford - … Photography... · Computational photography research at Stanford Marc Levoy Computer Science Department Stanford University January

! Marc Levoy

! this “headroom” permits more computation per pixel,or more frames per second, or less custom hardware

22

Premise: available computing power in cameras is rising faster than megapixels

0

3

6

9

12

2005 2006 2007 2008 2009 20100

17.5

35

52.5

70

billi

ons o

f pix

els /

seco

nd

meg

apix

els

Avg megapixels in new cameras, CAGR = 1.2NVIDIA GTX texture fill rate, CAGR = 1.8(CAGR for Moore’s law = 1.5)

Page 23: Computational photography research at Stanford - … Photography... · Computational photography research at Stanford Marc Levoy Computer Science Department Stanford University January

! Marc Levoy

Interfaces for computational photography

! the megapixel wars are over (long overdue)

! computational photography is the next battlegroundin the camera industry (it’s already starting)

! how will these features appear to consumers?• standard and invisible• standard and visible (and disable-able)• aftermarket plugins and apps for your camera

23

Page 24: Computational photography research at Stanford - … Photography... · Computational photography research at Stanford Marc Levoy Computer Science Department Stanford University January

! Marc Levoy

Interfaces for computational photography

! the megapixel wars are over (long overdue)

! computational photography is the next battlegroundin the camera industry (it’s already starting)

! how will these features appear to consumers?• standard and invisible• standard and visible (and disable-able)• aftermarket plugins and apps for your camera

! but aren’t cameras too complicated already?• won’t computational photography make this worse?

24

Page 25: Computational photography research at Stanford - … Photography... · Computational photography research at Stanford Marc Levoy Computer Science Department Stanford University January

! Marc Levoy

Sony NEX-5! manual is 159 pages long

• and full of grammatical mistakes

! scene modes are not explained well• Sunset: Shoots the red of the sunset beautifully. (page 43)

! user interface is difficult to navigate

25

1234567

89

10111213

Page 26: Computational photography research at Stanford - … Photography... · Computational photography research at Stanford Marc Levoy Computer Science Department Stanford University January

! Marc Levoy

ProHDR for the iPhone 4S

! most popular HDR capture method on the market

! app developers are making photography easy again!26

" 1 " " 2 "

Page 27: Computational photography research at Stanford - … Photography... · Computational photography research at Stanford Marc Levoy Computer Science Department Stanford University January

27

Free

Page 28: Computational photography research at Stanford - … Photography... · Computational photography research at Stanford Marc Levoy Computer Science Department Stanford University January

Free

28

Page 29: Computational photography research at Stanford - … Photography... · Computational photography research at Stanford Marc Levoy Computer Science Department Stanford University January

Free

29

Page 30: Computational photography research at Stanford - … Photography... · Computational photography research at Stanford Marc Levoy Computer Science Department Stanford University January

30

Free

circa 500,000 downloads in

2011

Page 31: Computational photography research at Stanford - … Photography... · Computational photography research at Stanford Marc Levoy Computer Science Department Stanford University January

31

iPhone 4, single HD video frame

Page 32: Computational photography research at Stanford - … Photography... · Computational photography research at Stanford Marc Levoy Computer Science Department Stanford University January

32

SynthCam, align & average ~30 frames

SNR increases as sqrt(# of frames)

Page 33: Computational photography research at Stanford - … Photography... · Computational photography research at Stanford Marc Levoy Computer Science Department Stanford University January

SynthCam

Page 34: Computational photography research at Stanford - … Photography... · Computational photography research at Stanford Marc Levoy Computer Science Department Stanford University January

Canon 5DII, 1/400 sec

Page 35: Computational photography research at Stanford - … Photography... · Computational photography research at Stanford Marc Levoy Computer Science Department Stanford University January

Canon 5DII, average of 15 shots of 1/6 sec

Page 36: Computational photography research at Stanford - … Photography... · Computational photography research at Stanford Marc Levoy Computer Science Department Stanford University January

(Marc Levoy)

Page 37: Computational photography research at Stanford - … Photography... · Computational photography research at Stanford Marc Levoy Computer Science Department Stanford University January

(Marc Levoy)

Page 38: Computational photography research at Stanford - … Photography... · Computational photography research at Stanford Marc Levoy Computer Science Department Stanford University January

! 2010 Marc Levoy

Removing foreground objectsby translating the camera

• align the shots• match histograms• apply median filter

Page 39: Computational photography research at Stanford - … Photography... · Computational photography research at Stanford Marc Levoy Computer Science Department Stanford University January

! 2010 Marc Levoy

Ongoing projects

! Frankencamera F3 (“FrankenSLR”)• SLR-quality sensor, but big, heavy, and expensive ($10K)

39

Micron MT9001• cell phone quality• $150

Cypress LUPA 4000• DSLR quality• $1500• arbitrary ROIs and non-destructive readout

Page 40: Computational photography research at Stanford - … Photography... · Computational photography research at Stanford Marc Levoy Computer Science Department Stanford University January

! 2010 Marc Levoy

Ongoing projects

! Frankencamera F3 (“FrankenSLR”)• SLR-quality sensor, but big, heavy, and expensive ($10K)

! high-speed burst-mode photography• all still cameras should capture at (up to) 500 fps• capture while aiming the camera - no shutter half-press• frameless photography - ROIs, MOIs (“M” = Moment)

40

Page 41: Computational photography research at Stanford - … Photography... · Computational photography research at Stanford Marc Levoy Computer Science Department Stanford University January

! 2010 Marc Levoy

Burst-mode photography

41

Page 42: Computational photography research at Stanford - … Photography... · Computational photography research at Stanford Marc Levoy Computer Science Department Stanford University January

! 2010 Marc Levoy

Ongoing projects

! Frankencamera F3 (“FrankenSLR”)• SLR-quality sensor, but big, heavy, and expensive ($10K)

! high-speed burst-mode photography• all still cameras should capture at (up to) 500 fps• capture while aiming the camera - no shutter half-press• frameless photography - ROIs, MOIs (“M” = Moment)

! languages and architectures for programmable cameras• FCam was an API for camera control• now we need APIs for photographic processing and computer vision

42

Page 43: Computational photography research at Stanford - … Photography... · Computational photography research at Stanford Marc Levoy Computer Science Department Stanford University January

! 2010 Marc Levoy

Ongoing projects

! Frankencamera F3 (“FrankenSLR”)• SLR-quality sensor, but big, heavy, and expensive ($10K)

! high-speed burst-mode photography• all still cameras should capture at (up to) 500 fps• capture while aiming the camera - no shutter half-press• frameless photography - ROIs, MOIs (“M” = Moment)

! languages and architectures for programmable cameras• FCam was an API for camera control• now we need APIs for photographic processing and computer vision

! computational videography & cinematography• stabilization, stereo, view interpolation, free-viewpoint video

43

Page 44: Computational photography research at Stanford - … Photography... · Computational photography research at Stanford Marc Levoy Computer Science Department Stanford University January

! Marc Levoy

3D video stabilization[Agarwala 2011]

44 (http://web.cecs.pdx.edu/~fliu/project/subspace_stabilization/demo.mp4)

Page 45: Computational photography research at Stanford - … Photography... · Computational photography research at Stanford Marc Levoy Computer Science Department Stanford University January

! 2009 Marc Levoy

Camera 3.0:multi-bucket pixels

! a CMOS pixel architecturewith multiple accumulation nodes per pixel

! can flip between TX1 and TX2 very fast• all pixels flip at once

45

storage nodes

photodetector

TX2TX1

transfer gates

Page 46: Computational photography research at Stanford - … Photography... · Computational photography research at Stanford Marc Levoy Computer Science Department Stanford University January

! 2009 Marc Levoy

Flash/noflash photography[Agrawal SIGGRAPH 2005]

46

TX2TX1put flash exposure

in middle of ambient exposure

another app:HDR capture

(with no differences in object motion

between exposures)

Page 47: Computational photography research at Stanford - … Photography... · Computational photography research at Stanford Marc Levoy Computer Science Department Stanford University January

!"#$%&'()*$+,-'.+"/

High Dynamic Range Photography

47

Short exposure Long exposure HDR Image

Page 48: Computational photography research at Stanford - … Photography... · Computational photography research at Stanford Marc Levoy Computer Science Department Stanford University January

!"#$%&'()*$+,-'.+"/

Motion artifacts in HDR

48

Page 49: Computational photography research at Stanford - … Photography... · Computational photography research at Stanford Marc Levoy Computer Science Department Stanford University January

!"#$%&'()*$+,-'.+"/

HDR Photography using 4-bucket pixels

49

Images with different exposures tightly interleaved

Page 50: Computational photography research at Stanford - … Photography... · Computational photography research at Stanford Marc Levoy Computer Science Department Stanford University January

!"#$%&'()*$+,-'.+"/

4-Bucket Pixel Layout (3D View)

50

Page 51: Computational photography research at Stanford - … Photography... · Computational photography research at Stanford Marc Levoy Computer Science Department Stanford University January

http://graphics.stanford.edu/projects/camera-2.0/

Image Sensor

Lens

Metadata

Actions

Flash

...

ApplicationProcessor

Con!gure1

Expose2

Readout3

Image Processing

StatisticsCollection

Imaging Processor

4

+

Devices

Requests

Images andStatistics

Sensor sensor;Shot low, med, high;

low.exposure = 1/80.;med.exposure = 1/20.;high.exposure = 1/5.;

sensor.capture(low);sensor.capture(med);sensor.capture(high);

Frame frames[3];frames[0] = sensor.getFrame();frames[1] = sensor.getFrame();frames[2] = sensor.getFrame();

fused = mergeHDR(frames);