Course 3: Computational PhotographyCourse 3: Computational Photography

Ramesh RaskarMitsubishi Electric Research Labs

Jack TumblinNorthwestern University

Course WebPage :

http://www.merl.com/people/raskar/photo

Course 3: Computational PhotographyCourse 3: Computational Photography

Course WebPage

http://www.merl.com/people/raskar/photo

Course Evaluation

http://www.siggraph.org/courses_evaluation

WelcomeWelcome

• Understanding Film-like PhotographyUnderstanding Film-like Photography

– Parameters, Nonlinearities, Ray-based conceptsParameters, Nonlinearities, Ray-based concepts

• Image Processing and Reconstruction ToolsImage Processing and Reconstruction Tools

– Multi-image Fusion, Gradient domain, Graph CutsMulti-image Fusion, Gradient domain, Graph Cuts

• Improving Camera PerformanceImproving Camera Performance

– Better dynamic range, focus, frame rate, resolutionBetter dynamic range, focus, frame rate, resolution

• Future DirectionsFuture Directions

– HDR cameras, Gradient sensing, Smart optics/lightingHDR cameras, Gradient sensing, Smart optics/lighting

Speaker: Jack TumblinSpeaker: Jack Tumblin

Jack Tumblin is an Assistant Professor of Computer Science at Northwestern University.

His interests include novel photographic sensors to assist museum curators in historical preservation, computer graphics and visual appearance, and image-based modeling and rendering. During his doctoral studies at Georgia Tech and post-doc at Cornell, he investigated tone-mapping methods to depict high-contrast scenes. His MS in Electrical Engineering (December 1990) and BSEE (1978), also from Georgia Tech, bracketed his work as co-founder of IVEX Corp., (>45 people as of 1990) where his flight simulator design work was granted 5 US Patents. He is an Associate Editor of ACM Transactions on Graphics, was a member of the SIGGRAPH Papers Committee (2003, 2004), and in 2001 was a Guest Editor of IEEE Computer Graphics and Applications.

http://www.cs.northwestern.edu/~jet

Speaker: Ramesh RaskarSpeaker: Ramesh Raskar

Ramesh Raskar is a Senior Research Scientist at MERL.

His research interests include projector-based graphics, computational photography and non-photorealistic rendering. He has published several articles on imaging and photography including multi-flash photography for depth edge detection, image fusion, gradient-domain imaging and projector-camera systems. His papers have appeared in SIGGRAPH, EuroGraphics, IEEE Visualization, CVPR and many other graphics and vision conferences. He was a course organizer at Siggraph 2002, 2003 and 2004. He is a panel organizer at the Symposium on Computational Photography and Video in Cambridge, MA in May 2005. He is a member of the ACM and IEEE.

http://www.merl.com/people/raskar/raskar.html

OpportunitiesOpportunities

– Unlocking PhotographyUnlocking Photography• How to expand camera capabilitiesHow to expand camera capabilities• Digital photography that goes beyond film-like photographyDigital photography that goes beyond film-like photography

– OpportunitiesOpportunities• Computing corrects for lens, sensor and lighting limitationsComputing corrects for lens, sensor and lighting limitations• Computing merges results from multiple imagesComputing merges results from multiple images• Computing reconstructs from coded image samplesComputing reconstructs from coded image samples• Cameras benefit from computerized light sourcesCameras benefit from computerized light sources

– Think beyond post-capture image processingThink beyond post-capture image processing• Computation well before image processing and editing Computation well before image processing and editing

– Learn how to build your own camera-toysLearn how to build your own camera-toys

Traditional PhotographyTraditional Photography

Lens

Detector

Pixels

Image

Computational PhotographyComputational Photography: :

Optics, Sensors and ComputationsOptics, Sensors and ComputationsGeneralized

Sensor

Generalized Optics

Computations

Picture

4D Ray Bender

Upto 4D Ray Sampler

Ray Reconstruction

Computational Computational PhotographyPhotography

Novel CamerasGeneralized

Sensor

Generalized Optics

Processing

Computational Computational PhotographyPhotography

Novel Illumination

Novel CamerasGeneralized

Sensor

Generalized Optics

Processing

Light Sources

Computational Computational PhotographyPhotography

Novel Illumination

Novel Cameras

Scene: 8D Ray Modulator

GeneralizedSensor

Generalized Optics

Processing

Light Sources

Computational Computational PhotographyPhotography

Novel Illumination

Novel Cameras

Scene: 8D Ray Modulator

Display

GeneralizedSensor

Generalized Optics

Processing

Recreate 4D Lightfield

Light Sources

Computational Computational PhotographyPhotography

Novel Illumination

Novel Cameras

Scene: 8D Ray Modulator

Display

GeneralizedSensor

Generalized Optics

Processing

4D Ray BenderUpto 4D

Ray Sampler

Ray Reconstruction

Generalized Optics

Recreate 4D Lightfield

Light Sources

Modulators

4D Incident Lighting

4D Light Field

A Teaser: Dual PhotographyA Teaser: Dual Photography

Scene

PhotocellProjector

A Teaser: Dual PhotographyA Teaser: Dual Photography

Scene

PhotocellProjector

A Teaser: Dual PhotographyA Teaser: Dual Photography

Scene

PhotocellProjector

A Teaser: Dual PhotographyA Teaser: Dual Photography

Scene

PhotocellProjector Camera

camera

The 4D transport matrix:The 4D transport matrix: Contribution of each projector pixel to each camera pixelContribution of each projector pixel to each camera pixel

scene

projector

camera

The 4D transport matrix:The 4D transport matrix: Contribution of each projector pixel to each camera pixelContribution of each projector pixel to each camera pixel

scene

projector

Sen et al, Siggraph 2005Sen et al, Siggraph 2005

camera

The 4D transport matrix:The 4D transport matrix: Which projector pixel contribute to each camera pixelWhich projector pixel contribute to each camera pixel

scene

projector

Sen et al, Siggraph 2005Sen et al, Siggraph 2005

??

Dual photographyDual photographyfrom diffuse reflectionsfrom diffuse reflections

the camera’s viewSen et al, Siggraph 2005Sen et al, Siggraph 2005

Camera Obscura, Gemma Frisius, 1558

1558A Brief History of Images

Lens Based Camera Obscura, 1568

15581568

A Brief History of Images

Still Life, Louis Jaques Mande Daguerre, 1837

1558

1837

1568A Brief History of Images

Silicon Image Detector, 1970

1558

1837

1568

1970

A Brief History of Images

1558

1837

1568

1970

1994

A Brief History of Images

Digital Cameras

Dream of A New PhotographyDream of A New Photography

Old New• People and Time ~Cheap Precious• Each photo Precious Free• Lighting Critical Automated*• External Sensors No Yes• ‘Stills / Video’ Disjoint Merged

• Exposure Settings Pre-select Post-Process• Exposure Time Pre-select Post-Process• Resolution/noise Pre-select Post-Process• ‘HDR’ range Pre-select Post-Process

SurveySurvey

• How many of you are How many of you are photographersphotographers ? ?• How many of you are How many of you are photo-artistsphoto-artists ? ?

• How many of you are involved in camera How many of you are involved in camera design/developmentdesign/development ? ?

• How many do How many do active programmingactive programming ? ?

• Field of workField of work: : Academics? Industry ? Research ? Art ?Academics? Industry ? Research ? Art ?

8:30 Introduction (Raskar)

8:40 Photographic Signal & Film-like Photography (Tumblin)

9:10 Image Processing Tools (Raskar)

9:40 Improving Film-like Photography (Tumblin)

10:15 Break

10:30 Image Reconstruction Techniques (Raskar)

11:15 Smart Lights and Beyond Photography (Tumblin)

11:45 Smart Optics and Sensors (Raskar)

12:05 Discussion

ScheduleSchedule

Course Page : http://www.merl.com/people/raskar/photo