View
2.388
Download
1
Category
Preview:
Citation preview
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Light Transport and
Computational Photography Inverse problems
MIT Media LabRamesh Raskar httpraskarinfo
raskarmitedu
Raskar Camera Culture MIT Media Lab
Ramesh Raskar MIT Media Lab
After X what is neXt
How to Invent
Ramesh Raskar MIT Media Lab
Full Presentation at httpwwwslidesharenetcameracultureraskar-ideahexagonapr2010
Ramesh Raskar MIT Media Lab
Xd
X++
X X+Y
X
X
neXt
Ramesh Raskar httpraskarinfo
Full Presentation at httpwwwslidesharenetcameracultureraskar-ideahexagonapr2010
Ramesh Raskar MIT Media Lab
Simple Exercise bull Image Compression
ndash Save Bandwidth and storage
What is neXt
Ramesh Raskar MIT Media Lab
Strategy 1 Xd
bull Extend it to next (or some other) dimension
Ramesh Raskar MIT Media Lab
X = bull Idea you just heardbull Conceptbull Patentbull New ProductBest projectinvention awardbull Product featurebull Designbull Artbull Algorithm
Ramesh Raskar MIT Media Lab
Xd
X++
X X+Y
X
X
neXt
Ramesh Raskar httpraskarinfo
Full Presentation at httpwwwslidesharenetcameracultureraskar-ideahexagonapr2010
Research bull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Is project worthwhile Heilmeiers Questions
bull Whatndash What are you trying to do Articulate your objectives using absolutely no jargon
bull Related workndash How is it done today and what are the limits of current practice
bull Contributionndash Whats new in your approach and why do you think it will be successful
bull Motivationndash Who caresndash If youre successful what difference will it make
bull Challengesndash What are the risks and the payoffsndash How much will it costndash How long will it take
bull Evaluationndash What are the midterm and final exams to check for success
bull Raskar additions ndash Why now (why not before whatrsquos new that makes possible)ndash Why us (wrong answers I am smart I can work harder than others)
httpenwikipediaorgwikiGeorge_H_HeilmeierHeilmeier27s_Catechism
Great Research Strive for Five
1 Before Five teamsBe first often let others do details
2 Beyond Five yearsWhat no one is thinking about
3 Within Five layers of lsquoHumanrsquo ImpactRelevance
4 Beyond Five minutes of descriptionDeep iterative participatory
5 Fusing Five+ ExpertiseMulti-disciplinary proactive
Ramesh Raskar httpraskarinfo
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Fernald Science [Sept 2006]
Shadow Refractive
Reflective
Tools for
Visual Computin
g
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Traditional Photography
Lens
Detector
Pixels
Image
Mimics Human Eye for a Single Snapshot
Single View Single Instant Fixed Dynamic range and Depth of field for given Illumination in a Static world Courtesy Shree
Nayar
Computational Camera + Photography
Optics Sensors and ComputationsGeneralized
Sensor
Generalized OpticsComputations
Picture
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Merged Views Programmable focus and dynamic range Closed-loop Controlled Illumination Coded exposureapertures
Computational Photography
Novel Illumination
Computational Cameras
Scene 8D Ray Modulator
Display
GeneralizedSensor
Generalized OpticsProcessing
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Generalized Optics
Recreate 4D Lightfield
Light Sources
Modulators
4D Incident Lighting
4D Light Field
Computational Photography [Raskar and Tumblin]
Resources
ICCP 2012 Seattle Apr 2012Papers due Dec 2nd 2011
httpwikipediaorgcomputational_photography
httpraskarinfophoto
captures a machine-readable representation of our world tohyper-realistically synthesize the essence of our visual experience
SynthesisAnalysis
Low Level Mid Level
HighLevel
Hyper realism
Raw
Angle spectrum
aware
Non-visual Data GPS
Metadata
Priors
Cap
ture
Comprehensive
8D reflectance field
Digital
Epsilon
Coded
Essence
Computational Photography aims to make progress on
both axis
Camera ArrayHDR FoV Focal stack
Decomposition problems
Depth
Spectrum
LightFields
Human Stereo Vision
Looking Around Corners
Virtual Object Insertion
Relighting
Augmented Human
Experience
Material editing from single photo
Scene completion from photos
Motion Magnification
Phototourism
Resolution
fgbg
DirectGlobal
Computational Photography
Bit Hacking
Phot
on H
acki
ng
Computer Vision
Optics
Sensors
Signal Processing
Displays
Machine Learning
Computational Light Transport
Computational PhotographyIllumination
Co-designing Optical and Digital Processing
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Wish List Open Research
Problems
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
Oacute 2007 Marc Levoy
Digital Refocusing using Light Field Camera
125μ square-sided microlenses[Ng et al 2005]
Motion Blur in Low Light
Traditional
Deblurred Image
Blurred Photo
Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
Flutter Shutter Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred PhotoPSF == Broadband Function
Fourier Transform
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Raskar Camera Culture MIT Media Lab
Ramesh Raskar MIT Media Lab
After X what is neXt
How to Invent
Ramesh Raskar MIT Media Lab
Full Presentation at httpwwwslidesharenetcameracultureraskar-ideahexagonapr2010
Ramesh Raskar MIT Media Lab
Xd
X++
X X+Y
X
X
neXt
Ramesh Raskar httpraskarinfo
Full Presentation at httpwwwslidesharenetcameracultureraskar-ideahexagonapr2010
Ramesh Raskar MIT Media Lab
Simple Exercise bull Image Compression
ndash Save Bandwidth and storage
What is neXt
Ramesh Raskar MIT Media Lab
Strategy 1 Xd
bull Extend it to next (or some other) dimension
Ramesh Raskar MIT Media Lab
X = bull Idea you just heardbull Conceptbull Patentbull New ProductBest projectinvention awardbull Product featurebull Designbull Artbull Algorithm
Ramesh Raskar MIT Media Lab
Xd
X++
X X+Y
X
X
neXt
Ramesh Raskar httpraskarinfo
Full Presentation at httpwwwslidesharenetcameracultureraskar-ideahexagonapr2010
Research bull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Is project worthwhile Heilmeiers Questions
bull Whatndash What are you trying to do Articulate your objectives using absolutely no jargon
bull Related workndash How is it done today and what are the limits of current practice
bull Contributionndash Whats new in your approach and why do you think it will be successful
bull Motivationndash Who caresndash If youre successful what difference will it make
bull Challengesndash What are the risks and the payoffsndash How much will it costndash How long will it take
bull Evaluationndash What are the midterm and final exams to check for success
bull Raskar additions ndash Why now (why not before whatrsquos new that makes possible)ndash Why us (wrong answers I am smart I can work harder than others)
httpenwikipediaorgwikiGeorge_H_HeilmeierHeilmeier27s_Catechism
Great Research Strive for Five
1 Before Five teamsBe first often let others do details
2 Beyond Five yearsWhat no one is thinking about
3 Within Five layers of lsquoHumanrsquo ImpactRelevance
4 Beyond Five minutes of descriptionDeep iterative participatory
5 Fusing Five+ ExpertiseMulti-disciplinary proactive
Ramesh Raskar httpraskarinfo
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Fernald Science [Sept 2006]
Shadow Refractive
Reflective
Tools for
Visual Computin
g
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Traditional Photography
Lens
Detector
Pixels
Image
Mimics Human Eye for a Single Snapshot
Single View Single Instant Fixed Dynamic range and Depth of field for given Illumination in a Static world Courtesy Shree
Nayar
Computational Camera + Photography
Optics Sensors and ComputationsGeneralized
Sensor
Generalized OpticsComputations
Picture
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Merged Views Programmable focus and dynamic range Closed-loop Controlled Illumination Coded exposureapertures
Computational Photography
Novel Illumination
Computational Cameras
Scene 8D Ray Modulator
Display
GeneralizedSensor
Generalized OpticsProcessing
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Generalized Optics
Recreate 4D Lightfield
Light Sources
Modulators
4D Incident Lighting
4D Light Field
Computational Photography [Raskar and Tumblin]
Resources
ICCP 2012 Seattle Apr 2012Papers due Dec 2nd 2011
httpwikipediaorgcomputational_photography
httpraskarinfophoto
captures a machine-readable representation of our world tohyper-realistically synthesize the essence of our visual experience
SynthesisAnalysis
Low Level Mid Level
HighLevel
Hyper realism
Raw
Angle spectrum
aware
Non-visual Data GPS
Metadata
Priors
Cap
ture
Comprehensive
8D reflectance field
Digital
Epsilon
Coded
Essence
Computational Photography aims to make progress on
both axis
Camera ArrayHDR FoV Focal stack
Decomposition problems
Depth
Spectrum
LightFields
Human Stereo Vision
Looking Around Corners
Virtual Object Insertion
Relighting
Augmented Human
Experience
Material editing from single photo
Scene completion from photos
Motion Magnification
Phototourism
Resolution
fgbg
DirectGlobal
Computational Photography
Bit Hacking
Phot
on H
acki
ng
Computer Vision
Optics
Sensors
Signal Processing
Displays
Machine Learning
Computational Light Transport
Computational PhotographyIllumination
Co-designing Optical and Digital Processing
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Wish List Open Research
Problems
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
Oacute 2007 Marc Levoy
Digital Refocusing using Light Field Camera
125μ square-sided microlenses[Ng et al 2005]
Motion Blur in Low Light
Traditional
Deblurred Image
Blurred Photo
Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
Flutter Shutter Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred PhotoPSF == Broadband Function
Fourier Transform
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Ramesh Raskar MIT Media Lab
After X what is neXt
How to Invent
Ramesh Raskar MIT Media Lab
Full Presentation at httpwwwslidesharenetcameracultureraskar-ideahexagonapr2010
Ramesh Raskar MIT Media Lab
Xd
X++
X X+Y
X
X
neXt
Ramesh Raskar httpraskarinfo
Full Presentation at httpwwwslidesharenetcameracultureraskar-ideahexagonapr2010
Ramesh Raskar MIT Media Lab
Simple Exercise bull Image Compression
ndash Save Bandwidth and storage
What is neXt
Ramesh Raskar MIT Media Lab
Strategy 1 Xd
bull Extend it to next (or some other) dimension
Ramesh Raskar MIT Media Lab
X = bull Idea you just heardbull Conceptbull Patentbull New ProductBest projectinvention awardbull Product featurebull Designbull Artbull Algorithm
Ramesh Raskar MIT Media Lab
Xd
X++
X X+Y
X
X
neXt
Ramesh Raskar httpraskarinfo
Full Presentation at httpwwwslidesharenetcameracultureraskar-ideahexagonapr2010
Research bull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Is project worthwhile Heilmeiers Questions
bull Whatndash What are you trying to do Articulate your objectives using absolutely no jargon
bull Related workndash How is it done today and what are the limits of current practice
bull Contributionndash Whats new in your approach and why do you think it will be successful
bull Motivationndash Who caresndash If youre successful what difference will it make
bull Challengesndash What are the risks and the payoffsndash How much will it costndash How long will it take
bull Evaluationndash What are the midterm and final exams to check for success
bull Raskar additions ndash Why now (why not before whatrsquos new that makes possible)ndash Why us (wrong answers I am smart I can work harder than others)
httpenwikipediaorgwikiGeorge_H_HeilmeierHeilmeier27s_Catechism
Great Research Strive for Five
1 Before Five teamsBe first often let others do details
2 Beyond Five yearsWhat no one is thinking about
3 Within Five layers of lsquoHumanrsquo ImpactRelevance
4 Beyond Five minutes of descriptionDeep iterative participatory
5 Fusing Five+ ExpertiseMulti-disciplinary proactive
Ramesh Raskar httpraskarinfo
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Fernald Science [Sept 2006]
Shadow Refractive
Reflective
Tools for
Visual Computin
g
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Traditional Photography
Lens
Detector
Pixels
Image
Mimics Human Eye for a Single Snapshot
Single View Single Instant Fixed Dynamic range and Depth of field for given Illumination in a Static world Courtesy Shree
Nayar
Computational Camera + Photography
Optics Sensors and ComputationsGeneralized
Sensor
Generalized OpticsComputations
Picture
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Merged Views Programmable focus and dynamic range Closed-loop Controlled Illumination Coded exposureapertures
Computational Photography
Novel Illumination
Computational Cameras
Scene 8D Ray Modulator
Display
GeneralizedSensor
Generalized OpticsProcessing
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Generalized Optics
Recreate 4D Lightfield
Light Sources
Modulators
4D Incident Lighting
4D Light Field
Computational Photography [Raskar and Tumblin]
Resources
ICCP 2012 Seattle Apr 2012Papers due Dec 2nd 2011
httpwikipediaorgcomputational_photography
httpraskarinfophoto
captures a machine-readable representation of our world tohyper-realistically synthesize the essence of our visual experience
SynthesisAnalysis
Low Level Mid Level
HighLevel
Hyper realism
Raw
Angle spectrum
aware
Non-visual Data GPS
Metadata
Priors
Cap
ture
Comprehensive
8D reflectance field
Digital
Epsilon
Coded
Essence
Computational Photography aims to make progress on
both axis
Camera ArrayHDR FoV Focal stack
Decomposition problems
Depth
Spectrum
LightFields
Human Stereo Vision
Looking Around Corners
Virtual Object Insertion
Relighting
Augmented Human
Experience
Material editing from single photo
Scene completion from photos
Motion Magnification
Phototourism
Resolution
fgbg
DirectGlobal
Computational Photography
Bit Hacking
Phot
on H
acki
ng
Computer Vision
Optics
Sensors
Signal Processing
Displays
Machine Learning
Computational Light Transport
Computational PhotographyIllumination
Co-designing Optical and Digital Processing
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Wish List Open Research
Problems
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
Oacute 2007 Marc Levoy
Digital Refocusing using Light Field Camera
125μ square-sided microlenses[Ng et al 2005]
Motion Blur in Low Light
Traditional
Deblurred Image
Blurred Photo
Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
Flutter Shutter Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred PhotoPSF == Broadband Function
Fourier Transform
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Ramesh Raskar MIT Media Lab
Xd
X++
X X+Y
X
X
neXt
Ramesh Raskar httpraskarinfo
Full Presentation at httpwwwslidesharenetcameracultureraskar-ideahexagonapr2010
Ramesh Raskar MIT Media Lab
Simple Exercise bull Image Compression
ndash Save Bandwidth and storage
What is neXt
Ramesh Raskar MIT Media Lab
Strategy 1 Xd
bull Extend it to next (or some other) dimension
Ramesh Raskar MIT Media Lab
X = bull Idea you just heardbull Conceptbull Patentbull New ProductBest projectinvention awardbull Product featurebull Designbull Artbull Algorithm
Ramesh Raskar MIT Media Lab
Xd
X++
X X+Y
X
X
neXt
Ramesh Raskar httpraskarinfo
Full Presentation at httpwwwslidesharenetcameracultureraskar-ideahexagonapr2010
Research bull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Is project worthwhile Heilmeiers Questions
bull Whatndash What are you trying to do Articulate your objectives using absolutely no jargon
bull Related workndash How is it done today and what are the limits of current practice
bull Contributionndash Whats new in your approach and why do you think it will be successful
bull Motivationndash Who caresndash If youre successful what difference will it make
bull Challengesndash What are the risks and the payoffsndash How much will it costndash How long will it take
bull Evaluationndash What are the midterm and final exams to check for success
bull Raskar additions ndash Why now (why not before whatrsquos new that makes possible)ndash Why us (wrong answers I am smart I can work harder than others)
httpenwikipediaorgwikiGeorge_H_HeilmeierHeilmeier27s_Catechism
Great Research Strive for Five
1 Before Five teamsBe first often let others do details
2 Beyond Five yearsWhat no one is thinking about
3 Within Five layers of lsquoHumanrsquo ImpactRelevance
4 Beyond Five minutes of descriptionDeep iterative participatory
5 Fusing Five+ ExpertiseMulti-disciplinary proactive
Ramesh Raskar httpraskarinfo
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Fernald Science [Sept 2006]
Shadow Refractive
Reflective
Tools for
Visual Computin
g
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Traditional Photography
Lens
Detector
Pixels
Image
Mimics Human Eye for a Single Snapshot
Single View Single Instant Fixed Dynamic range and Depth of field for given Illumination in a Static world Courtesy Shree
Nayar
Computational Camera + Photography
Optics Sensors and ComputationsGeneralized
Sensor
Generalized OpticsComputations
Picture
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Merged Views Programmable focus and dynamic range Closed-loop Controlled Illumination Coded exposureapertures
Computational Photography
Novel Illumination
Computational Cameras
Scene 8D Ray Modulator
Display
GeneralizedSensor
Generalized OpticsProcessing
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Generalized Optics
Recreate 4D Lightfield
Light Sources
Modulators
4D Incident Lighting
4D Light Field
Computational Photography [Raskar and Tumblin]
Resources
ICCP 2012 Seattle Apr 2012Papers due Dec 2nd 2011
httpwikipediaorgcomputational_photography
httpraskarinfophoto
captures a machine-readable representation of our world tohyper-realistically synthesize the essence of our visual experience
SynthesisAnalysis
Low Level Mid Level
HighLevel
Hyper realism
Raw
Angle spectrum
aware
Non-visual Data GPS
Metadata
Priors
Cap
ture
Comprehensive
8D reflectance field
Digital
Epsilon
Coded
Essence
Computational Photography aims to make progress on
both axis
Camera ArrayHDR FoV Focal stack
Decomposition problems
Depth
Spectrum
LightFields
Human Stereo Vision
Looking Around Corners
Virtual Object Insertion
Relighting
Augmented Human
Experience
Material editing from single photo
Scene completion from photos
Motion Magnification
Phototourism
Resolution
fgbg
DirectGlobal
Computational Photography
Bit Hacking
Phot
on H
acki
ng
Computer Vision
Optics
Sensors
Signal Processing
Displays
Machine Learning
Computational Light Transport
Computational PhotographyIllumination
Co-designing Optical and Digital Processing
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Wish List Open Research
Problems
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
Oacute 2007 Marc Levoy
Digital Refocusing using Light Field Camera
125μ square-sided microlenses[Ng et al 2005]
Motion Blur in Low Light
Traditional
Deblurred Image
Blurred Photo
Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
Flutter Shutter Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred PhotoPSF == Broadband Function
Fourier Transform
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Ramesh Raskar MIT Media Lab
Simple Exercise bull Image Compression
ndash Save Bandwidth and storage
What is neXt
Ramesh Raskar MIT Media Lab
Strategy 1 Xd
bull Extend it to next (or some other) dimension
Ramesh Raskar MIT Media Lab
X = bull Idea you just heardbull Conceptbull Patentbull New ProductBest projectinvention awardbull Product featurebull Designbull Artbull Algorithm
Ramesh Raskar MIT Media Lab
Xd
X++
X X+Y
X
X
neXt
Ramesh Raskar httpraskarinfo
Full Presentation at httpwwwslidesharenetcameracultureraskar-ideahexagonapr2010
Research bull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Is project worthwhile Heilmeiers Questions
bull Whatndash What are you trying to do Articulate your objectives using absolutely no jargon
bull Related workndash How is it done today and what are the limits of current practice
bull Contributionndash Whats new in your approach and why do you think it will be successful
bull Motivationndash Who caresndash If youre successful what difference will it make
bull Challengesndash What are the risks and the payoffsndash How much will it costndash How long will it take
bull Evaluationndash What are the midterm and final exams to check for success
bull Raskar additions ndash Why now (why not before whatrsquos new that makes possible)ndash Why us (wrong answers I am smart I can work harder than others)
httpenwikipediaorgwikiGeorge_H_HeilmeierHeilmeier27s_Catechism
Great Research Strive for Five
1 Before Five teamsBe first often let others do details
2 Beyond Five yearsWhat no one is thinking about
3 Within Five layers of lsquoHumanrsquo ImpactRelevance
4 Beyond Five minutes of descriptionDeep iterative participatory
5 Fusing Five+ ExpertiseMulti-disciplinary proactive
Ramesh Raskar httpraskarinfo
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Fernald Science [Sept 2006]
Shadow Refractive
Reflective
Tools for
Visual Computin
g
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Traditional Photography
Lens
Detector
Pixels
Image
Mimics Human Eye for a Single Snapshot
Single View Single Instant Fixed Dynamic range and Depth of field for given Illumination in a Static world Courtesy Shree
Nayar
Computational Camera + Photography
Optics Sensors and ComputationsGeneralized
Sensor
Generalized OpticsComputations
Picture
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Merged Views Programmable focus and dynamic range Closed-loop Controlled Illumination Coded exposureapertures
Computational Photography
Novel Illumination
Computational Cameras
Scene 8D Ray Modulator
Display
GeneralizedSensor
Generalized OpticsProcessing
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Generalized Optics
Recreate 4D Lightfield
Light Sources
Modulators
4D Incident Lighting
4D Light Field
Computational Photography [Raskar and Tumblin]
Resources
ICCP 2012 Seattle Apr 2012Papers due Dec 2nd 2011
httpwikipediaorgcomputational_photography
httpraskarinfophoto
captures a machine-readable representation of our world tohyper-realistically synthesize the essence of our visual experience
SynthesisAnalysis
Low Level Mid Level
HighLevel
Hyper realism
Raw
Angle spectrum
aware
Non-visual Data GPS
Metadata
Priors
Cap
ture
Comprehensive
8D reflectance field
Digital
Epsilon
Coded
Essence
Computational Photography aims to make progress on
both axis
Camera ArrayHDR FoV Focal stack
Decomposition problems
Depth
Spectrum
LightFields
Human Stereo Vision
Looking Around Corners
Virtual Object Insertion
Relighting
Augmented Human
Experience
Material editing from single photo
Scene completion from photos
Motion Magnification
Phototourism
Resolution
fgbg
DirectGlobal
Computational Photography
Bit Hacking
Phot
on H
acki
ng
Computer Vision
Optics
Sensors
Signal Processing
Displays
Machine Learning
Computational Light Transport
Computational PhotographyIllumination
Co-designing Optical and Digital Processing
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Wish List Open Research
Problems
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
Oacute 2007 Marc Levoy
Digital Refocusing using Light Field Camera
125μ square-sided microlenses[Ng et al 2005]
Motion Blur in Low Light
Traditional
Deblurred Image
Blurred Photo
Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
Flutter Shutter Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred PhotoPSF == Broadband Function
Fourier Transform
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Ramesh Raskar MIT Media Lab
Strategy 1 Xd
bull Extend it to next (or some other) dimension
Ramesh Raskar MIT Media Lab
X = bull Idea you just heardbull Conceptbull Patentbull New ProductBest projectinvention awardbull Product featurebull Designbull Artbull Algorithm
Ramesh Raskar MIT Media Lab
Xd
X++
X X+Y
X
X
neXt
Ramesh Raskar httpraskarinfo
Full Presentation at httpwwwslidesharenetcameracultureraskar-ideahexagonapr2010
Research bull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Is project worthwhile Heilmeiers Questions
bull Whatndash What are you trying to do Articulate your objectives using absolutely no jargon
bull Related workndash How is it done today and what are the limits of current practice
bull Contributionndash Whats new in your approach and why do you think it will be successful
bull Motivationndash Who caresndash If youre successful what difference will it make
bull Challengesndash What are the risks and the payoffsndash How much will it costndash How long will it take
bull Evaluationndash What are the midterm and final exams to check for success
bull Raskar additions ndash Why now (why not before whatrsquos new that makes possible)ndash Why us (wrong answers I am smart I can work harder than others)
httpenwikipediaorgwikiGeorge_H_HeilmeierHeilmeier27s_Catechism
Great Research Strive for Five
1 Before Five teamsBe first often let others do details
2 Beyond Five yearsWhat no one is thinking about
3 Within Five layers of lsquoHumanrsquo ImpactRelevance
4 Beyond Five minutes of descriptionDeep iterative participatory
5 Fusing Five+ ExpertiseMulti-disciplinary proactive
Ramesh Raskar httpraskarinfo
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Fernald Science [Sept 2006]
Shadow Refractive
Reflective
Tools for
Visual Computin
g
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Traditional Photography
Lens
Detector
Pixels
Image
Mimics Human Eye for a Single Snapshot
Single View Single Instant Fixed Dynamic range and Depth of field for given Illumination in a Static world Courtesy Shree
Nayar
Computational Camera + Photography
Optics Sensors and ComputationsGeneralized
Sensor
Generalized OpticsComputations
Picture
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Merged Views Programmable focus and dynamic range Closed-loop Controlled Illumination Coded exposureapertures
Computational Photography
Novel Illumination
Computational Cameras
Scene 8D Ray Modulator
Display
GeneralizedSensor
Generalized OpticsProcessing
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Generalized Optics
Recreate 4D Lightfield
Light Sources
Modulators
4D Incident Lighting
4D Light Field
Computational Photography [Raskar and Tumblin]
Resources
ICCP 2012 Seattle Apr 2012Papers due Dec 2nd 2011
httpwikipediaorgcomputational_photography
httpraskarinfophoto
captures a machine-readable representation of our world tohyper-realistically synthesize the essence of our visual experience
SynthesisAnalysis
Low Level Mid Level
HighLevel
Hyper realism
Raw
Angle spectrum
aware
Non-visual Data GPS
Metadata
Priors
Cap
ture
Comprehensive
8D reflectance field
Digital
Epsilon
Coded
Essence
Computational Photography aims to make progress on
both axis
Camera ArrayHDR FoV Focal stack
Decomposition problems
Depth
Spectrum
LightFields
Human Stereo Vision
Looking Around Corners
Virtual Object Insertion
Relighting
Augmented Human
Experience
Material editing from single photo
Scene completion from photos
Motion Magnification
Phototourism
Resolution
fgbg
DirectGlobal
Computational Photography
Bit Hacking
Phot
on H
acki
ng
Computer Vision
Optics
Sensors
Signal Processing
Displays
Machine Learning
Computational Light Transport
Computational PhotographyIllumination
Co-designing Optical and Digital Processing
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Wish List Open Research
Problems
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
Oacute 2007 Marc Levoy
Digital Refocusing using Light Field Camera
125μ square-sided microlenses[Ng et al 2005]
Motion Blur in Low Light
Traditional
Deblurred Image
Blurred Photo
Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
Flutter Shutter Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred PhotoPSF == Broadband Function
Fourier Transform
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Ramesh Raskar MIT Media Lab
X = bull Idea you just heardbull Conceptbull Patentbull New ProductBest projectinvention awardbull Product featurebull Designbull Artbull Algorithm
Ramesh Raskar MIT Media Lab
Xd
X++
X X+Y
X
X
neXt
Ramesh Raskar httpraskarinfo
Full Presentation at httpwwwslidesharenetcameracultureraskar-ideahexagonapr2010
Research bull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Is project worthwhile Heilmeiers Questions
bull Whatndash What are you trying to do Articulate your objectives using absolutely no jargon
bull Related workndash How is it done today and what are the limits of current practice
bull Contributionndash Whats new in your approach and why do you think it will be successful
bull Motivationndash Who caresndash If youre successful what difference will it make
bull Challengesndash What are the risks and the payoffsndash How much will it costndash How long will it take
bull Evaluationndash What are the midterm and final exams to check for success
bull Raskar additions ndash Why now (why not before whatrsquos new that makes possible)ndash Why us (wrong answers I am smart I can work harder than others)
httpenwikipediaorgwikiGeorge_H_HeilmeierHeilmeier27s_Catechism
Great Research Strive for Five
1 Before Five teamsBe first often let others do details
2 Beyond Five yearsWhat no one is thinking about
3 Within Five layers of lsquoHumanrsquo ImpactRelevance
4 Beyond Five minutes of descriptionDeep iterative participatory
5 Fusing Five+ ExpertiseMulti-disciplinary proactive
Ramesh Raskar httpraskarinfo
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Fernald Science [Sept 2006]
Shadow Refractive
Reflective
Tools for
Visual Computin
g
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Traditional Photography
Lens
Detector
Pixels
Image
Mimics Human Eye for a Single Snapshot
Single View Single Instant Fixed Dynamic range and Depth of field for given Illumination in a Static world Courtesy Shree
Nayar
Computational Camera + Photography
Optics Sensors and ComputationsGeneralized
Sensor
Generalized OpticsComputations
Picture
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Merged Views Programmable focus and dynamic range Closed-loop Controlled Illumination Coded exposureapertures
Computational Photography
Novel Illumination
Computational Cameras
Scene 8D Ray Modulator
Display
GeneralizedSensor
Generalized OpticsProcessing
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Generalized Optics
Recreate 4D Lightfield
Light Sources
Modulators
4D Incident Lighting
4D Light Field
Computational Photography [Raskar and Tumblin]
Resources
ICCP 2012 Seattle Apr 2012Papers due Dec 2nd 2011
httpwikipediaorgcomputational_photography
httpraskarinfophoto
captures a machine-readable representation of our world tohyper-realistically synthesize the essence of our visual experience
SynthesisAnalysis
Low Level Mid Level
HighLevel
Hyper realism
Raw
Angle spectrum
aware
Non-visual Data GPS
Metadata
Priors
Cap
ture
Comprehensive
8D reflectance field
Digital
Epsilon
Coded
Essence
Computational Photography aims to make progress on
both axis
Camera ArrayHDR FoV Focal stack
Decomposition problems
Depth
Spectrum
LightFields
Human Stereo Vision
Looking Around Corners
Virtual Object Insertion
Relighting
Augmented Human
Experience
Material editing from single photo
Scene completion from photos
Motion Magnification
Phototourism
Resolution
fgbg
DirectGlobal
Computational Photography
Bit Hacking
Phot
on H
acki
ng
Computer Vision
Optics
Sensors
Signal Processing
Displays
Machine Learning
Computational Light Transport
Computational PhotographyIllumination
Co-designing Optical and Digital Processing
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Wish List Open Research
Problems
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
Oacute 2007 Marc Levoy
Digital Refocusing using Light Field Camera
125μ square-sided microlenses[Ng et al 2005]
Motion Blur in Low Light
Traditional
Deblurred Image
Blurred Photo
Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
Flutter Shutter Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred PhotoPSF == Broadband Function
Fourier Transform
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Ramesh Raskar MIT Media Lab
Xd
X++
X X+Y
X
X
neXt
Ramesh Raskar httpraskarinfo
Full Presentation at httpwwwslidesharenetcameracultureraskar-ideahexagonapr2010
Research bull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Is project worthwhile Heilmeiers Questions
bull Whatndash What are you trying to do Articulate your objectives using absolutely no jargon
bull Related workndash How is it done today and what are the limits of current practice
bull Contributionndash Whats new in your approach and why do you think it will be successful
bull Motivationndash Who caresndash If youre successful what difference will it make
bull Challengesndash What are the risks and the payoffsndash How much will it costndash How long will it take
bull Evaluationndash What are the midterm and final exams to check for success
bull Raskar additions ndash Why now (why not before whatrsquos new that makes possible)ndash Why us (wrong answers I am smart I can work harder than others)
httpenwikipediaorgwikiGeorge_H_HeilmeierHeilmeier27s_Catechism
Great Research Strive for Five
1 Before Five teamsBe first often let others do details
2 Beyond Five yearsWhat no one is thinking about
3 Within Five layers of lsquoHumanrsquo ImpactRelevance
4 Beyond Five minutes of descriptionDeep iterative participatory
5 Fusing Five+ ExpertiseMulti-disciplinary proactive
Ramesh Raskar httpraskarinfo
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Fernald Science [Sept 2006]
Shadow Refractive
Reflective
Tools for
Visual Computin
g
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Traditional Photography
Lens
Detector
Pixels
Image
Mimics Human Eye for a Single Snapshot
Single View Single Instant Fixed Dynamic range and Depth of field for given Illumination in a Static world Courtesy Shree
Nayar
Computational Camera + Photography
Optics Sensors and ComputationsGeneralized
Sensor
Generalized OpticsComputations
Picture
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Merged Views Programmable focus and dynamic range Closed-loop Controlled Illumination Coded exposureapertures
Computational Photography
Novel Illumination
Computational Cameras
Scene 8D Ray Modulator
Display
GeneralizedSensor
Generalized OpticsProcessing
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Generalized Optics
Recreate 4D Lightfield
Light Sources
Modulators
4D Incident Lighting
4D Light Field
Computational Photography [Raskar and Tumblin]
Resources
ICCP 2012 Seattle Apr 2012Papers due Dec 2nd 2011
httpwikipediaorgcomputational_photography
httpraskarinfophoto
captures a machine-readable representation of our world tohyper-realistically synthesize the essence of our visual experience
SynthesisAnalysis
Low Level Mid Level
HighLevel
Hyper realism
Raw
Angle spectrum
aware
Non-visual Data GPS
Metadata
Priors
Cap
ture
Comprehensive
8D reflectance field
Digital
Epsilon
Coded
Essence
Computational Photography aims to make progress on
both axis
Camera ArrayHDR FoV Focal stack
Decomposition problems
Depth
Spectrum
LightFields
Human Stereo Vision
Looking Around Corners
Virtual Object Insertion
Relighting
Augmented Human
Experience
Material editing from single photo
Scene completion from photos
Motion Magnification
Phototourism
Resolution
fgbg
DirectGlobal
Computational Photography
Bit Hacking
Phot
on H
acki
ng
Computer Vision
Optics
Sensors
Signal Processing
Displays
Machine Learning
Computational Light Transport
Computational PhotographyIllumination
Co-designing Optical and Digital Processing
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Wish List Open Research
Problems
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
Oacute 2007 Marc Levoy
Digital Refocusing using Light Field Camera
125μ square-sided microlenses[Ng et al 2005]
Motion Blur in Low Light
Traditional
Deblurred Image
Blurred Photo
Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
Flutter Shutter Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred PhotoPSF == Broadband Function
Fourier Transform
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Research bull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Is project worthwhile Heilmeiers Questions
bull Whatndash What are you trying to do Articulate your objectives using absolutely no jargon
bull Related workndash How is it done today and what are the limits of current practice
bull Contributionndash Whats new in your approach and why do you think it will be successful
bull Motivationndash Who caresndash If youre successful what difference will it make
bull Challengesndash What are the risks and the payoffsndash How much will it costndash How long will it take
bull Evaluationndash What are the midterm and final exams to check for success
bull Raskar additions ndash Why now (why not before whatrsquos new that makes possible)ndash Why us (wrong answers I am smart I can work harder than others)
httpenwikipediaorgwikiGeorge_H_HeilmeierHeilmeier27s_Catechism
Great Research Strive for Five
1 Before Five teamsBe first often let others do details
2 Beyond Five yearsWhat no one is thinking about
3 Within Five layers of lsquoHumanrsquo ImpactRelevance
4 Beyond Five minutes of descriptionDeep iterative participatory
5 Fusing Five+ ExpertiseMulti-disciplinary proactive
Ramesh Raskar httpraskarinfo
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Fernald Science [Sept 2006]
Shadow Refractive
Reflective
Tools for
Visual Computin
g
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Traditional Photography
Lens
Detector
Pixels
Image
Mimics Human Eye for a Single Snapshot
Single View Single Instant Fixed Dynamic range and Depth of field for given Illumination in a Static world Courtesy Shree
Nayar
Computational Camera + Photography
Optics Sensors and ComputationsGeneralized
Sensor
Generalized OpticsComputations
Picture
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Merged Views Programmable focus and dynamic range Closed-loop Controlled Illumination Coded exposureapertures
Computational Photography
Novel Illumination
Computational Cameras
Scene 8D Ray Modulator
Display
GeneralizedSensor
Generalized OpticsProcessing
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Generalized Optics
Recreate 4D Lightfield
Light Sources
Modulators
4D Incident Lighting
4D Light Field
Computational Photography [Raskar and Tumblin]
Resources
ICCP 2012 Seattle Apr 2012Papers due Dec 2nd 2011
httpwikipediaorgcomputational_photography
httpraskarinfophoto
captures a machine-readable representation of our world tohyper-realistically synthesize the essence of our visual experience
SynthesisAnalysis
Low Level Mid Level
HighLevel
Hyper realism
Raw
Angle spectrum
aware
Non-visual Data GPS
Metadata
Priors
Cap
ture
Comprehensive
8D reflectance field
Digital
Epsilon
Coded
Essence
Computational Photography aims to make progress on
both axis
Camera ArrayHDR FoV Focal stack
Decomposition problems
Depth
Spectrum
LightFields
Human Stereo Vision
Looking Around Corners
Virtual Object Insertion
Relighting
Augmented Human
Experience
Material editing from single photo
Scene completion from photos
Motion Magnification
Phototourism
Resolution
fgbg
DirectGlobal
Computational Photography
Bit Hacking
Phot
on H
acki
ng
Computer Vision
Optics
Sensors
Signal Processing
Displays
Machine Learning
Computational Light Transport
Computational PhotographyIllumination
Co-designing Optical and Digital Processing
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Wish List Open Research
Problems
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
Oacute 2007 Marc Levoy
Digital Refocusing using Light Field Camera
125μ square-sided microlenses[Ng et al 2005]
Motion Blur in Low Light
Traditional
Deblurred Image
Blurred Photo
Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
Flutter Shutter Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred PhotoPSF == Broadband Function
Fourier Transform
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Is project worthwhile Heilmeiers Questions
bull Whatndash What are you trying to do Articulate your objectives using absolutely no jargon
bull Related workndash How is it done today and what are the limits of current practice
bull Contributionndash Whats new in your approach and why do you think it will be successful
bull Motivationndash Who caresndash If youre successful what difference will it make
bull Challengesndash What are the risks and the payoffsndash How much will it costndash How long will it take
bull Evaluationndash What are the midterm and final exams to check for success
bull Raskar additions ndash Why now (why not before whatrsquos new that makes possible)ndash Why us (wrong answers I am smart I can work harder than others)
httpenwikipediaorgwikiGeorge_H_HeilmeierHeilmeier27s_Catechism
Great Research Strive for Five
1 Before Five teamsBe first often let others do details
2 Beyond Five yearsWhat no one is thinking about
3 Within Five layers of lsquoHumanrsquo ImpactRelevance
4 Beyond Five minutes of descriptionDeep iterative participatory
5 Fusing Five+ ExpertiseMulti-disciplinary proactive
Ramesh Raskar httpraskarinfo
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Fernald Science [Sept 2006]
Shadow Refractive
Reflective
Tools for
Visual Computin
g
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Traditional Photography
Lens
Detector
Pixels
Image
Mimics Human Eye for a Single Snapshot
Single View Single Instant Fixed Dynamic range and Depth of field for given Illumination in a Static world Courtesy Shree
Nayar
Computational Camera + Photography
Optics Sensors and ComputationsGeneralized
Sensor
Generalized OpticsComputations
Picture
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Merged Views Programmable focus and dynamic range Closed-loop Controlled Illumination Coded exposureapertures
Computational Photography
Novel Illumination
Computational Cameras
Scene 8D Ray Modulator
Display
GeneralizedSensor
Generalized OpticsProcessing
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Generalized Optics
Recreate 4D Lightfield
Light Sources
Modulators
4D Incident Lighting
4D Light Field
Computational Photography [Raskar and Tumblin]
Resources
ICCP 2012 Seattle Apr 2012Papers due Dec 2nd 2011
httpwikipediaorgcomputational_photography
httpraskarinfophoto
captures a machine-readable representation of our world tohyper-realistically synthesize the essence of our visual experience
SynthesisAnalysis
Low Level Mid Level
HighLevel
Hyper realism
Raw
Angle spectrum
aware
Non-visual Data GPS
Metadata
Priors
Cap
ture
Comprehensive
8D reflectance field
Digital
Epsilon
Coded
Essence
Computational Photography aims to make progress on
both axis
Camera ArrayHDR FoV Focal stack
Decomposition problems
Depth
Spectrum
LightFields
Human Stereo Vision
Looking Around Corners
Virtual Object Insertion
Relighting
Augmented Human
Experience
Material editing from single photo
Scene completion from photos
Motion Magnification
Phototourism
Resolution
fgbg
DirectGlobal
Computational Photography
Bit Hacking
Phot
on H
acki
ng
Computer Vision
Optics
Sensors
Signal Processing
Displays
Machine Learning
Computational Light Transport
Computational PhotographyIllumination
Co-designing Optical and Digital Processing
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Wish List Open Research
Problems
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
Oacute 2007 Marc Levoy
Digital Refocusing using Light Field Camera
125μ square-sided microlenses[Ng et al 2005]
Motion Blur in Low Light
Traditional
Deblurred Image
Blurred Photo
Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
Flutter Shutter Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred PhotoPSF == Broadband Function
Fourier Transform
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Great Research Strive for Five
1 Before Five teamsBe first often let others do details
2 Beyond Five yearsWhat no one is thinking about
3 Within Five layers of lsquoHumanrsquo ImpactRelevance
4 Beyond Five minutes of descriptionDeep iterative participatory
5 Fusing Five+ ExpertiseMulti-disciplinary proactive
Ramesh Raskar httpraskarinfo
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Fernald Science [Sept 2006]
Shadow Refractive
Reflective
Tools for
Visual Computin
g
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Traditional Photography
Lens
Detector
Pixels
Image
Mimics Human Eye for a Single Snapshot
Single View Single Instant Fixed Dynamic range and Depth of field for given Illumination in a Static world Courtesy Shree
Nayar
Computational Camera + Photography
Optics Sensors and ComputationsGeneralized
Sensor
Generalized OpticsComputations
Picture
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Merged Views Programmable focus and dynamic range Closed-loop Controlled Illumination Coded exposureapertures
Computational Photography
Novel Illumination
Computational Cameras
Scene 8D Ray Modulator
Display
GeneralizedSensor
Generalized OpticsProcessing
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Generalized Optics
Recreate 4D Lightfield
Light Sources
Modulators
4D Incident Lighting
4D Light Field
Computational Photography [Raskar and Tumblin]
Resources
ICCP 2012 Seattle Apr 2012Papers due Dec 2nd 2011
httpwikipediaorgcomputational_photography
httpraskarinfophoto
captures a machine-readable representation of our world tohyper-realistically synthesize the essence of our visual experience
SynthesisAnalysis
Low Level Mid Level
HighLevel
Hyper realism
Raw
Angle spectrum
aware
Non-visual Data GPS
Metadata
Priors
Cap
ture
Comprehensive
8D reflectance field
Digital
Epsilon
Coded
Essence
Computational Photography aims to make progress on
both axis
Camera ArrayHDR FoV Focal stack
Decomposition problems
Depth
Spectrum
LightFields
Human Stereo Vision
Looking Around Corners
Virtual Object Insertion
Relighting
Augmented Human
Experience
Material editing from single photo
Scene completion from photos
Motion Magnification
Phototourism
Resolution
fgbg
DirectGlobal
Computational Photography
Bit Hacking
Phot
on H
acki
ng
Computer Vision
Optics
Sensors
Signal Processing
Displays
Machine Learning
Computational Light Transport
Computational PhotographyIllumination
Co-designing Optical and Digital Processing
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Wish List Open Research
Problems
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
Oacute 2007 Marc Levoy
Digital Refocusing using Light Field Camera
125μ square-sided microlenses[Ng et al 2005]
Motion Blur in Low Light
Traditional
Deblurred Image
Blurred Photo
Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
Flutter Shutter Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred PhotoPSF == Broadband Function
Fourier Transform
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Fernald Science [Sept 2006]
Shadow Refractive
Reflective
Tools for
Visual Computin
g
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Traditional Photography
Lens
Detector
Pixels
Image
Mimics Human Eye for a Single Snapshot
Single View Single Instant Fixed Dynamic range and Depth of field for given Illumination in a Static world Courtesy Shree
Nayar
Computational Camera + Photography
Optics Sensors and ComputationsGeneralized
Sensor
Generalized OpticsComputations
Picture
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Merged Views Programmable focus and dynamic range Closed-loop Controlled Illumination Coded exposureapertures
Computational Photography
Novel Illumination
Computational Cameras
Scene 8D Ray Modulator
Display
GeneralizedSensor
Generalized OpticsProcessing
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Generalized Optics
Recreate 4D Lightfield
Light Sources
Modulators
4D Incident Lighting
4D Light Field
Computational Photography [Raskar and Tumblin]
Resources
ICCP 2012 Seattle Apr 2012Papers due Dec 2nd 2011
httpwikipediaorgcomputational_photography
httpraskarinfophoto
captures a machine-readable representation of our world tohyper-realistically synthesize the essence of our visual experience
SynthesisAnalysis
Low Level Mid Level
HighLevel
Hyper realism
Raw
Angle spectrum
aware
Non-visual Data GPS
Metadata
Priors
Cap
ture
Comprehensive
8D reflectance field
Digital
Epsilon
Coded
Essence
Computational Photography aims to make progress on
both axis
Camera ArrayHDR FoV Focal stack
Decomposition problems
Depth
Spectrum
LightFields
Human Stereo Vision
Looking Around Corners
Virtual Object Insertion
Relighting
Augmented Human
Experience
Material editing from single photo
Scene completion from photos
Motion Magnification
Phototourism
Resolution
fgbg
DirectGlobal
Computational Photography
Bit Hacking
Phot
on H
acki
ng
Computer Vision
Optics
Sensors
Signal Processing
Displays
Machine Learning
Computational Light Transport
Computational PhotographyIllumination
Co-designing Optical and Digital Processing
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Wish List Open Research
Problems
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
Oacute 2007 Marc Levoy
Digital Refocusing using Light Field Camera
125μ square-sided microlenses[Ng et al 2005]
Motion Blur in Low Light
Traditional
Deblurred Image
Blurred Photo
Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
Flutter Shutter Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred PhotoPSF == Broadband Function
Fourier Transform
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Fernald Science [Sept 2006]
Shadow Refractive
Reflective
Tools for
Visual Computin
g
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Traditional Photography
Lens
Detector
Pixels
Image
Mimics Human Eye for a Single Snapshot
Single View Single Instant Fixed Dynamic range and Depth of field for given Illumination in a Static world Courtesy Shree
Nayar
Computational Camera + Photography
Optics Sensors and ComputationsGeneralized
Sensor
Generalized OpticsComputations
Picture
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Merged Views Programmable focus and dynamic range Closed-loop Controlled Illumination Coded exposureapertures
Computational Photography
Novel Illumination
Computational Cameras
Scene 8D Ray Modulator
Display
GeneralizedSensor
Generalized OpticsProcessing
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Generalized Optics
Recreate 4D Lightfield
Light Sources
Modulators
4D Incident Lighting
4D Light Field
Computational Photography [Raskar and Tumblin]
Resources
ICCP 2012 Seattle Apr 2012Papers due Dec 2nd 2011
httpwikipediaorgcomputational_photography
httpraskarinfophoto
captures a machine-readable representation of our world tohyper-realistically synthesize the essence of our visual experience
SynthesisAnalysis
Low Level Mid Level
HighLevel
Hyper realism
Raw
Angle spectrum
aware
Non-visual Data GPS
Metadata
Priors
Cap
ture
Comprehensive
8D reflectance field
Digital
Epsilon
Coded
Essence
Computational Photography aims to make progress on
both axis
Camera ArrayHDR FoV Focal stack
Decomposition problems
Depth
Spectrum
LightFields
Human Stereo Vision
Looking Around Corners
Virtual Object Insertion
Relighting
Augmented Human
Experience
Material editing from single photo
Scene completion from photos
Motion Magnification
Phototourism
Resolution
fgbg
DirectGlobal
Computational Photography
Bit Hacking
Phot
on H
acki
ng
Computer Vision
Optics
Sensors
Signal Processing
Displays
Machine Learning
Computational Light Transport
Computational PhotographyIllumination
Co-designing Optical and Digital Processing
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Wish List Open Research
Problems
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
Oacute 2007 Marc Levoy
Digital Refocusing using Light Field Camera
125μ square-sided microlenses[Ng et al 2005]
Motion Blur in Low Light
Traditional
Deblurred Image
Blurred Photo
Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
Flutter Shutter Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred PhotoPSF == Broadband Function
Fourier Transform
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Traditional Photography
Lens
Detector
Pixels
Image
Mimics Human Eye for a Single Snapshot
Single View Single Instant Fixed Dynamic range and Depth of field for given Illumination in a Static world Courtesy Shree
Nayar
Computational Camera + Photography
Optics Sensors and ComputationsGeneralized
Sensor
Generalized OpticsComputations
Picture
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Merged Views Programmable focus and dynamic range Closed-loop Controlled Illumination Coded exposureapertures
Computational Photography
Novel Illumination
Computational Cameras
Scene 8D Ray Modulator
Display
GeneralizedSensor
Generalized OpticsProcessing
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Generalized Optics
Recreate 4D Lightfield
Light Sources
Modulators
4D Incident Lighting
4D Light Field
Computational Photography [Raskar and Tumblin]
Resources
ICCP 2012 Seattle Apr 2012Papers due Dec 2nd 2011
httpwikipediaorgcomputational_photography
httpraskarinfophoto
captures a machine-readable representation of our world tohyper-realistically synthesize the essence of our visual experience
SynthesisAnalysis
Low Level Mid Level
HighLevel
Hyper realism
Raw
Angle spectrum
aware
Non-visual Data GPS
Metadata
Priors
Cap
ture
Comprehensive
8D reflectance field
Digital
Epsilon
Coded
Essence
Computational Photography aims to make progress on
both axis
Camera ArrayHDR FoV Focal stack
Decomposition problems
Depth
Spectrum
LightFields
Human Stereo Vision
Looking Around Corners
Virtual Object Insertion
Relighting
Augmented Human
Experience
Material editing from single photo
Scene completion from photos
Motion Magnification
Phototourism
Resolution
fgbg
DirectGlobal
Computational Photography
Bit Hacking
Phot
on H
acki
ng
Computer Vision
Optics
Sensors
Signal Processing
Displays
Machine Learning
Computational Light Transport
Computational PhotographyIllumination
Co-designing Optical and Digital Processing
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Wish List Open Research
Problems
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
Oacute 2007 Marc Levoy
Digital Refocusing using Light Field Camera
125μ square-sided microlenses[Ng et al 2005]
Motion Blur in Low Light
Traditional
Deblurred Image
Blurred Photo
Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
Flutter Shutter Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred PhotoPSF == Broadband Function
Fourier Transform
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Traditional Photography
Lens
Detector
Pixels
Image
Mimics Human Eye for a Single Snapshot
Single View Single Instant Fixed Dynamic range and Depth of field for given Illumination in a Static world Courtesy Shree
Nayar
Computational Camera + Photography
Optics Sensors and ComputationsGeneralized
Sensor
Generalized OpticsComputations
Picture
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Merged Views Programmable focus and dynamic range Closed-loop Controlled Illumination Coded exposureapertures
Computational Photography
Novel Illumination
Computational Cameras
Scene 8D Ray Modulator
Display
GeneralizedSensor
Generalized OpticsProcessing
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Generalized Optics
Recreate 4D Lightfield
Light Sources
Modulators
4D Incident Lighting
4D Light Field
Computational Photography [Raskar and Tumblin]
Resources
ICCP 2012 Seattle Apr 2012Papers due Dec 2nd 2011
httpwikipediaorgcomputational_photography
httpraskarinfophoto
captures a machine-readable representation of our world tohyper-realistically synthesize the essence of our visual experience
SynthesisAnalysis
Low Level Mid Level
HighLevel
Hyper realism
Raw
Angle spectrum
aware
Non-visual Data GPS
Metadata
Priors
Cap
ture
Comprehensive
8D reflectance field
Digital
Epsilon
Coded
Essence
Computational Photography aims to make progress on
both axis
Camera ArrayHDR FoV Focal stack
Decomposition problems
Depth
Spectrum
LightFields
Human Stereo Vision
Looking Around Corners
Virtual Object Insertion
Relighting
Augmented Human
Experience
Material editing from single photo
Scene completion from photos
Motion Magnification
Phototourism
Resolution
fgbg
DirectGlobal
Computational Photography
Bit Hacking
Phot
on H
acki
ng
Computer Vision
Optics
Sensors
Signal Processing
Displays
Machine Learning
Computational Light Transport
Computational PhotographyIllumination
Co-designing Optical and Digital Processing
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Wish List Open Research
Problems
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
Oacute 2007 Marc Levoy
Digital Refocusing using Light Field Camera
125μ square-sided microlenses[Ng et al 2005]
Motion Blur in Low Light
Traditional
Deblurred Image
Blurred Photo
Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
Flutter Shutter Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred PhotoPSF == Broadband Function
Fourier Transform
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Computational Camera + Photography
Optics Sensors and ComputationsGeneralized
Sensor
Generalized OpticsComputations
Picture
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Merged Views Programmable focus and dynamic range Closed-loop Controlled Illumination Coded exposureapertures
Computational Photography
Novel Illumination
Computational Cameras
Scene 8D Ray Modulator
Display
GeneralizedSensor
Generalized OpticsProcessing
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Generalized Optics
Recreate 4D Lightfield
Light Sources
Modulators
4D Incident Lighting
4D Light Field
Computational Photography [Raskar and Tumblin]
Resources
ICCP 2012 Seattle Apr 2012Papers due Dec 2nd 2011
httpwikipediaorgcomputational_photography
httpraskarinfophoto
captures a machine-readable representation of our world tohyper-realistically synthesize the essence of our visual experience
SynthesisAnalysis
Low Level Mid Level
HighLevel
Hyper realism
Raw
Angle spectrum
aware
Non-visual Data GPS
Metadata
Priors
Cap
ture
Comprehensive
8D reflectance field
Digital
Epsilon
Coded
Essence
Computational Photography aims to make progress on
both axis
Camera ArrayHDR FoV Focal stack
Decomposition problems
Depth
Spectrum
LightFields
Human Stereo Vision
Looking Around Corners
Virtual Object Insertion
Relighting
Augmented Human
Experience
Material editing from single photo
Scene completion from photos
Motion Magnification
Phototourism
Resolution
fgbg
DirectGlobal
Computational Photography
Bit Hacking
Phot
on H
acki
ng
Computer Vision
Optics
Sensors
Signal Processing
Displays
Machine Learning
Computational Light Transport
Computational PhotographyIllumination
Co-designing Optical and Digital Processing
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Wish List Open Research
Problems
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
Oacute 2007 Marc Levoy
Digital Refocusing using Light Field Camera
125μ square-sided microlenses[Ng et al 2005]
Motion Blur in Low Light
Traditional
Deblurred Image
Blurred Photo
Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
Flutter Shutter Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred PhotoPSF == Broadband Function
Fourier Transform
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Computational Photography
Novel Illumination
Computational Cameras
Scene 8D Ray Modulator
Display
GeneralizedSensor
Generalized OpticsProcessing
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Generalized Optics
Recreate 4D Lightfield
Light Sources
Modulators
4D Incident Lighting
4D Light Field
Computational Photography [Raskar and Tumblin]
Resources
ICCP 2012 Seattle Apr 2012Papers due Dec 2nd 2011
httpwikipediaorgcomputational_photography
httpraskarinfophoto
captures a machine-readable representation of our world tohyper-realistically synthesize the essence of our visual experience
SynthesisAnalysis
Low Level Mid Level
HighLevel
Hyper realism
Raw
Angle spectrum
aware
Non-visual Data GPS
Metadata
Priors
Cap
ture
Comprehensive
8D reflectance field
Digital
Epsilon
Coded
Essence
Computational Photography aims to make progress on
both axis
Camera ArrayHDR FoV Focal stack
Decomposition problems
Depth
Spectrum
LightFields
Human Stereo Vision
Looking Around Corners
Virtual Object Insertion
Relighting
Augmented Human
Experience
Material editing from single photo
Scene completion from photos
Motion Magnification
Phototourism
Resolution
fgbg
DirectGlobal
Computational Photography
Bit Hacking
Phot
on H
acki
ng
Computer Vision
Optics
Sensors
Signal Processing
Displays
Machine Learning
Computational Light Transport
Computational PhotographyIllumination
Co-designing Optical and Digital Processing
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Wish List Open Research
Problems
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
Oacute 2007 Marc Levoy
Digital Refocusing using Light Field Camera
125μ square-sided microlenses[Ng et al 2005]
Motion Blur in Low Light
Traditional
Deblurred Image
Blurred Photo
Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
Flutter Shutter Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred PhotoPSF == Broadband Function
Fourier Transform
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Computational Photography [Raskar and Tumblin]
Resources
ICCP 2012 Seattle Apr 2012Papers due Dec 2nd 2011
httpwikipediaorgcomputational_photography
httpraskarinfophoto
captures a machine-readable representation of our world tohyper-realistically synthesize the essence of our visual experience
SynthesisAnalysis
Low Level Mid Level
HighLevel
Hyper realism
Raw
Angle spectrum
aware
Non-visual Data GPS
Metadata
Priors
Cap
ture
Comprehensive
8D reflectance field
Digital
Epsilon
Coded
Essence
Computational Photography aims to make progress on
both axis
Camera ArrayHDR FoV Focal stack
Decomposition problems
Depth
Spectrum
LightFields
Human Stereo Vision
Looking Around Corners
Virtual Object Insertion
Relighting
Augmented Human
Experience
Material editing from single photo
Scene completion from photos
Motion Magnification
Phototourism
Resolution
fgbg
DirectGlobal
Computational Photography
Bit Hacking
Phot
on H
acki
ng
Computer Vision
Optics
Sensors
Signal Processing
Displays
Machine Learning
Computational Light Transport
Computational PhotographyIllumination
Co-designing Optical and Digital Processing
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Wish List Open Research
Problems
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
Oacute 2007 Marc Levoy
Digital Refocusing using Light Field Camera
125μ square-sided microlenses[Ng et al 2005]
Motion Blur in Low Light
Traditional
Deblurred Image
Blurred Photo
Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
Flutter Shutter Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred PhotoPSF == Broadband Function
Fourier Transform
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
SynthesisAnalysis
Low Level Mid Level
HighLevel
Hyper realism
Raw
Angle spectrum
aware
Non-visual Data GPS
Metadata
Priors
Cap
ture
Comprehensive
8D reflectance field
Digital
Epsilon
Coded
Essence
Computational Photography aims to make progress on
both axis
Camera ArrayHDR FoV Focal stack
Decomposition problems
Depth
Spectrum
LightFields
Human Stereo Vision
Looking Around Corners
Virtual Object Insertion
Relighting
Augmented Human
Experience
Material editing from single photo
Scene completion from photos
Motion Magnification
Phototourism
Resolution
fgbg
DirectGlobal
Computational Photography
Bit Hacking
Phot
on H
acki
ng
Computer Vision
Optics
Sensors
Signal Processing
Displays
Machine Learning
Computational Light Transport
Computational PhotographyIllumination
Co-designing Optical and Digital Processing
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Wish List Open Research
Problems
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
Oacute 2007 Marc Levoy
Digital Refocusing using Light Field Camera
125μ square-sided microlenses[Ng et al 2005]
Motion Blur in Low Light
Traditional
Deblurred Image
Blurred Photo
Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
Flutter Shutter Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred PhotoPSF == Broadband Function
Fourier Transform
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Bit Hacking
Phot
on H
acki
ng
Computer Vision
Optics
Sensors
Signal Processing
Displays
Machine Learning
Computational Light Transport
Computational PhotographyIllumination
Co-designing Optical and Digital Processing
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Wish List Open Research
Problems
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
Oacute 2007 Marc Levoy
Digital Refocusing using Light Field Camera
125μ square-sided microlenses[Ng et al 2005]
Motion Blur in Low Light
Traditional
Deblurred Image
Blurred Photo
Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
Flutter Shutter Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred PhotoPSF == Broadband Function
Fourier Transform
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Wish List Open Research
Problems
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
Oacute 2007 Marc Levoy
Digital Refocusing using Light Field Camera
125μ square-sided microlenses[Ng et al 2005]
Motion Blur in Low Light
Traditional
Deblurred Image
Blurred Photo
Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
Flutter Shutter Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred PhotoPSF == Broadband Function
Fourier Transform
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Computational Photography
Wish List Open Research
Problems
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
Oacute 2007 Marc Levoy
Digital Refocusing using Light Field Camera
125μ square-sided microlenses[Ng et al 2005]
Motion Blur in Low Light
Traditional
Deblurred Image
Blurred Photo
Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
Flutter Shutter Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred PhotoPSF == Broadband Function
Fourier Transform
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
Oacute 2007 Marc Levoy
Digital Refocusing using Light Field Camera
125μ square-sided microlenses[Ng et al 2005]
Motion Blur in Low Light
Traditional
Deblurred Image
Blurred Photo
Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
Flutter Shutter Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred PhotoPSF == Broadband Function
Fourier Transform
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Oacute 2007 Marc Levoy
Digital Refocusing using Light Field Camera
125μ square-sided microlenses[Ng et al 2005]
Motion Blur in Low Light
Traditional
Deblurred Image
Blurred Photo
Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
Flutter Shutter Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred PhotoPSF == Broadband Function
Fourier Transform
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Motion Blur in Low Light
Traditional
Deblurred Image
Blurred Photo
Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
Flutter Shutter Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred PhotoPSF == Broadband Function
Fourier Transform
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Traditional
Deblurred Image
Blurred Photo
Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
Flutter Shutter Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred PhotoPSF == Broadband Function
Fourier Transform
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Fluttered Shutter CameraRaskar Agrawal Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
Flutter Shutter Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred PhotoPSF == Broadband Function
Fourier Transform
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Flutter Shutter Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo
Blurred PhotoPSF == Broadband Function
Fourier Transform
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Motion Blur in Low Light
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Fast periodic phenomena
4000 fps hi-speed camera
Vocal folds flapping at 404 Hz
500 fps hi-speed camera
Bottling line
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Compressive Sensing
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Periodic signals
Periodic signal with period P and band-limited to fMax = 500 Hz
Fourier transform is non-zero only at multiples of fP=1P ~ 63Hz
4fP3fP0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP
t
Periodic signal x(t) with period P
P = 16ms
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
High speed camera
Periodic signal has regularly spaced sparse Fourier coefficients Is it necessary to use a high-speed video camera Why waste bandwidth
0 fMax- fMax fP=1P 2fP-fP-2fP-4fP -3fP 4fP3fP
Nyquist Sampling of x(t)
P = 16ms Ts = 1(2 fMax)
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
P t
Strobing animation credit Wikipedia
Traditional Strobing
Use low frame-rate camera and generate beat frequencies
Low exposure to avoid blurring Low light throughputPeriod known apriori
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Random Projections Per Frame of Camera using Coded Strobing Photography
In every exposure duration observe different linear combinations of the periodic signal
Advantage of the design bull Exposure coding independent of the frequency
bull On an average light throughput is 50
tP
Coded Strobing Photography Reddy D Veeraraghavan A Raskar R IEEE PAMI 2011
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Observation Model
x at 2000fpsy at 25fps
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Signal Model
x at 2000fpsy at 25fps
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Signal amp Observation Model
A is M x N MltltN
x at 2000fpsy at 25fps
N M = 2000 25 = 80
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Recovery Sparsity
Very few non-zero elements
Structured Sparse Coefficients
y = A sMixing matrix
Basis Pursuit De-noising
Asytss min1
Observed values
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Simulation on hi-speed toothbrush
25fps normal camera 25fps coded strobing camera
Reconstructed frames 2000fps hi-speed camera
~100X speedup
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Rotating mill tool
Normal Video 25fpsReconstructed Video at 2000fps
rotating at 150Hz
Coded Strobing Video 25fps
Mill tool rotating at 50Hz
rotating at 100Hz rotating at 200Hz
Blur increases as rotational velocity increases
increasing blur
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Compressive Sensing for Images A good idea
NMNMxy where
Single Pixel Camera
compressive image measurement matrix
image
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Is Randomized Projection-based Captureapt for Natural Images
[Pandharkar Veeraraghavan Raskar 2009]
Randomized Projections
Prog
ress
ive
Pro
jecti
ons
Compression Ratio
Periodic Signals
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Ramesh Raskar Computational Illumination
Compact Programmable
Lights
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 1
Ultimate Post-capture Control
bullDigital Refocus and Motion blur
bullEmulate studio light from compact flash
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 2
Freedom from Form
bull Size Weight Power UIbull Flat camera
Bidirectional screen (BiDi)
bull Shallow DoF from tiny lens
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Convert single 2D photo into 3D Snavely Seitz SzeliskiU of WashingtonMicrosoft Photosynth
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Exploit Community Photo CollectionsU of WashingtonMicrosoft Photosynth
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 3
Understand the World
bullIdentifyrecognize Materialsbull3D Awareness
bullInteract with information
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 4
Sharing Visual Experience
bullLifeLog Auto-summarybullPrivacy in public and
authentication bullHyper-real Photo Frames
bullPrint lsquomaterialrsquo
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Wish 5
Capturing Essence
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
What are the problems with lsquorealrsquo photo in conveying information
Why do we hire artists to draw what can be photographed
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Shadows
Clutter
Many Colors
Highlight Shape Edges
Mark moving parts
Basic colors
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Depth Edges with MultiFlashRaskar Tan Feris Jingyi Yu Turk ndash ACM SIGGRAPH 2004
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Depth Discontinuities
Internal and externalShape boundaries Occluding contour Silhouettes
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Our MethodCanny
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Raskar Camera Culture MIT Media Lab
Questionsbull What will a camera look like in 1020 yearsbull How will a billion networked and
portable cameras change the social culture bull How will online photo collections
transform visual social computingbull How will movie makingnew reporting change
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Photos of tomorrow computed not recorded
httpscalarmotionwordpresscom20090315propeller-image-aliasing
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Camera Culture Group MIT Media Lab Ramesh Raskar httpraskarinfo
bull Post-capture controlbull Emulate studio lights with compact flashbull Focus and motion blur
bull New formsbull Flat camera large LCDs as camerasbull Image destabilization for larger aperture
bull Understand the worldbull Real or fakebull Place 2D photo into 3Dbull Look around cornerbull Bokode long distance barcode
bull Sharingbull Lifelogs auto summarybull PrivacyVerificationbull 6D photoframes
bull Essencebull New visual artsbull Multi-flash camerabull Delta-camera and Blind-camera
Computational Photography Wish ListSensor
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Every Photon has a Story
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
What isaround the corner
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Can you look around the corner
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Femto-Photography (Transient Imaging)FemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
bull 2011 CVPR (Pandharkar Velten Bardagjy Bawendi Raskar)bull 2009 Marr Prize Honorable Mention (Kirmani Hutchinson Davis Raskar
ICCVrsquo2009)bull 2008 Transient Light Transport (Raskar Davis March 2008)
With M Bawendi
MIT Chemistry
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Inverting Light Transport
Multiple Scattering DirectGlobal
Dual Photography
[Seitz Kutulakos Matsushita 2005] [Nayar Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos Steger 2005]
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Multi-Dimensional Light Transport
5-D Transport
Gigapan
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Collision avoidance robot navigation hellip
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Steady State 4D
Impulse Response 5D
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Scene with hidden elements
Raw Time profiles
treg
Signal Proc
Novel light transport models and inference
algorithms
Photo geometry reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Femto-PhotographyTime Resolved Multi-path Imaging
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Raskar Camera Culture MIT Media Lab
Camera Culture
Ramesh Raskar
Team
Moungi G Bawendi Professor Dept of Chemistry MITJames Davis UC Santa CruzAndreas Velten Postdoctoral Associate MIT Media LabRohit Pandharkar RA MIT Media LabOtkrist Gupta RA MIT Media LabAndrew Matthew Bardagjy RA MIT Media LabNikhil Naik RA MIT Media LabTyler Hutchison RA MIT Media LabEverett Lawson MIT Media LabRamesh Raskar Asso Prof MIT Media Lab
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Capture Setup
Photos from Streak Camera
Hidden Scene
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Motion beyond line of sight
Pandharkar Velten Bardagjy Lawson Bawendi Raskar CVPR 2011
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
hellip bronchoscopies hellip
Participating Media
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
[Nayar Krishnan Grossberg Raskar 2006]
Photo
First Bounce
Later Bounces
Direct Global
+
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Each frame = ~2ps = 06 mm of Light Travel
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Ripples of Waves
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
Inverse Problems
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
View Dependent Appearance and Iridescent color Cross section through a single M rhetenor scale
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Two Layer Displays
barrier
sensordisplay
lenslet
sensordisplay
PB = dim displaysLenslets = fixed spatial and angular resolution
Dynamic Masks = Brighter High spatial resolution
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Parallax barrier
LCD display
Limitations of 3D Display
Lanman Hirsch Kim Raskar Siggraph Asia 2010
Front
Back
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
][][][ kgifkiL
`
i
k
gfL
light box
Light Field Analysis of Barriers
g[k]k
f[i]i
L[ik]
L[ik]
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
f[i]
g[k]
L[ik]
light box
`
FGL ~
G
Content-Adaptive Parallax Barriers
k
i F L~
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Implementation
Componentsbull 22 inch ViewSonic FuHzion VX2265wm LCD [1680times1050 120 fps]
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
f[i]
g[k]
L[ik]
light box
`
FGL ~
F
G
L~
Content-Adaptive Parallax Barriers
k
i
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
0for 21 min arg 2
GF GFFGL
F
G
`L~ =
Content-Adaptive Parallax Barriers
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Rank-Constrained Displays and LF Adaptation
All dual layer display = rank-1 constraint
Light field display is a matrix approximation problem
Exploit content-adaptive parallax barriers
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Lanman Hirsch Kim Raskar Siggraph Asia 2010
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
Optimization Iteration 1
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Optimization Iteration 10
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Optimization Iteration 20
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Optimization Iteration 30
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Optimization Iteration 40
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Optimization Iteration 50
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Optimization Iteration 60
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Optimization Iteration 70
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Optimization Iteration 80
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Optimization Iteration 90
rear mask f1[ij] front mask g1[kl]
reconstruction (central view)
))](([)]([
]))([(])[(
FGWFLWFGG
GFGWGLWFF
t
t
t
t
Daniel Lee and Sebastian Seung Non-negative Matrix Factorization 1999Vincent Blondel et al Weighted Non-negative Matrix Factorization 2008
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Content-Adaptive Front Mask (1 of 9)
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Content-Adaptive Rear Mask (1 of 9)
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Emitted 4D Light Field
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Conclusion
bull Described a rank constraint for all dual-layer displays‒ With a fixed pair of masks emitted light field is rank-1
bull Achieved higher-rank approximation using temporal multiplexing‒ With T time-multiplexed masks emitted light field is rank-T‒ Constructed a prototype using off-the-shelf panels
bull Demonstrated light field display is a matrix approximation problembull Introduced content-adaptive parallax barriers
‒ Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs
0for 21 min arg 2
GF GFFGL
W
F
G
L~ =
Content-Adaptive Parallax Barriers
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Parallax Barrier Np=103 pix
Hologram NH=105 pix
xH =100 patches
θp=10 pix
w
θH =1000 pix
xp=100 slits
ϕPpropwd ϕHpropλtH
Fourier Patch
Horstmeyer Oh Cuypers Barbastathis Raskar 2009
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Augmented Light Field
118
Wigner Distribution Function
Traditional Light Field
WDF
Traditional Light Field
Augmented LF
Interference amp DiffractionInteraction w optical elements
ray optics basedsimple and powerful
wave optics basedrigorous but cumbersome
Oh Raskar Barbastathis 2009 Augmented Light Field
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Light Fields
Referenceplane
position angle
Goal Representing propagation interaction and image formation of light using purely position and angle parameters
LF propagation
(diffractive)optical
element
LF LF LF LF
LF propagation
light field transformer
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Augmented Lightfield for Wave Optics Effects
Wigner Distribution Function
Light Field
LF lt WDF
Lacks phase propertiesIgnores diffraction interferrence
Radiance = Positive
LF
Augmented Light Field
WDF
ALF ~ WDF
Supports coherentincoherent
Radiance = PositiveNegative
Virtual light sources
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Free-space propagation
Light field transformer
Virtual light projectorPossibly negative radiance
121
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Lightfield vs Hologram Displays
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
L(xθ) W(xu) Wm= sincd = delta
q Wm
pq
p d(θ)
pqq d(θ)
p Wm
Rays No Bending 1 Fresnel HG Patch
θ u
Zooming into the Light Field
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
s1
m2 m2s1
s1
(a) Parallax Barrier
(b) Hologram (c) Hybrid
Rank-1 Rank-1
Algebraic Rank Constraint
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
x
u
-Transform
ltt(x+xʹ2)t(x-xʹ2)gt
Interferencexʹ
x
(a) Two Slits Coherent
t(x+xʹ2)t(x-xʹ2)W(xu)
2x
1x
Rank-1
t(x1)t(x2)
Transform-1
u R45 D
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
L2L1
L3
ϕ1
ϕ1
ϕ1
ϕ1
L1(xθ)L2(xθ)
L3(xθ)
d
z1
hH
r
z2
L1(xθ) L2(xθ) L3(xθ)
s1m2
(a)
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Is hologram just another ray-based light fieldCan a hologram create any intensity distribution in 3DWhy hologram creates a lsquowavefrontrsquo but PB does notWhy hologram creates automatic accommodation cuesWhat is the effective resolution of HG vs PB
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Three Questionsbull What are the benefits of higher dimensional imaging
bull Why is the algebraic rank of a Light Field not full
bull What makes looking around the corner possible
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
How to do Research in Imagingbull httpraskarinfo
ndash How to come up w ideas Idea Hexagonndash How to write a paperndash How to give a talkndash Open research problemsndash How to decide merit of a projectndash How to attend a conference brainstormndash Facebookcom rRaskar
bull Tipsndash Get on SeminarTalks mailing lists worldwidendash httpwwwcsvirginiaedu~robinsYouAndYourResearchhtml ndash Why do so few scientists make significant contributions and so many are
forgotten in the long runndash Highly recommended Hamming talk at Bell Labs
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Take home pointsbull Co-design of hwsw
ndash Avoid computational or optical chauvinism in imaging ndash (Camera flashKinect)ndash Hardware cost going to zero Parallel technology trendsndash Computer vision not just mimicking human visionperceptionndash Borrow ideas from other fields astronomy scientific imaging audio
communicationsbull Photons not just Pixelsbull Change the rules of the game
ndash Optics Sensors Illum ndash Priors Sparsity Transformsndash Meta-data Internet collection Crowdsourcing
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
MIT Media Lab raskarmitedu httpcameracultureinfofbcomrraskar
bull How to do Research in Imagingbull Inverse Problems Reconstruction
Rank and Sparsitybull Co-design of Optics and Computation
ndash Photons not just pixelsndash Mid-level cues
bull Computational Photographyndash Open research problemsndash Compressive Sensing for High Speed Events
bull Limits of CS for general imagingbull Computational Light Transport
ndash Looking Around Corners trillion fpsndash Lightfields 3D Displays and Holograms
bull Apply for internshipspost-doc
Inverse ProblemsneXt
Recommended