Embed Size (px)
Citation preview
Light field microscopy
Marc Levoy, Ren Ng, Andrew Adams
Matthew Footer, Mark Horowitz
Stanford ComputerGraphics Laboratory
Marc Levoy
Executive summary
• captures the 4D light field inside a microscope
• yields perspective flyarounds and focal stacks from a single snapshot, but at lower spatial resolution
• focal stack → deconvolution microscopy → volume data
Marc Levoy
Devices for recording light fields
smallscenes
bigscenes
• handheld camera [Buehler 2001]
• camera gantry [Stanford 2002]
• array of cameras [Wilburn 2005]
• plenoptic camera [Ng 2005]
• light field microscope (this paper)
(using geometrical optics)
Marc Levoy
Light fields at micron scales
• wave optics must be considered– diffraction limits the spatial × angular resolution
• most objects are no longer opaque– each pixel is a line integral through the object
» of attenuation
» or emission
– can reconstruct 3D structure from these integrals» tomography
» 3D deconvolution
Marc Levoy
Conventional versus plenoptic camera
Marc Levoy
Conventional versus plenoptic camera
uv-plane st-plane
square-sided microlenses
125μ
Marc Levoy
Digital refocusing
• refocusing = summing windows extracted from several microlenses
Σ
Σ
Marc Levoy
Example of digital refocusing
Marc Levoy
Refocusing portraits
Marc Levoy
Macrophotography
Marc Levoy
Digitally moving the observer
• moving the observer = moving the window we extract from the microlenses
Σ
Σ
Marc Levoy
Example of moving the observer
Marc Levoy
Moving backward and forward
Marc Levoy
A light field microscope (LFM)
objective
specimen
intermediateimage plane
eyepiece
Marc Levoy
A light field microscope (LFM)
• 40x / 0.95NA objective
↓
0.26μ spot on specimen× 40x = 10.4μ on sensor
↓
2400 spots over 25mm field
• 1252-micron microlenses
↓
200 × 200 microlenses with12 × 12 spots per microlens
objective
specimen
intermediateimage plane
eyepiecesensor
→ reduced lateral resolution on specimen= 0.26μ × 12 spots = 3.1μ
Marc Levoy
A light field microscope (LFM)
sensor
160mm
2.5mm
0.2mm
Marc Levoy
Example light field micrograph
• orange fluorescent crayon
• mercury-arc source + blue dichroic filter
• 16x / 0.5NA (dry) objective
• f/20 microlens array
• 65mm f/2.8 macro lens at 1:1
• Canon 20D digital camera
ordinary microscope light field microscope
200μ
Marc Levoy
The geometry of the light fieldin a microscope
• microscopes make orthographic views
• translating the stage in X or Y provides no parallax on the specimen
• out-of-plane features don’t shift position when they come into focus
f
objective lensesare telecentric
Marc Levoy
Panning and focusing
panning sequence focal stack
Marc Levoy
Mouse embryo lung(16x / 0.5NA water immersion)
light fieldpan focal stack
200μ
Marc Levoy
Axial resolution(a.k.a. depth of field)
• wave term + geometrical optics term
• ordinary microscope (16x/0.4NA (dry), e = 0)
• with microlens array (e = 125μ)
• stopped down to one pixel per microlens
eNAM
n
NA
nλDOFtot
2
3.34.0
1535.02
8.225.193.31254.016
1
4.0
1535.02
237spots125.193.3
→ number of slicesin focal stack
= 12
(geometrical optics dominates)
(wave optics dominates)
Marc Levoy
3D reconstruction
• confocal scanning [Minsky 1957]
• shape-from-focus [Nayar 1990]
• deconvolution microscopy [Agard 1984]
– 4D light field → digital refocusing →3D focal stack → deconvolution microscopy →3D volume data
(UMIC SUNY/Stonybrook) (Noguchi) (DeltaVision)
Marc Levoy
3D deconvolution
• object * PSF → focus stack {object} × {PSF} → {focus stack} {focus stack} {PSF} → {object}
• spectrum contains zeros, due to missing rays
• imaging noise is amplified by division by ~zeros
• reduce by regularization, e.g. smoothing
focus stack of a point in 3-space is the 3D PSF of that imaging system
[McNally 1999]
{PSF}
Marc Levoy
Silkworm mouth(40x / 1.3NA oil immersion)
slice of focal stack slice of volume volume rendering
100μ
Marc Levoy
Insect legs(16x / 0.4NA dry)
volume rendering all-focus image[Agarwala 2004]
200μ
Marc Levoy
3D reconstruction (revisited)
• 4D light field → digital refocusing →3D focal stack → deconvolution microscopy →3D volume data
• 4D light field → tomographic reconstruction →3D volume data
(from Kak & Slaney)
(DeltaVision)
Marc Levoy
Implications of this equivalence
• light fields of minimally scattering volumes contain only 3D worth of information, not 4D
• the extra dimension serves to reduce noise, but could be re-purposed?
OpticalProjectionTomography[Sharpe 2002]
Marc Levoy
Conclusions
• captures 3D structure of microscopic objects in a single snapshot, and at a single instant in time
Calcium fluorescent imagingof zebrafish larvae optic tectumduring changing visual stimula
Marc Levoy
Conclusions
• captures 3D structure of microscopic objects in a single snapshot, and at a single instant in time
but...
• sacrifices spatial resolution to obtain control over viewpoint and focus
• 3D reconstruction fails if specimen is too thick or too opaque
Marc Levoy
Future work
• extending the field of view by correcting digitally for objective aberrations
Nikon 40x 0.95NA (dry) Plan-Apo
Marc Levoy
Future work
• extending the field of view by correcting digitally for objective aberrations
• microlenses in the illumination path→ an imaging microscope scatterometer
200μ
angular dependenceof reflection fromsingle squid iridophore
Marc Levoyhttp://graphics.stanford.edu/projects/lfmicroscope
