Embed Size (px)
Citation preview
Microscopy lecture 14. June 2019
Structured Illumination Microscopy (plus a little bit of 4Pi…)
Dr. Kai Wicker Team Head – Imaging and Smart Sensor Systems Carl Zeiss AG, Corporate Research and Technology
Revision: - McCutchen Aperture - Incoherent wide-field OTF Increasing the NA: - 4Pi Microscopy Beyond the Abbe-limit: Structured Illumination Microscopy (SIM) - Basic setup - PSF and OTF - SIM image formation and reconstruction
Today:
Quick revision: the incoherent wide-field OTF (and the missing cone)
Inverse
Fourier
Transform
Fourier
Transform
Inverse
Fourier
Transform
Fourier
Transform
Absolute square Convolution (Auto-correlation)
PSF
APSF
OTF
ATF
Optical Transfer Function (OTF): For incoherent microscopy techniques, e.g. fluorescence microscopy
Missing cone
Optical Transfer Function (OTF): For incoherent microscopy techniques, e.g. fluorescence microscopy
Lateral support
Missing cone
Optical Transfer Function (OTF): For incoherent microscopy techniques, e.g. fluorescence microscopy
Lateral support
Axial support
Missing cone
Optical Transfer Function (OTF): For incoherent microscopy techniques, e.g. fluorescence microscopy
Lateral support
Axial support
The missing cone and optical sectioning:
Missing cone
Lateral support
Axial support
The missing cone and optical sectioning:
Missing cone
Lateral support
Axial support
The missing cone and optical sectioning:
Missing cone
Lateral support
Axial support
Enlarging the NA:
4Pi Microscopy
Aperture increase: 4 Pi Microscope (Type C)
Fluorescence Intensity
z
z
Stefan W. Hell Max Planck Institute of Biophysical Chemistry
Göttingen, Germany
Aperture increase: 4 Pi Microscope (Type C)
Fluorescence Intensity
z
z
Stefan W. Hell Max Planck Institute of Biophysical Chemistry
Göttingen, Germany
Aperture increase: 4 Pi Microscope (Type C)
Fluorescence Intensity
z
z
Stefan W. Hell Max Planck Institute of Biophysical Chemistry
Göttingen, Germany
Aperture increase: 4 Pi Microscope (Type C)
Fluorescence Intensity
z
z
Stefan W. Hell Max Planck Institute of Biophysical Chemistry
Göttingen, Germany
Aperture increase: 4 Pi Microscope (Type C)
Sample between Coverslips
Fluorescence Intensity
z
z
Stefan W. Hell Max Planck Institute of Biophysical Chemistry
Göttingen, Germany
Aperture increase: 4 Pi Microscope (Type C)
Sample between Coverslips
Fluorescence Intensity
z
z
Stefan W. Hell Max Planck Institute of Biophysical Chemistry
Göttingen, Germany
Aperture increase: 4 Pi Microscope (Type C)
Sample between Coverslips
Illumination Emission
Fluorescence Intensity
z
z
Stefan W. Hell Max Planck Institute of Biophysical Chemistry
Göttingen, Germany
Aperture increase: 4 Pi Microscope (Type C)
Sample between Coverslips
Illumination Emission
Fluorescence Intensity
z
z
Stefan W. Hell Max Planck Institute of Biophysical Chemistry
Göttingen, Germany
Aperture increase: 4 Pi Microscope (Type C)
Sample between Coverslips
Illumination Emission
Fluorescence Intensity
z
z
Stefan W. Hell Max Planck Institute of Biophysical Chemistry
Göttingen, Germany
Aperture increase: 4 Pi Microscope (Type C)
Sample between Coverslips
Illumination Emission
Fluorescence Intensity
z
z
Stefan W. Hell Max Planck Institute of Biophysical Chemistry
Göttingen, Germany
Aperture increase: 4 Pi Microscope (Type C)
Sample between Coverslips
Illumination Emission
Fluorescence Intensity
z
z
Stefan W. Hell Max Planck Institute of Biophysical Chemistry
Göttingen, Germany
Aperture increase: 4 Pi Microscope (Type C)
Sample between Coverslips
Illumination Emission
Fluorescence Intensity
z
z
Dichromatic Beamsplitter
Stefan W. Hell Max Planck Institute of Biophysical Chemistry
Göttingen, Germany
Aperture increase: 4 Pi Microscope (Type C)
Sample between Coverslips
Illumination Emission
Detector Pinhole
Fluorescence Intensity
z
z
Dichromatic Beamsplitter
Stefan W. Hell Max Planck Institute of Biophysical Chemistry
Göttingen, Germany
Aperture increase: 4 Pi Microscope (Type C)
Sample between Coverslips
Illumination Emission
Detector Pinhole
High Sidelobes
Fluorescence Intensity
z
z
Dichromatic Beamsplitter
Stefan W. Hell Max Planck Institute of Biophysical Chemistry
Göttingen, Germany
Aperture increase: 4 Pi Microscope (Type C)
Sample between Coverslips
Illumination Emission
Detector Pinhole
High Sidelobes
Fluorescence Intensity
z
z
Dichromatic Beamsplitter
Stefan W. Hell Max Planck Institute of Biophysical Chemistry
Göttingen, Germany
Aperture increase: 4 Pi Microscope (Type C)
Sample between Coverslips
Illumination Emission
Detector Pinhole
High Sidelobes
Fluorescence Intensity
z
z
Dichromatic Beamsplitter
Stefan W. Hell Max Planck Institute of Biophysical Chemistry
Göttingen, Germany
2 Photon Effect
Aperture increase: 4 Pi Microscope (Type C)
ATF OTF
widefield
4Pi
widefield, l=500nm 4Pi, l=500nm
widefield, l=1000nm 4Pi, l=1000nm
4Pi PSFs
widefield, l=500nm 4Pi, l=500nm
widefield, l=1000nm 4Pi, l=1000nm
4Pi PSFs
widefield, l=500nm 4Pi, l=500nm
widefield, l=1000nm 4Pi, l=1000nm 2-photon, l=1000nm 4Pi, l=1000nm, 2-photon
4Pi PSFs
widefield, l=500nm 4Pi, l=500nm
widefield, l=1000nm 4Pi, l=1000nm
4Pi PSFs
widefield, l=500nm 4Pi, l=500nm
widefield, l=1000nm 4Pi, l=1000nm
4Pi PSFs
widefield, l=500nm 4Pi, l=500nm
widefield, l=1000nm 4Pi, l=1000nm 2-photon, l=1000nm 4Pi, l=1000nm, 2-photon
4Pi PSFs
Leica 4Pi
http://www.leica-microsystems.com
4Pi images Deviding Escherichia Coli
From: Bahlmann, K., S. Jakob, and S. W. Hell (2001). Ultramicr. 87: 155-164.
Image: Joerg Bewersdorf, Max-Planck-Institute for Biophysical Chemistry, Goettingen, Germany
Confocal
4Pi
Beyond the Abbe limit:
Structured Illumination Microscopy (SIM)
Real space Fourier space
Sample Sample frequencies
Structured Illumination Microscopy How it works
Real space Fourier space
Sample Sample frequencies
Structured Illumination Microscopy How it works
The moiré effect
The moiré effect
moiré patterns
high frequency
detail
Structured Illumination Microscopy How it works
The Moiré effect
high frequency
detail
high frequency
grid
Structured Illumination Microscopy How it works
The Moiré effect
high frequency
detail
high frequency
grid
low frequency
Moiré fringes
Structured Illumination Microscopy How it works
The Moiré effect
The Moiré effect
Sample
Illumination
The Moiré effect
Real space Real space
Sample Illumination
Structured Illumination Microscopy How it works
Real space Real space
Sample Illumination
Structured Illumination Microscopy How it works
Real space Real space
Sample Illumination ∙
Structured Illumination Microscopy How it works
Real space Real space
Sample Illumination ∙
Emission pattern
Structured Illumination Microscopy How it works
Sample ∙ Illumination
Real space Fourier space
„Convolution theorem“
Structured Illumination Microscopy How it works
Sample ∙ Illumination
Real space Fourier space
„Convolution theorem“
Structured Illumination Microscopy How it works
Sample ∙ Illumination
Real space Fourier space
FT
„Convolution theorem“
Multiplication
Structured Illumination Microscopy How it works
Sample ∙ Illumination
Real space Fourier space
FT
„Convolution theorem“
Multiplication
Structured Illumination Microscopy How it works
Sample ∙ Illumination
Real space Fourier space
FT
„Convolution theorem“
Multiplication
Sample Illumination
Convolution
Structured Illumination Microscopy How it works
Structured Illumination Microscopy How it works
Fourier space Fourier space
Sample Illumination
Convolution: Take the sample as a brush to „paint“ the illumination.
Structured Illumination Microscopy How it works
Fourier space Fourier space
Sample Illumination
Convolution: Take the sample as a brush to „paint“ the illumination.
Structured Illumination Microscopy How it works
Fourier space Fourier space
Sample Illumination
Convolution: Take the sample as a brush to „paint“ the illumination.
Structured Illumination Microscopy How it works
Fourier space Fourier space
Sample Illumination
Convolution: Take the sample as a brush to „paint“ the illumination.
Structured Illumination Microscopy How it works
Fourier space Fourier space
Sample Illumination
Convolution: Take the sample as a brush to „paint“ the illumination.
Real space Fourier space
Emission pattern Emission pattern frequencies
Structured Illumination Microscopy How it works
Real space Fourier space
Emission pattern Emission pattern frequencies
Structured Illumination Microscopy How it works
Real space Fourier space
Emission pattern Emission pattern frequencies SIM raw image
Structured Illumination Microscopy How it works
Real space Fourier space
Emission pattern Emission pattern frequencies SIM raw image SIM raw image frequencies
Structured Illumination Microscopy How it works
Real space Fourier space
SIM raw image SIM raw image frequencies
enhanced contrast
Moiré fringes
Structured Illumination Microscopy How it works
Structured Illumination Microscopy How it works
Real space Real space
Phase stepping of illumination pattern provides information for linear unmixing of components.
ZOOM
Structured Illumination Microscopy How it works
Real space Real space
Phase stepping of illumination pattern provides information for linear unmixing of components.
ZOOM
Structured Illumination Microscopy How it works
Real space Real space
Phase stepping of illumination pattern provides information for linear unmixing of components.
ZOOM
Structured Illumination Microscopy How it works
Real space Real space
Phase stepping of illumination pattern provides information for linear unmixing of components.
ZOOM
Structured Illumination Microscopy How it works
Real space Real space
Phase stepping of illumination pattern provides information for linear unmixing of components.
ZOOM
Structured Illumination Microscopy How it works
Real space Real space
Phase stepping of illumination pattern provides information for linear unmixing of components.
ZOOM
Structured Illumination Microscopy How it works
Real space Real space
Phase stepping of illumination pattern provides information for linear unmixing of components.
ZOOM
Fourier space
1. Separate the 3 components
Structured Illumination Microscopy How it works
Fourier space
1. Separate the 3 components
Structured Illumination Microscopy How it works
Fourier space
1. Separate the 3 components
Structured Illumination Microscopy How it works
Fourier space
1. Separate the 3 components
Structured Illumination Microscopy How it works
Fourier space
1. Separate the 3 components 2. Shift them to the correct frequencies
Structured Illumination Microscopy How it works
Fourier space
1. Separate the 3 components 2. Shift them to the correct frequencies 3. Recombine them (weighted averaging)
Structured Illumination Microscopy How it works
Fourier space
1. Separate the 3 components 2. Shift them to the correct frequencies 3. Recombine them (weighted averaging)
Structured Illumination Microscopy How it works
Fourier space
1. Separate the 3 components 2. Shift them to the correct frequencies 3. Recombine them (weighted averaging)
Structured Illumination Microscopy How it works
Fourier space
1. Separate the 3 components 2. Shift them to the correct frequencies 3. Recombine them (weighted averaging)
Real space
SIM image (x only)
Structured Illumination Microscopy How it works
Fourier space
1. Separate the 3 components 2. Shift them to the correct frequencies 3. Recombine them (weighted averaging)
Real space
SIM image (x only)
Structured Illumination Microscopy How it works
Real space Fourier space
SIM image (x only) SIM image frequencies
Structured Illumination Microscopy How it works
Real space Fourier space
SIM image (x only) SIM image frequencies
Structured Illumination Microscopy How it works
Real space Fourier space
SIM image (x only) SIM image frequencies Wide field image Wide field image frequencies
Structured Illumination Microscopy How it works
Full-field illumination
1 focus in back focal plane
Missing cone – no optical sectioning
Full-field illumination
1 focus in back focal plane
2-beam structured illumination
2 foci in back focal plane
Missing cone – no optical sectioning
2-beam structured illumination
2 foci in back focal plane
3-beam structured illumination
3 foci in back focal plane
Missing cone filled – optical sectioning
3-beam structured illumination
3 foci in back focal plane
Missing cone filled – optical sectioning
3-beam structured illumination
3 foci in back focal plane better
z-resolution
SIM-Setup
SIM-Setup
SIM-Setup
Image formation in SIM
Raw image: 𝐸 𝒓 = {𝐼 𝑆} ⊗ ℎ 𝒓
In real space
Raw image: 𝐸 𝒓 = {𝐼 𝑆} ⊗ ℎ 𝒓
Illumination pattern: 𝐼 𝒓 = 1 + cos 𝒌𝑔𝒓 + 𝜃
grating vector grating position
In real space
Raw image: 𝐸 𝒓 = {𝐼 𝑆} ⊗ ℎ 𝒓
Illumination pattern: 𝐼 𝒓 = 1 + cos 𝒌𝑔𝒓 + 𝜃
grating vector grating position
nth illumination pattern: 𝐼𝑛 𝒓 = 1 + cos 𝒌𝑔𝒓 + 𝜃𝒏
In real space
Raw image: 𝐸 𝒓 = {𝐼 𝑆} ⊗ ℎ 𝒓
Illumination pattern: 𝐼 𝒓 = 1 + cos 𝒌𝑔𝒓 + 𝜃
grating vector grating position
nth illumination pattern: 𝐼𝑛 𝒓 = 1 + cos 𝒌𝑔𝒓 + 𝜃𝒏
= 1 +1
2exp 𝑖{𝒌𝑔𝒓 + 𝜃𝒏} +
1
2exp −𝑖{𝒌𝑔𝒓 + 𝜃𝒏}
In real space
nth illumination pattern: 𝐼𝑛 𝒓 = 1 + cos 𝒌𝑔𝒓 + 𝜃𝒏
Raw image: 𝐸 𝒓 = {𝐼 𝑆} ⊗ ℎ 𝒓
= 1 +1
2exp 𝑖{𝒌𝑔𝒓 + 𝜃𝒏} +
1
2exp −𝑖{𝒌𝑔𝒓 + 𝜃𝒏} = 1 +
1
2exp 𝑖{𝒌𝑔𝒓 + 𝜃𝒏} +
1
2exp −𝑖{𝒌𝑔𝒓 + 𝜃𝒏}
Raw image: 𝐸 𝒓 = {𝐼 𝑆} ⊗ ℎ 𝒓
nth illumination pattern: 𝐼𝑛 𝒓 = 1 + cos 𝒌𝑔𝒓 + 𝜃𝒏
Illumination pattern: 𝐼 𝒓 = 1 + cos 𝒌𝑔𝒓 + 𝜃
grating vector grating position
In real space
nth illumination pattern: 𝐼𝑛 𝒓 = 1 + cos 𝒌𝑔𝒓 + 𝜃𝒏
Raw image: 𝐸 𝒓 = {𝐼 𝑆} ⊗ ℎ 𝒓
= 1 +1
2exp 𝑖{𝒌𝑔𝒓 + 𝜃𝒏} +
1
2exp −𝑖{𝒌𝑔𝒓 + 𝜃𝒏} = 1 +
1
2exp 𝑖{𝒌𝑔𝒓 + 𝜃𝒏} +
1
2exp −𝑖{𝒌𝑔𝒓 + 𝜃𝒏}
Raw image: 𝐸 𝒓 = {𝐼 𝑆} ⊗ ℎ 𝒓
nth illumination pattern: 𝐼𝑛 𝒓 = 1 + cos 𝒌𝑔𝒓 + 𝜃𝒏
Illumination pattern: 𝐼 𝒓 = 1 + cos 𝒌𝑔𝒓 + 𝜃
grating vector grating position
In real space In Fourier space
nth illumination pattern: 𝐼𝑛 𝒓 = 1 + cos 𝒌𝑔𝒓 + 𝜃𝒏
Raw image: 𝐸 𝒓 = {𝐼 𝑆} ⊗ ℎ 𝒓
= 1 +1
2exp 𝑖{𝒌𝑔𝒓 + 𝜃𝒏} +
1
2exp −𝑖{𝒌𝑔𝒓 + 𝜃𝒏} = 1 +
1
2exp 𝑖{𝒌𝑔𝒓 + 𝜃𝒏} +
1
2exp −𝑖{𝒌𝑔𝒓 + 𝜃𝒏}
Raw image: 𝐸 𝒓 = {𝐼 𝑆} ⊗ ℎ 𝒓
nth illumination pattern: 𝐼𝑛 𝒓 = 1 + cos 𝒌𝑔𝒓 + 𝜃𝒏
Illumination pattern: 𝐼 𝒓 = 1 + cos 𝒌𝑔𝒓 + 𝜃
grating vector grating position
In real space In Fourier space
𝐸 𝑛 𝒌 = 𝐼 𝑛 ⊗ 𝑆 ℎ 𝒌
𝐼 𝑛 𝒌 = 𝛿 𝒌 +1
2𝒆𝑖𝜃𝒏 𝛿 𝒌 − 𝒌𝑔 +
1
2𝒆−𝑖𝜃𝒏 𝛿 𝒌 + 𝒌𝑔
In Fourier space
𝐸 𝑛 𝒌 = 𝐼 𝑛 ⊗ 𝑆 ℎ 𝒌
𝐼 𝑛 𝒌 = 𝛿 𝒌 +1
2𝒆𝑖𝜃𝒏 𝛿 𝒌 − 𝒌𝑔 +
1
2𝒆−𝑖𝜃𝒏 𝛿 𝒌 + 𝒌𝑔
In Fourier space
𝐸 𝑛 𝒌 = 𝐼 𝑛 ⊗ 𝑆 ℎ 𝒌
𝐼 𝑛 𝒌 = 𝛿 𝒌 +1
2𝒆𝑖𝜃𝒏 𝛿 𝒌 − 𝒌𝑔 +
1
2𝒆−𝑖𝜃𝒏 𝛿 𝒌 + 𝒌𝑔
In Fourier space
𝐸 𝑛 𝒌 = 𝐼 𝑛 ⊗ 𝑆 ℎ 𝒌
𝐼 𝑛 𝒌 = 𝛿 𝒌 +1
2𝒆𝑖𝜃𝒏 𝛿 𝒌 − 𝒌𝑔 +
1
2𝒆−𝑖𝜃𝒏 𝛿 𝒌 + 𝒌𝑔
𝐸 𝑛 𝒌 = 𝑆 𝒌 ℎ 𝒌 + 𝒆𝑖𝜃𝒏1
2𝑆 𝒌 − 𝒌𝑔 ℎ 𝒌 + 𝒆−𝑖𝜃𝒏
1
2𝑆 𝒌 + 𝒌𝑔 ℎ 𝒌
In Fourier space
𝐸 𝑛 𝒌 = 𝐼 𝑛 ⊗ 𝑆 ℎ 𝒌
𝐼 𝑛 𝒌 = 𝛿 𝒌 +1
2𝒆𝑖𝜃𝒏 𝛿 𝒌 − 𝒌𝑔 +
1
2𝒆−𝑖𝜃𝒏 𝛿 𝒌 + 𝒌𝑔
𝐸 𝑛 𝒌 = 𝑆 𝒌 ℎ 𝒌 + 𝒆𝑖𝜃𝒏1
2𝑆 𝒌 − 𝒌𝑔 ℎ 𝒌 + 𝒆−𝑖𝜃𝒏
1
2𝑆 𝒌 + 𝒌𝑔 ℎ 𝒌
𝐶 0 𝒌 𝐶 1 𝒌 𝐶 −1 𝒌 Components:
Images:
In Fourier space
𝐸 𝑛 𝒌 = 𝐼 𝑛 ⊗ 𝑆 ℎ 𝒌
𝐼 𝑛 𝒌 = 𝛿 𝒌 +1
2𝒆𝑖𝜃𝒏 𝛿 𝒌 − 𝒌𝑔 +
1
2𝒆−𝑖𝜃𝒏 𝛿 𝒌 + 𝒌𝑔
𝐸 𝑛 𝒌 = 𝑆 𝒌 ℎ 𝒌 + 𝒆𝑖𝜃𝒏1
2𝑆 𝒌 − 𝒌𝑔 ℎ 𝒌 + 𝒆−𝑖𝜃𝒏
1
2𝑆 𝒌 + 𝒌𝑔 ℎ 𝒌
𝐶 0 𝒌 𝐶 1 𝒌 𝐶 −1 𝒌 Components:
Images:
In Fourier space
𝐸 𝑛 𝒌 = 𝐼 𝑛 ⊗ 𝑆 ℎ 𝒌
𝐼 𝑛 𝒌 = 𝛿 𝒌 +1
2𝒆𝑖𝜃𝒏 𝛿 𝒌 − 𝒌𝑔 +
1
2𝒆−𝑖𝜃𝒏 𝛿 𝒌 + 𝒌𝑔
𝐸 𝑛 𝒌 = 𝑆 𝒌 ℎ 𝒌 + 𝒆𝑖𝜃𝒏1
2𝑆 𝒌 − 𝒌𝑔 ℎ 𝒌 + 𝒆−𝑖𝜃𝒏
1
2𝑆 𝒌 + 𝒌𝑔 ℎ 𝒌
𝐶 0 𝒌 𝐶 1 𝒌 𝐶 −1 𝒌 Components:
Images:
In Fourier space
𝐸 𝑛 𝒌 = 𝐼 𝑛 ⊗ 𝑆 ℎ 𝒌
𝐼 𝑛 𝒌 = 𝛿 𝒌 +1
2𝒆𝑖𝜃𝒏 𝛿 𝒌 − 𝒌𝑔 +
1
2𝒆−𝑖𝜃𝒏 𝛿 𝒌 + 𝒌𝑔
𝐸 𝑛 𝒌 = 𝑆 𝒌 ℎ 𝒌 + 𝒆𝑖𝜃𝒏1
2𝑆 𝒌 − 𝒌𝑔 ℎ 𝒌 + 𝒆−𝑖𝜃𝒏
1
2𝑆 𝒌 + 𝒌𝑔 ℎ 𝒌
𝐶 0 𝒌 𝐶 1 𝒌 𝐶 −1 𝒌 Components:
Images:
In Fourier space
𝐸 𝑛 𝒌 = 𝐼 𝑛 ⊗ 𝑆 ℎ 𝒌
𝐼 𝑛 𝒌 = 𝛿 𝒌 +1
2𝒆𝑖𝜃𝒏 𝛿 𝒌 − 𝒌𝑔 +
1
2𝒆−𝑖𝜃𝒏 𝛿 𝒌 + 𝒌𝑔
𝐸 𝑛 𝒌 = 𝑆 𝒌 ℎ 𝒌 + 𝒆𝑖𝜃𝒏1
2𝑆 𝒌 − 𝒌𝑔 ℎ 𝒌 + 𝒆−𝑖𝜃𝒏
1
2𝑆 𝒌 + 𝒌𝑔 ℎ 𝒌
𝐶 0 𝒌 𝐶 1 𝒌 𝐶 −1 𝒌 Components:
Images:
In Fourier space
𝐸 𝑛 𝒌 = 𝐼 𝑛 ⊗ 𝑆 ℎ 𝒌
𝐼 𝑛 𝒌 = 𝛿 𝒌 +1
2𝒆𝑖𝜃𝒏 𝛿 𝒌 − 𝒌𝑔 +
1
2𝒆−𝑖𝜃𝒏 𝛿 𝒌 + 𝒌𝑔
𝐸 𝑛 𝒌 = 𝑆 𝒌 ℎ 𝒌 + 𝒆𝑖𝜃𝒏1
2𝑆 𝒌 − 𝒌𝑔 ℎ 𝒌 + 𝒆−𝑖𝜃𝒏
1
2𝑆 𝒌 + 𝒌𝑔 ℎ 𝒌
𝐶 0 𝒌 𝐶 1 𝒌 𝐶 −1 𝒌 Components:
Images:
𝐶 0 𝒌
𝐶 1 𝒌
𝐶 −1 𝒌
In Fourier space
𝐸 𝑛 𝒌 = 𝐼 𝑛 ⊗ 𝑆 ℎ 𝒌
𝐼 𝑛 𝒌 = 𝛿 𝒌 +1
2𝒆𝑖𝜃𝒏 𝛿 𝒌 − 𝒌𝑔 +
1
2𝒆−𝑖𝜃𝒏 𝛿 𝒌 + 𝒌𝑔
𝐸 𝑛 𝒌 = 𝑆 𝒌 ℎ 𝒌 + 𝒆𝑖𝜃𝒏1
2𝑆 𝒌 − 𝒌𝑔 ℎ 𝒌 + 𝒆−𝑖𝜃𝒏
1
2𝑆 𝒌 + 𝒌𝑔 ℎ 𝒌
𝐶 0 𝒌 𝐶 1 𝒌 𝐶 −1 𝒌 Components:
Images:
In Fourier space
𝐸 𝑛 𝒌 = 𝐼 𝑛 ⊗ 𝑆 ℎ 𝒌
𝐼 𝑛 𝒌 = 𝛿 𝒌 +1
2𝒆𝑖𝜃𝒏 𝛿 𝒌 − 𝒌𝑔 +
1
2𝒆−𝑖𝜃𝒏 𝛿 𝒌 + 𝒌𝑔
𝐸 𝑛 𝒌 = 𝑆 𝒌 ℎ 𝒌 + 𝒆𝑖𝜃𝒏1
2𝑆 𝒌 − 𝒌𝑔 ℎ 𝒌 + 𝒆−𝑖𝜃𝒏
1
2𝑆 𝒌 + 𝒌𝑔 ℎ 𝒌
𝐶 0 𝒌 𝐶 1 𝒌 𝐶 −1 𝒌 Components:
Images:
Matrix notation:
In Fourier space
𝐸 𝑛 𝒌 = 𝐼 𝑛 ⊗ 𝑆 ℎ 𝒌
𝐼 𝑛 𝒌 = 𝛿 𝒌 +1
2𝒆𝑖𝜃𝒏 𝛿 𝒌 − 𝒌𝑔 +
1
2𝒆−𝑖𝜃𝒏 𝛿 𝒌 + 𝒌𝑔
𝐸 𝑛 𝒌 = 𝑆 𝒌 ℎ 𝒌 + 𝒆𝑖𝜃𝒏1
2𝑆 𝒌 − 𝒌𝑔 ℎ 𝒌 + 𝒆−𝑖𝜃𝒏
1
2𝑆 𝒌 + 𝒌𝑔 ℎ 𝒌
𝐶 0 𝒌 𝐶 1 𝒌 𝐶 −1 𝒌 Components:
Images:
Matrix notation:
𝑬 𝒌 =
𝐸 1 𝒌
𝐸 2 𝒌
𝐸 3 𝒌
Vector of images:
In Fourier space
𝐸 𝑛 𝒌 = 𝐼 𝑛 ⊗ 𝑆 ℎ 𝒌
𝐼 𝑛 𝒌 = 𝛿 𝒌 +1
2𝒆𝑖𝜃𝒏 𝛿 𝒌 − 𝒌𝑔 +
1
2𝒆−𝑖𝜃𝒏 𝛿 𝒌 + 𝒌𝑔
𝐸 𝑛 𝒌 = 𝑆 𝒌 ℎ 𝒌 + 𝒆𝑖𝜃𝒏1
2𝑆 𝒌 − 𝒌𝑔 ℎ 𝒌 + 𝒆−𝑖𝜃𝒏
1
2𝑆 𝒌 + 𝒌𝑔 ℎ 𝒌
𝐶 0 𝒌 𝐶 1 𝒌 𝐶 −1 𝒌 Components:
Images:
Matrix notation:
𝑬 𝒌 =
𝐸 1 𝒌
𝐸 2 𝒌
𝐸 3 𝒌
Vector of images: 𝑪 𝒌 =
𝐶 −1 𝒌
𝐶 0 𝒌
𝐶 1 𝒌
Vector of components:
In Fourier space
𝐸 𝑛 𝒌 = 𝐼 𝑛 ⊗ 𝑆 ℎ 𝒌
𝐼 𝑛 𝒌 = 𝛿 𝒌 +1
2𝒆𝑖𝜃𝒏 𝛿 𝒌 − 𝒌𝑔 +
1
2𝒆−𝑖𝜃𝒏 𝛿 𝒌 + 𝒌𝑔
𝐸 𝑛 𝒌 = 𝑆 𝒌 ℎ 𝒌 + 𝒆𝑖𝜃𝒏1
2𝑆 𝒌 − 𝒌𝑔 ℎ 𝒌 + 𝒆−𝑖𝜃𝒏
1
2𝑆 𝒌 + 𝒌𝑔 ℎ 𝒌
𝐶 0 𝒌 𝐶 1 𝒌 𝐶 −1 𝒌 Components:
Images:
Matrix notation:
𝑬 𝒌 =
𝐸 1 𝒌
𝐸 2 𝒌
𝐸 3 𝒌
Vector of images: 𝑪 𝒌 =
𝐶 −1 𝒌
𝐶 0 𝒌
𝐶 1 𝒌
Vector of components:
𝑬 𝒌 = 𝑴 𝑪 𝒌
𝑴 =𝑒−𝑖𝜃1 1 𝑒𝑖𝜃1
𝑒−𝑖𝜃2 1 𝑒𝑖𝜃2
𝑒−𝑖𝜃3 1 𝑒𝑖𝜃3
Component mixing matrix:
Image reconstruction in SIM
Image reconstruction in SIM
a. Component separation
b. Component recombination
𝑬 𝒌 =
𝐸 1 𝒌
𝐸 2 𝒌
𝐸 3 𝒌
Vector of images: 𝑪 𝒌 =
𝐶 −1 𝒌
𝐶 0 𝒌
𝐶 1 𝒌
Vector of components:
𝑴 =𝑒−𝑖𝜃1 1 𝑒𝑖𝜃1
𝑒−𝑖𝜃2 1 𝑒𝑖𝜃2
𝑒−𝑖𝜃3 1 𝑒𝑖𝜃3
Component mixing matrix:
𝑬 𝒌 = 𝑴 𝑪 𝒌
Component separation
𝑬 𝒌 =
𝐸 1 𝒌
𝐸 2 𝒌
𝐸 3 𝒌
Vector of images: 𝑪 𝒌 =
𝐶 −1 𝒌
𝐶 0 𝒌
𝐶 1 𝒌
Vector of components:
𝑴 =𝑒−𝑖𝜃1 1 𝑒𝑖𝜃1
𝑒−𝑖𝜃2 1 𝑒𝑖𝜃2
𝑒−𝑖𝜃3 1 𝑒𝑖𝜃3
Component mixing matrix:
𝑬 𝒌 = 𝑴 𝑪 𝒌
invert equation
𝑪 𝒌 = 𝑴 −𝟏 𝑬 𝒌
Component separation
𝑬 𝒌 =
𝐸 1 𝒌
𝐸 2 𝒌
𝐸 3 𝒌
Vector of images: 𝑪 𝒌 =
𝐶 −1 𝒌
𝐶 0 𝒌
𝐶 1 𝒌
Vector of components:
𝑴 =𝑒−𝑖𝜃1 1 𝑒𝑖𝜃1
𝑒−𝑖𝜃2 1 𝑒𝑖𝜃2
𝑒−𝑖𝜃3 1 𝑒𝑖𝜃3
Component mixing matrix:
𝑬 𝒌 = 𝑴 𝑪 𝒌
invert equation
𝑪 𝒌 = 𝑴 −𝟏 𝑬 𝒌
𝐶 𝑚 𝒌 = 𝑴 𝒎𝒏−𝟏
𝟑
𝒏=𝟏
𝐸 𝑛 𝒌 Extracts the components from the recorded images.
Component separation
Component recombination
𝑪 𝒌 =
𝐶 −1 𝒌
𝐶 0 𝒌
𝐶 1 𝒌
=
1
2𝑆 𝒌 + 𝒌𝑔 ℎ 𝒌
𝑆 𝒌 ℎ 𝒌1
2𝑆 𝒌 − 𝒌𝑔 ℎ 𝒌
Vector of components:
Component recombination
𝑪 𝒌 =
𝐶 −1 𝒌
𝐶 0 𝒌
𝐶 1 𝒌
=
1
2𝑆 𝒌 + 𝒌𝑔 ℎ 𝒌
𝑆 𝒌 ℎ 𝒌1
2𝑆 𝒌 − 𝒌𝑔 ℎ 𝒌
Vector of components:
First step: shift components back to their true frequencies.
Component recombination
𝑪 𝒌 =
𝐶 −1 𝒌
𝐶 0 𝒌
𝐶 1 𝒌
=
1
2𝑆 𝒌 + 𝒌𝑔 ℎ 𝒌
𝑆 𝒌 ℎ 𝒌1
2𝑆 𝒌 − 𝒌𝑔 ℎ 𝒌
Vector of components:
First step: shift components back to their true frequencies.
𝐶 −1 𝒌 − 𝒌𝑔 =1
2𝑆 𝒌 ℎ 𝒌 − 𝒌𝑔
𝐶 0 𝒌 = 𝑆 𝒌 ℎ 𝒌
𝐶 1 𝒌 + 𝒌𝑔 =1
2𝑆 𝒌 ℎ 𝒌 + 𝒌𝑔
Component recombination
𝑪 𝒌 =
𝐶 −1 𝒌
𝐶 0 𝒌
𝐶 1 𝒌
=
1
2𝑆 𝒌 + 𝒌𝑔 ℎ 𝒌
𝑆 𝒌 ℎ 𝒌1
2𝑆 𝒌 − 𝒌𝑔 ℎ 𝒌
Vector of components:
First step: shift components back to their true frequencies.
𝐶 −1 𝒌 − 𝒌𝑔 =1
2𝑆 𝒌 ℎ 𝒌 − 𝒌𝑔
𝐶 0 𝒌 = 𝑆 𝒌 ℎ 𝒌
𝐶 1 𝒌 + 𝒌𝑔 =1
2𝑆 𝒌 ℎ 𝒌 + 𝒌𝑔
Component recombination
𝑪 𝒌 =
𝐶 −1 𝒌
𝐶 0 𝒌
𝐶 1 𝒌
=
1
2𝑆 𝒌 + 𝒌𝑔 ℎ 𝒌
𝑆 𝒌 ℎ 𝒌1
2𝑆 𝒌 − 𝒌𝑔 ℎ 𝒌
Vector of components:
First step: shift components back to their true frequencies.
𝐶 −1 𝒌 − 𝒌𝑔 =1
2𝑆 𝒌 ℎ 𝒌 − 𝒌𝑔
𝐶 0 𝒌 = 𝑆 𝒌 ℎ 𝒌
𝐶 1 𝒌 + 𝒌𝑔 =1
2𝑆 𝒌 ℎ 𝒌 + 𝒌𝑔
Second step: recombine components.
(e.g. simple averaging)
Final image: 𝐹 𝒌 =
1
3𝐶 −1 𝒌 − 𝒌𝑔 + 𝐶 0 𝒌 + 𝐶 1 𝒌 + 𝒌𝑔
= 𝑆 𝒌1
3
1
2ℎ 𝒌 − 𝒌𝑔 + ℎ 𝒌 +
1
2ℎ 𝒌 + 𝒌𝑔
Component recombination
𝑪 𝒌 =
𝐶 −1 𝒌
𝐶 0 𝒌
𝐶 1 𝒌
=
1
2𝑆 𝒌 + 𝒌𝑔 ℎ 𝒌
𝑆 𝒌 ℎ 𝒌1
2𝑆 𝒌 − 𝒌𝑔 ℎ 𝒌
Vector of components:
First step: shift components back to their true frequencies.
𝐶 −1 𝒌 − 𝒌𝑔 =1
2𝑆 𝒌 ℎ 𝒌 − 𝒌𝑔
𝐶 0 𝒌 = 𝑆 𝒌 ℎ 𝒌
𝐶 1 𝒌 + 𝒌𝑔 =1
2𝑆 𝒌 ℎ 𝒌 + 𝒌𝑔
Second step: recombine components.
(e.g. simple averaging)
Final image: 𝐹 𝒌 =
1
3𝐶 −1 𝒌 − 𝒌𝑔 + 𝐶 0 𝒌 + 𝐶 1 𝒌 + 𝒌𝑔
= 𝑆 𝒌1
3
1
2ℎ 𝒌 − 𝒌𝑔 + ℎ 𝒌 +
1
2ℎ 𝒌 + 𝒌𝑔
Effective SIM OTF
4.
Weighted averaging in Fourier space (not covered in lecture)
For signal-to-noise reasons, a simple averaging of components is not ideal!
Component recombination
𝐶 −1 𝒌 − 𝒌𝑔
𝐶 0 𝒌
𝐶 1 𝒌 + 𝒌𝑔
ℎ
𝒌
𝐹 𝒌 =1
3𝐶 −1 𝒌 − 𝒌𝑔 + 𝐶 0 𝒌 + 𝐶 1 𝒌 + 𝒌𝑔
= 𝑆 𝒌1
3
1
2ℎ 𝒌 − 𝒌𝑔 + ℎ 𝒌 +
1
2ℎ 𝒌 + 𝒌𝑔
Example: For high frequencies, only one component will contribute
information, but all components will contribute to noise!
Solution: Weighted averaging!
Component recombination
𝐹 𝒌 =𝑤 −1 𝒌 𝐶 −1 𝒌 − 𝒌𝑔 + 𝑤 0 𝒌 𝐶 0 𝒌 + 𝑤 1 𝒌 𝐶 1 𝒌 + 𝒌𝑔
𝑤 −1 𝒌 + 𝑤 0 𝒌 + 𝑤 1 𝒌
= 𝑆 𝒌𝑤 −1 𝒌
12 ℎ 𝒌 − 𝒌𝑔 + 𝑤 0 𝒌 ℎ 𝒌 + 𝑤 1 𝒌
12 ℎ 𝒌 + 𝒌𝑔
𝑤 −1 𝒌 + 𝑤 0 𝒌 + 𝑤 1 𝒌
Solution: Weighted averaging!
Component recombination
𝐹 𝒌 =𝑤 −1 𝒌 𝐶 −1 𝒌 − 𝒌𝑔 + 𝑤 0 𝒌 𝐶 0 𝒌 + 𝑤 1 𝒌 𝐶 1 𝒌 + 𝒌𝑔
𝑤 −1 𝒌 + 𝑤 0 𝒌 + 𝑤 1 𝒌
= 𝑆 𝒌𝑤 −1 𝒌
12 ℎ 𝒌 − 𝒌𝑔 + 𝑤 0 𝒌 ℎ 𝒌 + 𝑤 1 𝒌
12 ℎ 𝒌 + 𝒌𝑔
𝑤 −1 𝒌 + 𝑤 0 𝒌 + 𝑤 1 𝒌
𝑤 −1 𝒌 =1
2ℎ 𝒌 − 𝒌𝑔
𝑤 0 𝒌 = ℎ 𝒌
𝑤 1 𝒌 =1
2ℎ 𝒌 + 𝒌𝑔
With weights:
Solution: Weighted averaging!
Component recombination
𝐹 𝒌 =
12 ℎ 𝒌 − 𝒌𝑔 𝐶 −1 𝒌 − 𝒌𝑔 + ℎ 𝒌 𝐶 0 𝒌 +
12 ℎ 𝒌 + 𝒌𝑔 𝐶 1 𝒌 + 𝒌𝑔
12 ℎ 𝒌 − 𝒌𝑔 + ℎ 𝒌 +
12 ℎ 𝒌 + 𝒌𝑔
= 𝑆 𝒌
14 ℎ 2 𝒌 − 𝒌𝑔 + ℎ 2 𝒌 +
14 ℎ 2 𝒌 + 𝒌𝑔
12 ℎ 𝒌 − 𝒌𝑔 + ℎ 𝒌 +
12 ℎ 𝒌 + 𝒌𝑔
Solution: Weighted averaging!
Component recombination
𝐹 𝒌 =
12 ℎ 𝒌 − 𝒌𝑔 𝐶 −1 𝒌 − 𝒌𝑔 + ℎ 𝒌 𝐶 0 𝒌 +
12 ℎ 𝒌 + 𝒌𝑔 𝐶 1 𝒌 + 𝒌𝑔
12 ℎ 𝒌 − 𝒌𝑔 + ℎ 𝒌 +
12 ℎ 𝒌 + 𝒌𝑔
= 𝑆 𝒌
14 ℎ 2 𝒌 − 𝒌𝑔 + ℎ 2 𝒌 +
14 ℎ 2 𝒌 + 𝒌𝑔
12 ℎ 𝒌 − 𝒌𝑔 + ℎ 𝒌 +
12 ℎ 𝒌 + 𝒌𝑔
Effective SIM OTF
Other things to consider
a. Separate and recombine components for several pattern orientations at once.
b. Wiener filter final reconstructed Fourier image.
SIM is available in the ZEISS Elyra S.1 & PS.1
ZEISS Elyra S.1 & PS.1 Example Images
SIM is available in the ZEISS Elyra S.1 & PS.1
ZEISS Elyra S.1 & PS.1 Example Images
Shigella sp. Actin (yellow) and Chromatin (cyan) Images by: • Volker Brinkmann,
MPI for Infection Biology • Stephan Kuppig
Carl Zeiss Microscopy
Shigella sp. Actin (yellow) and Chromatin (cyan) Images by: • Volker Brinkmann,
MPI for Infection Biology • Stephan Kuppig
Carl Zeiss Microscopy
Shigella sp. Actin (yellow) and Chromatin (cyan) Images by: • Volker Brinkmann,
MPI for Infection Biology • Stephan Kuppig
Carl Zeiss Microscopy
Shigella sp. Actin (yellow) and Chromatin (cyan) Images by: • Volker Brinkmann,
MPI for Infection Biology • Stephan Kuppig
Carl Zeiss Microscopy
Shigella sp. Actin (yellow) and Chromatin (cyan) Images by: • Volker Brinkmann,
MPI for Infection Biology • Stephan Kuppig
Carl Zeiss Microscopy
Shigella sp. Actin (yellow) and Chromatin (cyan) Images by: • Volker Brinkmann,
MPI for Infection Biology • Stephan Kuppig
Carl Zeiss Microscopy
Non-linear Structured Illumination
Microscopy
Images: Mats Gustafsson
Structured Illumination Microscopy How it works
Actin Filaments
488nm, > 510 nm
24 lp/mm = 88% of frequency limit
Plan-Apochromat 100x/1.4 oil iris
0.5 µm
We need powerful reconstruction algorithms! Otherwise we will get reconstruction artefacts.
Structured Illumination Microscopy How it works
Actin Filaments
488nm, > 510 nm
24 lp/mm = 88% of frequency limit
Plan-Apochromat 100x/1.4 oil iris
0.5 µm 0.5 µm
We need powerful reconstruction algorithms! Otherwise we will get reconstruction artefacts.
0.5 µm
Structured Illumination Microscopy How it works
We need powerful reconstruction algorithms! Otherwise we will get reconstruction artefacts.
Actin Filaments
488nm, > 510 nm
24 lp/mm = 88% of frequency limit
Plan-Apochromat 100x/1.4 oil iris
0.5 µm
Thank you for your attention!
