Upload
shriram-desai
View
643
Download
7
Tags:
Embed Size (px)
DESCRIPTION
Super Resolution in Digital Image processing
Citation preview
Super-Resolution
3/7/2003 Super-Resolution 2
Super-resolution
• convolutions, blur, and de-blurring• Bayesian methods
• Wiener filtering and Markov Random Fields
• sampling, aliasing, and interpolation• multiple (shifted) images• prior-based methods
• MRFs• learned models• domain-specific models (faces)- Gary
3/7/2003 Super-Resolution 3
Linear systems
Basic properties• homogeneity T[a X] = a T[X]
• additivity T[X1+X2] = T[X1]+T[X2]
• superposition T[aX1+bX2] = aT[X1]+bT[X2]
Linear system superposition
Examples:• matrix operations (additions, multiplication)• convolutions
3/7/2003 Super-Resolution 4
Signals and linear operators
Continuous I(x)Discrete I[k] or Ik
Vector form I
Discrete linear operator y = A x
Continuous linear operator:convolution integral
g(x) = sh(,x) f() d, h(,x): impulse response
g(x) = s h(-x) f() d= [f * h](x) shift invariant
3/7/2003 Super-Resolution 5
2-D signals and convolutions
Continuous I(x,y)Discrete I[k,l] or Ik,l
2-D convolutions (discrete)
g[k,l] = m,n f[m,n] h[k-m,l-n]
= m,n f[m,n] h1[k-m]h2[l-n] separable
Gaussian kernel is separable and radial
h(x,y) = (22)-1exp-(x2+y2)/2
3/7/2003 Super-Resolution 6
Convolution and blurring
3/7/2003 Super-Resolution 7
Separable binomial low-pass filter
3/7/2003 Super-Resolution 8
Fourier transforms
Project onto a series of complex sinusoids
F[m,n] = k f[k,l] e-i 2(km+ln)
Properties:
• shifting g(x-x0) exp(-i 2fxx0)G(fx)
• differentiation dg(x)/dx i 2fxG(fx)
• convolution[f * g](x) [F G] (fx)
3/7/2003 Super-Resolution 9
Blurring examples
Increasing amounts of blur + Fourier transform
3/7/2003 Super-Resolution 10
Sharpening
Unsharp mask (darkroom photography):• remove some low-frequency content
y’ = y + s (y – g * y)
spatial (blur, sharp) freq (blur,sharp)
3/7/2003 Super-Resolution 11
Sharpening - result
Unsharp mask: original, blur (σ=1), sharp(s=0, 1, 2)
3/7/2003 Super-Resolution 12
Deconvolution
Filter by inverse of blur• easiest to do in the Fourier domain• problem: high-frequency noise amplification
3/7/2003 Super-Resolution 13
Bayesian modeling
Use prior model for image and noise
• y = g * x + n, x is original, y is blurred• p(x|y) = p(y|x)p(x)
= exp(-|y – g*x|2/2σn-2) exp(-|x|2/2σx
-2)
• -log p(x|y) |y – g*x|2σn-2 + |x|2σx
-2
where the norm || is summed squares over all pixels
3/7/2003 Super-Resolution 14
Parseval’s Theorem
Energy equivalence in spatial ↔ frequency domain
• |x|2 = |F(x)|2
• -log p(x|y) |Y(f) – G(f)X(f)|2σn-2 + |X(f)|2σx
-2
• least squares solution (∂/∂X = 0)X(f) = G(f)Y(f) / [G2(f) + σn
2/σx2]
3/7/2003 Super-Resolution 15
Wiener filtering
Optimal linear filter given noise and signal statistics
• X(f) = G(f)Y(f) / [G2(f) + σn2/σx
2]
• low frequencies: X(f) ≈ G-1(f)Y(f)boost by inverse gain (blur)
• high frequencies:X(f) ≈ G(f) σn-2σx
2 Y(f)attenuate by blur (gain)
3/7/2003 Super-Resolution 16
Wiener filtering – white noise prior
Assume all frequencies equally likely
• p(x) ~ N(0,σx2)
• X(f) = G(f)Y(f) / [G2(f) + σn2/σx
2]
• solution is too noisy in high frequencies
3/7/2003 Super-Resolution 17
Wiener filtering – pink noise prior
Assume frequency falloff (“natural statistics”)
• p(X(f)) ~ N(0,|f|-βσx2)
• X(f) = G(f)Y(f) / [G2(f) + |f|βσn2/σx
2]
• greater attenuation at high frequencies
G(f) H(f)
3/7/2003 Super-Resolution 18
Markov Random Field modeling
Use spatial neighborhood prior for image
• -log p(x) = ijCρ(xi-xj)where ρ(v) is a robust norm:• ρ(v) = v2: quadratic norm ↔ pink noise• ρ(v) = |v|: total variation (popular with maths)• ρ(v) = |v|β: natural statistics• ρ(v) = v2,|v|: Huber norm
[Schultz, R.R.; Stevenson, IEEE TIP, 1996]
i j
3/7/2003 Super-Resolution 19
MRF estimation
Set up discrete energy (quadratic or non-)
• -log p(x|y) σn-2 |y – Gx|2 + ijCρ(xi-xj)
where G is sparse convolution matrix• quadratic: solve sparse linear system• non-quadratic: use sparse non-linear least
squares (Levenberg-Marquardt, gradient descent, conjugate gradient, …)
3/7/2003 Super-Resolution 20
Sampling a signal
• sampling:• creating a discrete signal from a continuous signal
• downsampling (decimation)• subsampling a discrete signal
• upsampling• introducing zeros between samples
• aliasing• two sampled signals that differ in their original
form (many → one mapping)
3/7/2003 Super-Resolution 21
Sampling
interpolation
3/7/2003 Super-Resolution 22
Nyquist sampling theorem
Signal to be (down-) sampled must have a bandwidth no larger than twice the sample frequency
s = 2 / ns > 2 0
3/7/2003 Super-Resolution 23
Box filter (top hat)
3/7/2003 Super-Resolution 24
Ideal low-pass filter
3/7/2003 Super-Resolution 25
Simplified camera optics
1. Blur = pill-box*Bessel2 (diffr.) ≈ Gaussian
2. Integrate = box filter
3. Sample = produce single digital sample
4. Noise = additive white noise
3/7/2003 Super-Resolution 26
Aliasing
Aliasing (“jaggies” and “crawl”) is present ifblur amount < sampling (σ = 1)
• shift each image in previous pipeline by 1
3/7/2003 Super-Resolution 27
Aliasing - less
Less aliasing (“jaggies” and “crawl”) is present ifblur amount ~ sampling (σ = 2)
• shift each image in previous pipeline by 1
3/7/2003 Super-Resolution 28
Multi-image super-resolution
Exploit aliasing to recover frequencies above Nyquist cutoff
kσn-2 |yk – Gkx|2 + ijCρ(xi-xj)
where Gk are sparse convolution matrices
• quadratic: solve sparse linear system• non-quadratic: use sparse non-linear least
squares (Levenberg-Marquardt, gradient descent, conjugate gradient, …)
• projection onto convex sets (POCS)
3/7/2003 Super-Resolution 29
Multi-image super-resolution
Need:• accurate (sub-pixel) motion estimates
(Wednesday’s lecture)• accurate models of blur (pre-filtering)• accurate photometry• no (or known) non-linear pre-processing
(Bayer mosaics)• sufficient images and low-noise relative to
amount of aliasing
3/7/2003 Super-Resolution 30
Prior-based Super-Resolution
“Classical” non-Gaussian priors:
• robust or natural statistics• maximum entropy (least blurry)• constant colors (black & white images)
3/7/2003 Super-Resolution 31
Example-based Super-Resolution
William T. Freeman, Thouis R. Jones, and Egon C. Pasztor,IEEE Computer Graphics and Applications, March/April, 2002
• learn the association between low-resolution patches and high-resolution patches
• use Markov Network Model (another name for Markov Random Field) to encourage adjacent patch coherence
3/7/2003 Super-Resolution 32
Example-based Super-Resolution
William T. Freeman, Thouis R. Jones, and Egon C. Pasztor,IEEE Computer Graphics and Applications, March/April, 2002
3/7/2003 Super-Resolution 33
References – “classic”Irani, M. and Peleg. Improving Resolution by Image Registration. Graphical Models and Image
Processing, 53(3), May 1991, 231-239.
Schultz, R.R.; Stevenson, R.L. Extraction of high-resolution frames from video sequences. IEEE Trans. Image Proc., 5(6), Jun 1996, 996-1011.
Elad, M.; Feuer, A.. Restoration of a single superresolution image from several blurred, noisy, and undersampled measured images. IEEE Trans. Image Proc., 6(12) , Dec 1997, 1646-1658.
Elad, M.; Feuer, A.. Super-resolution reconstruction of image sequences. IEEE PAMI 21(9), Sep 1999, 817-834.
Capel, D.; Zisserman, A.. Super-resolution enhancement of text image sequences. CVPR 2000, I-600-605 vol. 1.
Chaudhuri, S. (editor). Super-Resolution Imaging. Kluwer Academic Publishers. 2001.
08/03/2014 Super-Resolution 34
References – strong priorsFreeman, W.T.; Pasztor, E.C.. Learning low-level vision, CVPR 1999, 182-1189 vol.2
William T. Freeman, Thouis R. Jones, and Egon C. Pasztor, Example-based super-resolution, IEEE Computer Graphics and Applications, March/April, 2002
Baker, S.; Kanade, T. Hallucinating faces. Automatic Face Gesture Recognition, 2000, 83-88.
Ce Liu; Heung-Yeung Shum; Chang-Shui Zhang. A two-step approach to hallucinating faces: global parametric model and local nonparametric model. CVPR 2001. I-192-8.