Super-Resolution Digital Photography CSE558, Spring 2003 Richard Szeliski

Super-Resolution

Digital PhotographyCSE558, Spring 2003

Richard Szeliski

3/7/2003 Super-Resolution 2

Super-resolution

• convolutions, blur, and de-blurring• Bayesian methods

• Wiener filtering and Markov Random Fields

• sampling, aliasing, and interpolation• multiple (shifted) images• prior-based methods

• MRFs• learned models• domain-specific models (faces)- Gary


Linear systems

Basic properties• homogeneity T[a X] = a T[X]

• additivity T[X1+X2] = T[X1]+T[X2]

• superposition T[aX1+bX2] = aT[X1]+bT[X2]

Linear system superposition

Examples:• matrix operations (additions, multiplication)• convolutions


Signals and linear operators

Continuous I(x)Discrete I[k] or Ik

Vector form I

Discrete linear operator y = A x

Continuous linear operator:convolution integral

g(x) = sh(,x) f() d, h(,x): impulse response

g(x) = s h(-x) f() d= [f * h](x) shift invariant


2-D signals and convolutions

Continuous I(x,y)Discrete I[k,l] or Ik,l

2-D convolutions (discrete)

g[k,l] = m,n f[m,n] h[k-m,l-n]

= m,n f[m,n] h1[k-m]h2[l-n] separable

Gaussian kernel is separable and radial

h(x,y) = (22)-1exp-(x2+y2)/2


Convolution and blurring


Separable binomial low-pass filter


Fourier transforms

Project onto a series of complex sinusoids

F[m,n] = k f[k,l] e-i 2(km+ln)

Properties:

• shifting g(x-x0) exp(-i 2fxx0)G(fx)

• differentiation dg(x)/dx i 2fxG(fx)

• convolution[f * g](x) [F G] (fx)


Blurring examples

Increasing amounts of blur + Fourier transform


Sharpening

Unsharp mask (darkroom photography):• remove some low-frequency content

y’ = y + s (y – g * y)

spatial (blur, sharp) freq (blur,sharp)


Sharpening - result

Unsharp mask: original, blur (σ=1), sharp(s=0, 1, 2)


Deconvolution

Filter by inverse of blur• easiest to do in the Fourier domain• problem: high-frequency noise amplification


Bayesian modeling

Use prior model for image and noise

• y = g * x + n, x is original, y is blurred• p(x|y) = p(y|x)p(x)

= exp(-|y – g*x|2/2σn-2) exp(-|x|2/2σx

-2)

• -log p(x|y) |y – g*x|2σn-2 + |x|2σx

-2

where the norm || is summed squares over all pixels


Parseval’s Theorem

Energy equivalence in spatial ↔ frequency domain

• |x|2 = |F(x)|2

• -log p(x|y) |Y(f) – G(f)X(f)|2σn-2 + |X(f)|2σx

-2

• least squares solution (∂/∂X = 0)X(f) = G(f)Y(f) / [G2(f) + σn

2/σx2]


Wiener filtering

Optimal linear filter given noise and signal statistics

• X(f) = G(f)Y(f) / [G2(f) + σn2/σx

2]

• low frequencies: X(f) ≈ G-1(f)Y(f)boost by inverse gain (blur)

• high frequencies:X(f) ≈ G(f) σn-2σx

2 Y(f)attenuate by blur (gain)


Wiener filtering – white noise prior

Assume all frequencies equally likely

• p(x) ~ N(0,σx2)

• X(f) = G(f)Y(f) / [G2(f) + σn2/σx

2]

• solution is too noisy in high frequencies


Wiener filtering – pink noise prior

Assume frequency falloff (“natural statistics”)

• p(X(f)) ~ N(0,|f|-βσx2)

• X(f) = G(f)Y(f) / [G2(f) + |f|βσn2/σx

2]

• greater attenuation at high frequencies

G(f) H(f)


Markov Random Field modeling

Use spatial neighborhood prior for image

• -log p(x) = ijCρ(xi-xj)where ρ(v) is a robust norm:• ρ(v) = v2: quadratic norm ↔ pink noise• ρ(v) = |v|: total variation (popular with maths)• ρ(v) = |v|β: natural statistics• ρ(v) = v2,|v|: Huber norm

[Schultz, R.R.; Stevenson, IEEE TIP, 1996]

i j


MRF estimation

Set up discrete energy (quadratic or non-)

• -log p(x|y) σn-2 |y – Gx|2 + ijCρ(xi-xj)

where G is sparse convolution matrix• quadratic: solve sparse linear system• non-quadratic: use sparse non-linear least

squares (Levenberg-Marquardt, gradient descent, conjugate gradient, …)


Sampling a signal

• sampling:• creating a discrete signal from a continuous signal

• downsampling (decimation)• subsampling a discrete signal

• upsampling• introducing zeros between samples

• aliasing• two sampled signals that differ in their original

form (many → one mapping)


Sampling

interpolation


Nyquist sampling theorem

Signal to be (down-) sampled must have a bandwidth no larger than twice the sample frequency

s = 2 / ns > 2 0


Box filter (top hat)


Ideal low-pass filter


Simplified camera optics

1. Blur = pill-box*Bessel2 (diffr.) ≈ Gaussian

2. Integrate = box filter

3. Sample = produce single digital sample

4. Noise = additive white noise


Aliasing

Aliasing (“jaggies” and “crawl”) is present ifblur amount < sampling (σ = 1)

• shift each image in previous pipeline by 1


Aliasing - less

Less aliasing (“jaggies” and “crawl”) is present ifblur amount ~ sampling (σ = 2)

• shift each image in previous pipeline by 1


Multi-image super-resolution

Exploit aliasing to recover frequencies above Nyquist cutoff

kσn-2 |yk – Gkx|2 + ijCρ(xi-xj)

where Gk are sparse convolution matrices

• quadratic: solve sparse linear system• non-quadratic: use sparse non-linear least

squares (Levenberg-Marquardt, gradient descent, conjugate gradient, …)

• projection onto convex sets (POCS)


Multi-image super-resolution

Need:• accurate (sub-pixel) motion estimates

(Wednesday’s lecture)• accurate models of blur (pre-filtering)• accurate photometry• no (or known) non-linear pre-processing

(Bayer mosaics)• sufficient images and low-noise relative to

amount of aliasing


Prior-based Super-Resolution

“Classical” non-Gaussian priors:

• robust or natural statistics• maximum entropy (least blurry)• constant colors (black & white images)


Example-based Super-Resolution

William T. Freeman, Thouis R. Jones, and Egon C. Pasztor,IEEE Computer Graphics and Applications, March/April, 2002

• learn the association between low-resolution patches and high-resolution patches

• use Markov Network Model (another name for Markov Random Field) to encourage adjacent patch coherence


Example-based Super-Resolution

William T. Freeman, Thouis R. Jones, and Egon C. Pasztor,IEEE Computer Graphics and Applications, March/April, 2002


References – “classic”Irani, M. and Peleg. Improving Resolution by Image Registration. Graphical Models and Image

Processing, 53(3), May 1991, 231-239.

Schultz, R.R.; Stevenson, R.L. Extraction of high-resolution frames from video sequences. IEEE Trans. Image Proc., 5(6), Jun 1996, 996-1011.

Elad, M.; Feuer, A.. Restoration of a single superresolution image from several blurred, noisy, and undersampled measured images. IEEE Trans. Image Proc., 6(12) , Dec 1997, 1646-1658.

Elad, M.; Feuer, A.. Super-resolution reconstruction of image sequences. IEEE PAMI 21(9), Sep 1999, 817-834.

Capel, D.; Zisserman, A.. Super-resolution enhancement of text image sequences. CVPR 2000, I-600-605 vol. 1.

Chaudhuri, S. (editor). Super-Resolution Imaging. Kluwer Academic Publishers. 2001.


References – strong priorsFreeman, W.T.; Pasztor, E.C.. Learning low-level vision, CVPR 1999, 182-1189 vol.2

William T. Freeman, Thouis R. Jones, and Egon C. Pasztor, Example-based super-resolution, IEEE Computer Graphics and Applications, March/April, 2002

Baker, S.; Kanade, T. Hallucinating faces. Automatic Face Gesture Recognition, 2000, 83-88.

Ce Liu; Heung-Yeung Shum; Chang-Shui Zhang. A two-step approach to hallucinating faces: global parametric model and local nonparametric model. CVPR 2001. I-192-8.

Documents

Super-Resolution Digital Photography CSE558, Spring 2003 Richard Szeliski