Image Priors and Optimization Methods for Low-level Vision

Dilip Krishnan

Introduction• We consider underconstrained image processing problems such as denoising, deblurring and image inpainting.• Also called Linear Inverse Problems.• To overcome ill-posed nature and to introduce robustness to noise, image priors (regularization) are needed.

Introduction• Canonical problem for prior-based methods:

• There are 2 issues one seeks to address: • Develop better priors: lead to higher quality results (visual and/or SNR).• Develop fast numerical algorithms to solve the resulting problems (to handle large image sizes) fast and accurately.

minx kBx ¡ gk22 + r(x)

Existing Approaches to Denoising• Prior-based Methods: Nonlinear Total Variation based Noise Removal Algorithms, Rudin, Osher, Fatemi, 1992.

• Multiscale (wavelet) Decomposition Methods: Image Denoising using Scale Mixtures of Gaussians in the Wavelet Domain, Portilla, Strela, Wainwright, Simoncelli, 2003.

Existing Approaches to Denoising• Dictionary-based Methods: Learning Multiscale Sparse Representations for Image and Video Restoration, Mairal, Shapiro, Elad, 2008.

• Filtering Methods: A Fast Approximation of the Bilateral Filter using a Signal Processing Approach, Paris, Durand, 2006.• Extension: called non-local means which provides state-of-the art results.

Existing Approaches to Non-Blind Deblurring

• In non-blind deblurring, we assume that the blurring kernel (blur filter) is known.• Wiener Filter: Classical method; tries to find an ‘inverse blur filter’ based on noise characteristics – requires knowledge of unknown signal and noise.• Richardson-Lucy: Simple iterative method – causes ringing.

Existing Approaches to Non-Blind Deblurring

• Prior-based methods:• l_1: Linearized Bregman Methods for Frame-Based Image Deblurring, Cai, Osher, Shen, 2009.• Total Variation: A Nonlinear Primal-Dual Method for Total Variation-Based Image Restoration, Chan, Golub, Mulet, 1999: • inspired by earlier idea of Conn and Overton on joint primal-dual minimization for minimizing sum of Euclidean norms.

• Sub l_1 on gradients: Fast Image Deconvolution using Hyper-Laplacian Priors, Krishnan, Fergus 2009.

Existing Approaches to Blind Deconvolution

• Blurring kernel is unknown.• Difficult problem as blurring kernel and image are all unknown, and there is noise in the image.• Prior-based methods: • l_1: , Blind Motion Deblurring from a single image using Sparse Approximation Priors, Cai et. al. 2009.• Gradient Based: Removing Camera Shake from a Single Photograph, Fergus et. al. 2006. • Sub l_1 on gradients: Blind Image Deblurring using Image Statistics, Levin, 2006.

Spatially Varying Deconvolution

• Most generalized (and therefore, difficult) problem where the blur kernel is spatially varying.• Potentially, there could be a different blur kernel at every point in the image. • Only gradient-based priors used currently.

Commonly Used Priors and Related Assumptions

• prior : Image to be recovered is sparse under some transformation (e.g. wavelet); problem formulation:

• : inverse wavelet transform; : blur kernel; • Other possibilities are DFT, curvelets etc.

minx kBF x ¡ gk22 + kxk1

l1

F B

Commonly Used Priors and Related Assumptions

• Total Variation: Gradient of image is sparse i.e. image is piecewise smooth (many flat areas, sharp transitions):

• Content aware prior: Cho et. al. 2010: Spatially varying prior that depends on image content:

minx kBx ¡ gk22 + kxkT V

kxkT V =P p

jr xxj2 + jr yxj2

minx kBx ¡ gk22 +

Pi (jr xxj®i

i + jr yxj®ii )

Commonly Used Priors and Related Assumptions

• Sparsity under learned filters: Natural Image Statistics and Efficient Coding, Olshausen and Field 1996.• Filters are learnt using a neural network; and sparsity is imposed on the coefficients of the image convolved with these filters.

minx kBx ¡ gk22 +

Pi kx ©f i k1

Commonly Used Priors and Related Assumptions

• Examples of such learned filters (from the paper):

Comments on Priors

• These priors work reasonably well for denoising; and other approaches are also available for denoising problem.• Not so well for deblurring or blind deconvolution. • Cost of prior is higher for unblurred images than for blurry images. • This motivates the search for new priors.

Optimization Techniques for

• We consider commonly used techniques for solving problems of the type:

• Problems arise in deblurring, denoising and compressed sensing. • Large size of images makes efficient techniques a necessity.• Even an average sized 1000 x 1000 image means ~1 million unknowns.• Second order methods such as interior point are infeasible for large number of unknowns.

minx12kAx ¡ bk2 + ¸kxk1

l2 ¡ l1

Optimization Techniques for

• Reference: L1-L2 Optimization in Signal and Image Processing, Zibulevsky and Elad, 2010.• Wide variety of techniques are used: mostly first-order or iterative methods. • Iterative Shrinkage Methods (ISTA) are of the type:

• S is a component-wise shrinkage operation.• Derived from a quadratic approximation to the original problem.

l2 ¡ l1

xk+1 = S¸ =c(1cAT (Axk ¡ b))

Optimization Techniques for

• This basic shrinkage was recently accelerated using Nesterov methods: A Method for Solving the Convex Programming Problem with Convergence Rate O(1/k^2), Y. E. Nesterov, 1983.• Resulting algorithm is called FISTA:

l2 ¡ l1

xk+1 = S¸ =c(1cAT (Azk ¡ b))

zk = xk + tk ¡ 1tk +1(xk ¡ xk¡ 1)

tk = (1+q

1+ 4t2k¡ 1);t1 = 1

Optimization Techniques for

• Bregman-type methods: Developed recently by Osher and fellow-researchers at UCLA.• Motivation: use a different approximation of the objective function based on Bregman distance.• Interestingly, the resulting algorithm actually ends up very similar to Iterative Shrinkage Methods:

l2 ¡ l1

xk+1 = (1=c)S¸ (AT bk+1)

bk+1 = bk + (b¡ Axk)

Optimization Techniques for

• Now consider TV-regularized problems:

• Previous methods for can be extended by using splitting techniques.• Introduce dummy variables and make a new problem:

• Then solve alternately for x,w. • Problem in x is quadratic – often, can be solved very fast (using FFT’s).• Problem in w is of type. • The Split Bregman method for L1 Regularized Problems, Goldstein et. al., 2009.

l2 ¡ TV

minx12kAx ¡ bk2 + ¸kxkT V

l2 ¡ l1

minx;w12kAx ¡ bk2 + ¯kr x ¡ wk2 + ¸kwk1

l2 ¡ l1

Optimization Techniques for

• Problems of the type:

• D(x) is identity or gradient operator. • Reweighted least squares can be used: slow, but accurate: Iteratively Re-weighted Least Squares Minimization for Sparse Recovery, Daubechies et. al., 2009.• Splitting methods can also be used in conjuction with look-up tables to solve the seperable shrinkage problem: very effective in practice.

l2 ¡ lp(N onconvex)

minx12kAx ¡ bk2 + ¸

Pi jD(x)jpi ;p< 1

Previous Work

Overview

• Dark Flash Photography: SIGGRAPH 2009.

• Enables dazzle-free photography in low-light conditions.

• Fast Image Deconvolution using Hyper Laplacian Priors: NIPS 2009.

• Fast scheme to solve deconvolution/denoising problems that use nonconvex priors.

Overview

• Dark Flash Photography: SIGGRAPH 2009.

• Enables dazzle-free photography in low-light conditions.

• Fast Image Deconvolution using Hyper Laplacian Priors: NIPS 2009.

• Fast scheme to solve deconvolution/denoising problems that use nonconvex priors.

Our Camera & Dark Flash

Dark Flash

Emits Ultraviolet (UV) and Infrared (IR) light just outside visible wavelength range

Dark Flash Photography• Dark flash is ~200 times dimmer than conventional

Ground truthReconstruction2. Ambient image1. Dark Flash image

Key Challenges

1. How to add light to the scene without it being perceived by people.

2. How to obtain an image with correct colors.

Human Retina Spectral Sensitivity

(Vos 1978)

Normal Camera Spectral Sensitivity• Spectral response

curves for an unmodified camera

Modify standard camera: 1. Remove IR-blocksensor filter

2. Add lens filter to remove IR >800nm

Some sensitivityin UV & IR just outside visible

Dark Flash Camera Spectral Sensitivity

IR

Comparison with Human Sensitivity

• Camera’s response is significantly broader

• Possible to addlight without being easilyperceived

Human Response

Curve

IR

Dark Flash Emission Spectrum

CameraSpectralSensitivity

Dark FlashEmission

Key Challenges

1. How to add light to the scene without it being perceived by people.

2. How to obtain an image with correct colors.

Two Images: Five Spectral Bands

• In Dark Flash image:– “Blue” channel records UV– “Red” channel records IR

Assumptions

1. Little ambient UV and IR light

2. UV/IR flash dominates ambient visible light

2. Ambient

R G B

1. Dark Flash

IR UV

Reconstruction of Red Channel

Ambient Red

Ground Truth Red

Inte

nsi

ties

• Compare to ground truth reference

• Ambient red has correct intensities but is noisy

Intensity Constraint

Ambient Red

Output Red

Inte

nsi

ties

Gra

die

nt

s

?

?

≈

Spectral Constraint

• Gradients in Red and IR are similar at most pixels

• But differ when material changes

Ground Truth Gradients in RedGradients in

IR

• Need to allow occasional disagreement between gradients

Spectral Constraint

(Also for UV flash)

Gradients of Output Red

0.7-Gradients in IR

?Minimize:

Final Cost Function

• Sum over all pixels . • = 0.7.• is IR channel; is UV.• 3 cost functions – one for each channel . • and are weighting terms.• - mask to handle specularities/highlights

F1 F3

p®

¹ j ·m(p)

j

argminR j

X

p

[¹ j m(p)(R j (p) ¡ A j (p))2 + · m(p)jr R j (p)j®

+jr R j (p) ¡ r F1(p)j®+ jr R j (p) ¡ r F3(p)j®]

UV Spectral Term

IR Spectral Term

Likelihood

Sparsity

a=0.7 a=1 a=2

Implies that spectral reflectances are the

same in UV/IR and visible

NOT TRUE

Effect of Varying α in Spectral Term

Non-convexoptimization

• Models actual distribution• Later part of talk focuses on fast solution scheme.

Convex optimization

Fast: ~2 mins in Matlabfor 1.4 megapixel image

Can be easily be implemented on GPU

Best results Acceptable results Unacceptable results

Results

Ambient : 1/20th sec

Reconstruction

Long exposure: 4 sec

1/64t

h 1/90th 1/256th

Reconstruction

Ambient (Fraction of Normal Illumination)

Limitations

• Rudimentary handling of shadows and highlights– Currently use simple mask (as in existing flash/no-flash work)

• Need some ambient illumination– Won’t work in complete darkness

• Color drift & loss of detail at very high noise levels

• Lack of UV/IR edges causes reconstruction to suffer

Taking Photos in Low Light Levels

Overview of Talk

• Dark Flash Photography: SIGGRAPH 2009.

• Enables dazzle-free photography in low-light conditions.

• Fast Image Deconvolution using Hyper Laplacian Priors: NIPS 2009.

• Fast scheme to solve deconvolution problems that use nonconvex priors.

Hyper-Laplacian Priors

• Good models of gradient distributions in natural images.• Heavy tails are well modeled. • Used in deblurring, denoising, super-resolution etc.• Drawback: cause problems to become non-convex.

Non-blind Deconvolution• We consider the standard non-blind deconvolution problem:

minx

#pixelsX

i=1

(¸2

(x ©k ¡ y)2i +

#f i lter sX

j =1

j(x ©f j )i j®)

Likelihood Prior

• For >= 1, problem is convex and a global minimum can be found.• We consider the solution of this problem when < 1.• We want , given blurry/noise image and kernel . • are gradient filters.

x ®

x y kf j

®

®

x

Existing Methods• Iterative Reweighted Least Squares/Conjugate Gradients • Very slow for large image sizes. • IRLS takes ~ 1 hour for a 1 megapixel image.• E.g.: Levin et. al. SIGGRAPH 2007.

• For = 1, Bregman iterations/iterative shrinkage techniques work well.• Second-order methods are infeasible.

®

Our Algorithm• We use a splitting method. • Introduce a new auxiliary variable for each derivative filter. • Old cost function:

• New cost function:

• minx;w

X(¸2

(x ©k ¡ y)2i +

X ¯2

((x ©f j )i ¡ jwj ji )2 + jwj j®i ))

minx

#pixelsX

i=1

(¸2

(x ©k ¡ y)2i +

#f i lter sX

j =1

j(x ©f j )i j®)x ®

x ®x

x

Our Algorithm• We use a splitting method. • Introduce a new auxiliary variable for each derivative filter. • Old cost function:

• New cost function:

• minx;w

X(¸2

(x ©k ¡ y)2i +

X ¯2

((x ©f j )i ¡ jwj ji )2 + jwj j®i ))

minx

#pixelsX

i=1

(¸2

(x ©k ¡ y)2i +

#f i lter sX

j =1

j(x ©f j )i j®)x ®

x ®x

Split

x

Our Algorithm• •

•As , the solution to this problem approaches that of the original problem.• Now we can use an alternating mechanism to first minimize for , and then minimize for each . • Coupled with a continuation method on .

wjx

¯ ! 1

¯

minx;w

X(¸2

(x ©k ¡ y)2i +

X ¯2

((x ©f j )i ¡ jwj ji )2 + jwj j®i ))x ®x

x sub-problem• is kept fixed from previous iteration.• sub-problem is quadratic:

• Can be solved using FFTs (assuming circular boundary conditions).• Need to solve:

• For 2 derivative filters, we require only 3 FFT’s at every iteration.

x

minX

x

(¸2

(x ©k ¡ y)2i +

X ¯2

((x ©f j )i ¡ jwj ji )2)

x = F ¡ 1((P

j F (F j )¤ ±F (wj )) + (¸=̄ )F (K )¤ ±F (y)P

j (F (F j )¤ ±F (F j )) + (¸=̄ )F (K )¤ ±F (K ))

x x

x

w

w sub-problem• sub-problem is component-wise separable; is fixed from previous iteration.• For a fixed pixel and filter , it reduces to:

• Simplified:

• Can be solved very accurately using Lookup Tables.• For the problem can be solved analytically.

w

minw

¯2

((x ©f j )i ¡ jwj ji )2 + jwj j®i

i j

®= 1=2;2=3;1;2

minw

¯2

(w¡ v)2 + jwj®

w ®

®

x

w

w w

j j

Overall Algorithm• A : Start with a small value of . • Alternate between and sub-problems till convergence.

• Increase the value of by a small amount. GOTO A.

• Achieves over 2 orders of magnitude speedup over IRLS with similar SNR quality. • Timing for de-blurring with a 13 x 13 kernel:

¯x w

¯

Image Size IRLS Ours

1024 x 1024 2.34 1281.3 2.78

3072 x 3072 22.40 > 10000 24.07

l1 ®= 4=5 ®= 2=3

Original

Blurred SNR=7.31

Ours

SNR=18.96t=1.2

IRLS

SNR=19.05t=483.9

®= 2=3®= 4=5

Dark Flash ResultLong Exposure

Ambient2.5 Lux

IRLS ~ 1 hour

Ours ~ 1 minute

®= 2=3®= 2=3

Summary and Drawbacks

• Fast method for nonconvex nonblind deconvolution

• Orders of magnitude faster than IRLS.• Can be used for other applications.– E.g. super-resolution, dark flash, denoising.

• FFTs and LUTs are well suited for hardware implementation.

• Circular boundary conditions cause boundary artefacts – negligible as resolution increases.