Non-blind Image Deconvolution with Adaptive Regularization

Jong-Ho Lee and Yo-Sung Ho

Gwangju Institute of Science and Technology (GIST)

261 Cheomdan-gwagiro, Buk-gu, Gwangju 500-712, Korea {jongho, hoyo}@gist.ac.kr

Abstract. Ringing and noise amplification are the most dominant artifacts in image deconvolution. These artifacts can be reduced by introducing image prior into the deconvolution process. A regularization weighting factor can control strength of the regularization. Ringing and noise can be reduced significantly with the strong weighting factor, but details can be lost. We propose a non-blind image deconvolution method with adaptive regularization that can reduce ringing and noise in the smoothed region and preserve image details in the textured region simultaneously. For adaptive regularization, we make a reference image that gives proper edge information and helps to restore a latent image. The reference image guides strength of weighting factor on the pixel of the blurred image. Experimental results show that ringing and noise are suppressed efficiently, while preserving image details.

Keywords: Non-blind image deconvolution, adaptive regularization.

1 Introduction

One of the most common and unpleasant defects in photography is motion blur caused by camera shake. If a picture is taken in the dim light conditions, it takes long time to get enough light, and the camera shake by hands results in a degraded image. If a motion blur is shift-invariant, recovering a true latent image from the degraded image reduces to the image deconvolution. The blurring is commonly modeled as a convolution of the true latent image and a kernel, or point spread function (PSF), with additive noise:

NKIB +⊗= (1)

where B is a degraded image, I is the true image, K is the kernel, N is additive noise, and ⊗ denotes the convolution operator. Image deconvolution is to reconstruct I from B.

(a) (b) (c)

Fig. 1. Ringing artifacts in image deconvolution. (a) Blurred image and kernel. (b) Richardson-Lucy result. (c) A magnified patch from (b). If both the kernel and the latent image are unknown, the problem is called blind deconvolution. On the other hand, if only latent image is unknown, the problem is called non-blind deconvolution.

In this paper, our target is non-blind deconvolution. In many cases, the kernel can be estimated effectively. Fergus et al. estimated the kernel from a single image [2]. Ben-Ezra and Nayer estimated the kernel using a secondary sensor [1]. Yuan et al. estimated the kernel from a blurred image and a noisy image [3].

However, non-blind deconvolution is a still ill-posed problem although the kernel is known, so it gives rise to artifacts in the deconvolved image. The main artifacts are ringing and noise amplification. Ringing is the ripple-like artifact that appears around strong edges in the deconvolved image as shown in the (c) of Fig. 1. The blurred image loses its high frequency components during blurring process. Kernel is often band-limited, so its frequency response shows zero or near-zero values at the high frequency. Therefore, the direct inverse of the kernel with the blurred image causes large signal amplification at the high frequency components. This is represented as the ringing near the edges and amplified noise. Especially, kernel estimate errors accelerate the ringing artifacts and give very unpleasant deconvolved results [4].

It is useful to include image prior into the deconvolution process to overcome these artifacts. The prior gives the information on the result image, so it helps to reduce ringing and noise while recovering the latent image. However, this regularization technique works well only for relative small kernels. If the kernel size is large, the artifacts are also increased, and strong regularization is needed to reduce these artifacts. The strength of regularization can be controlled with a regularization weighting factor. The large regularization weighting factor can reduce ringing and noise significantly, but also reduce image details. If the regularization weighting factor is decreased to preserve image details, then the artifacts are remained.

We propose a non-blind image deconvolution method with adaptive regularization that can reduce ringing and noise effectively in a locally smooth region and preserve image details in a textured region simultaneously. For adaptive regularization, proper edge information of the latent image is needed, but it is impossible to extract proper edge information from the blurred image. Thus, we make a reference image that gives proper edge information and helps to restore the latent image. The reference image is generated by modified fast image deconvolution using hyper-Laplacian priors [5]. Since the reference image indicates the smoothed region and the textured region

better than the blurred image, regularization can be applied adaptively. The experimental results show that ringing and noise are suppressed efficiently preserving image details.

In the following, we review related work in section 2. Section 3 and section 4 describe two main algorithms, reference image estimation and adaptive regularization respectively. Experimental results are given in section 5, and we conclude this paper in section 5.

2 Related Work

In non-blind deconvolution, the kernel is assumed to be known or estimated in other ways. Thus, the remaining task is to estimate true latent image. Wiener filtering [6] and Richardson-Lucy method [7] are traditional and popular non-blind deconvolution methods. They were proposed decades ago, but are still widely used for image restoration because they are simple, and fast, and gives good results in case of the relative small blur. However, these methods are weak to image noise and kernel error, so they suffer from ringing artifacts. Donatelli et al. use a PDE-based model to recover a latent image with reduced ringing by incorporating an anti-reflective boundary condition and a re-blurring step [8]. Regularization techniques using image prior are also proposed to avoid ringing artifacts. Dey et al. proposed total variation regularization with Laplacian prior [9]. Levein et al. used a sparse prior with heavy-tailed distribution that shows excellent results without ringing artifacts. All these non-blind deconvolution methods are effective when the kernel size is small and the kernel has no error. If the kernel size is large and the estimated kernel is incorrect, then the deconvolved image contains severe ringing artifacts.

In blind deconvolution, the problem is more challenging because the kernel is also unknown. Additional input is used to facilitate the problem in some methods. Yuan et al use a pair of images, a blurry image and a noisy image, to estimate the kernel and deblur the blurry image [3]. Ben-Ezra and Nayar attach a low-resolution video camera to a high-resolution still camera to help in recording the kernel [1]. Recently, the kernel was estimated from a single image. Fergus et al. used a variational Bayesian method with natural image statistics to estimate the kernel. Jia used an alpha matte that describes transparency changes caused by a motion blur for kernel estimation [11]. Shan et al. proposed the deblurring method using a Maximum a Posteriori (MAP) to estimate the kernel and the latent image from a single image.

However, all the above methods fail to preserve image details and reduce artifacts at the same time in the deconvolution because it is difficult to locate proper edge information in the blurred image.

3 Reference Image Estimation

For adaptive regularization, right edge information is needed to locate smoothed region and textured region correctly. Because the blurred image cannot provide this information, we estimate reference image that provide right edge information. We

adopt fast image deconvolution using hyper-Laplacian priors [5] with the model of the spatially random distribution of image noise [4]. Fast image deconvolution using hyper-Laplacian uses an alternating minimization scheme where the non-convex part of the problem is solved in one phase, followed by a quadratic phase which can be efficiently solved in the frequency domain using Fast Fourier Transforms (FFTs). FFTs are efficient tools to recover image very quickly, but causes boundary artifacts. Thus, we first propose the simple algorithm to reduce boundary artifacts.

3.1 Reducing Boundary Artifacts

In the blurring process, the blurred pixels are generated with not only the image inside the Field of View (FOV) of the given observation but also part of the scenery outside the FOV. The part outside the FOV cannot be used in the deconvolution and this missing information causes ringing artifacts around the image boundary when the image is deconvolved using the FFT. Discrete Fourier Transform (DFT) assumes the periodicity of the data, so the missing information will be taken from opposite side of the image when performing FFT. However, since this data taken from the opposite side is not the same as the original data, discontinuities appear along the boundaries, and ringing artifacts are produced around the boundary. This is called boundary artifacts.

The main idea to reduce the boundary artifacts is as follows. We expand the original blurred image such that the intensity and gradient are maintained at the border between the original image and the expanded part. The basic approach to solve the problem is similar to Liu and Jia’s algorithm [12], but our algorithm is faster, and requires lower memory achieving comparable quality.

Fig. 2. Expansion of the image. O: Original image. A, B, C: Padding blocks.

Figure 2 represents the expanded image for reducing boundary artifacts. O is the original image, and A, B, and C are the three padding blocks. Each padding block is constructed such that the periodicity of the image is guaranteed and pixels in the padding block have smooth intensities not to cause ringing artifacts. We first start with the reconstruction method for padding block A.

Let X(i, :) and X(:, j), denote the i-th row and j-th column in a image block X. The size of the original image is MⅹN, and the size of the padding block A is ex_widthⅹN, where ex_width is the expansion width. The first row and the last row of block A is filled as

:) ,(:) ,1( mOA = (2)

:) ,1(:) ,_( OwidthexA = (3)

The other pixels in the block A are padded row by row alternatively. First, the upper line is padded according to:

21

21 ) ,2_() ,1(),(ww

kjiwidthexAwkjiAwjiA+

++−++−= (4)

where w1 and w2 are distances from (i, j) to (ex_width-i+2, j+k) and from (i, j) to (i-1, j+k) respectively, and k∈{-win_size,…, win_size}, and win_size is a window size that controls the smoothness in the horizontal direction. Second, the lower line is padded according to:

43

43 ) ,2_() ,(),(ww

kjiwidthexAwkjiAwjiA+

++−++= (5)

where w3 and w4 are distances from (ex_width-i+1, j) to (ex_width-i+2, j+k) and from (ex_width-i+1, j) to (i, j+k) respectively. This procedure is repeated for i=2 to ex_width/2. Block B is constructed with the same manner. The only difference is the padded direction. The pixels in the block B are padded column by column.

Block C is computed after block A and block B are constructed. First, the most outer pixels of the block C are padded according to:

) (:,)1 (:, NAC = (6)

)1 (:,)_ (:, AwidthexC = (7)

:) ,(): ,1( MBC = (8)

): ,1(): ,_( BwidthexC =

After that, the next outer pixels of the block C are padded using the pixels that are already padded:

(9)

4321

44332211),(wwww

CwCwCwCwjiC++++++

= (10)

where C1, C2, C3, and C4 are reference pixel values from the upper, bottom, left, and right directions respectively. The number of reference pixel values from each Ci is 2win_size+1. The weighting factors w1, w2, w3, and w4 are defined according to the distance from the current pixel location to each direction. This procedure is repeated until all the other pixels inside the C block are padded.

Figure 3 shows the expanded blurred image using above algorithm, and deconvolved images without and with our reducing boundary artifacts (RBA) algorithm respectively. Richardson-Lucy method is used to test the performance of our algorithm. The figure shows that the boundary artifacts are reduced significantly.

(a) (b) (c)

Fig. 3. Reducing boundary artifacts. (a) Expanded blurred image. (b) Richardson-Lucy result without RBA algorithm. (c) Richardson-Lucy result with RBA algorithm.

3.2 Reference Image Estimation

The reference image is estimated by combination of fast image deconvolution using hyper-Laplacian priors [5] and the model of the spatially random distribution of image noise [4]. According to the fast image deconvolution using hyper-Laplacian priors, the latent image can be estimated very quickly with following equations.

)||||

)||)(||||)((||2

)(2

(

)|)(|)(2

(*

211

22

22

22

11

2

,

1

2

1

2

αα

α

βλ

λ

ii

N

iiiiii

wI

N

i jiji

I

ww

wfIwfIBKImin

fIBKIminI

++

−⊗+−⊗+−⊗=

⊗+−⊗=

∑

∑ ∑

=

= =

(11)

where I*is the estimated latent image, i is the pixel index, f1 and f2 are two first order derivative filters, f1=[1 -1], f2=[1 -1]T, β is a weight that will be varied during the optimization, wi

1 and wi2(together denoted as w) are auxiliary variables that allow the

term (I⊗ fj )i to be moved outside the |.|α expression. The model of the spatially random distribution of image noise states that noise, N=B-I⊗P, follows Gaussian distribution and its higher-order partial derivatives also follow Gaussian distribution with different standard distributions as follows [4].

),)(*|*()|( 2

*qi

ii KIBNIBp σ⊗∂∂=∏∏

Θ∈∂

(12)

where ∂* denotes the operator of any partial derivative with k(∂*)=q representing its order. ∂*N=∂*(B-I ⊗ P) follows a Gaussian distributions with standard deviation

σq= q2 σ0, where σ0 denotes the standard deviation of N. Θ = {∂0, ∂x, ∂y, ∂xx, ∂xy, ∂yy} represents a set of partial derivative operators [4].

The reference image is generated by combining (11) and (12) as follows.

)||||)||)(||||)((||2

)||)**(||(2

(*

21222

211

1

2

**)(

,

ααβ

τλ

iiiiii

N

iik

wI

wwwfIwfI

BKIminI

++−⊗+−⊗+

∂−⊗∂= ∑ ∑= Θ∈∂

∂

(13)

where τk(∂*)=1, 0.5, and 0.25 when k(∂*)=0, 1, and 2 respectively. The combination of fast image deconvolution using hyper-Laplacian priors and the model of the spatially random distribution of image noise provides a powerful tool to estimate reference image. To solve (13), w={w1, w2} values are calculated first with the method introduced in [5], and the reference image I* is estimated with (14) given a fixed value of w.

}

})b)(b)(b)((25.0

)b)(b)((5.0b)(

i ))()()((25.0

))()((5.0)(

121222221111

22112211

121222221111

22112211

FFCFFFFCFFFFCFF

FCFFCFCwFwF

CFFCFFCFFCFFCFFCFF

CFCFCFCFKKFFFF

TK

TK

TK

TK

TK

TK

TT

KT

KKT

KKT

K

KT

KKT

KTTT

+++

⎩⎨⎧ ++++=

+++

⎩⎨⎧ ++++

λβ

λβ

(14)

where i and b are the vector forms of I and B respectively, Fj i=fj⊗ I, and CK i=K⊗ I. The above equation can be calculated using 2D FFT very quickly assuming

circular boundary conditions. We applied the algorithm proposed in 3.1 to reduce the boundary artifacts caused by FFT calculation. Figure 4 shows the synthesized blurred image, estimated kernel and the estimated reference image. The kernel is estimated by Fergus et al.’s algorithm [2] and λ=2ⅹ10 and α=2/3 are used to estimate the reference image

(a) (b) (c)

Fig. 4. Reference image estimation. (a) Blurred image. (b) Estimated Kernel. (c) Estimated reference image.

4 Adaptive Regularization

The reference image indicates smoothed region and textured region better than the blurred image, but it still has small ringing artifacts and its image details are over smoothed. Thus, we regularize the image with regularization weighting factor that is changed according to the region based on the edge information from the reference image.

At first, a shock filter is used to restore edges in the reference image. A shock filter is an effective filter to recover sharp edges from blurred step signals [13]. The evolution equation of a shock filter is as follows.

dtIIsignII tttt ||||)( 21 ∇∇−=+ (15)

where It is an image at time t, and 2∇ and ∇ are the Laplacian and gradient, respectively. dt is the time step for a single evolution. After restoring edges with the shock filter, the edge information is extracted from the shock-filtered image. A 3ⅹ3 window is centered on a pixel in the shock-filtered image, and edge strength on the pixel is calculated as follows.

n

jiWjiW

jiEg ji jiyx∑ ∑+

= , ,

),(),(

),( (16)

where Eg(i,j) is the edge strength at the pixel location, (i,j), Wx=W ⊗ [1 -1], Wy=W⊗ [1 -1]T, W is the 3ⅹ3 window, and n is the total number of pixels of the window. The noise in Eg is removed by thresholding. The threshold value is 5% of maximum of Eg. Based on this edge information of the reference image, we formulate the deconvolution problem as follows.

According to Bayes framework, the posteriori for the latent image is written as:

)()|()|( IpIBpBIp ∝ (17)

where p(B|I) denotes the likelihood of the blurred image given the latent image, and p(I) represents the image prior. The maximum a posteriori (MAP) solution of I can be obtained by minimizing the following energy:

)(* IEargminII

= (18)

where

)( )|( )|( )( IplogIBplogBIplogIE −−=−= (19)

The likelihood is based on noise, N= B-I⊗K. For this likelihood, we adopt the model of the spatially random distribution of image noise [4] that is described in (12). We also adopt the sparse distribution as the image prior [10]:

)|||(|()( 8.02

8.01 fIfIexpIp ⊗+⊗−= α (20)

where f1=[1 -1] and f2=[1 -1]T.

By taking the likelihood and prior into (21), we get

)|||(|||)**(||)( 8.02

8.01

2

**)( fIfIBKIIE ik ⊗+⊗+∂−⊗∂= ∑

Θ∈∂∂ ητ (21)

where η=2σ02α. Here, η is the regularization weighting factor that controls the strength

of regularization and we use η=0.003 for the experiments. For adaptive regularization, we change the above equation to:

)|||)(|||( ||)**(||)( 8.02

8.01

22

**)( fIfIEgexpBKIIE

eik ⊗+⊗−+∂−⊗∂= ∑

Θ∈∂∂ σ

ητ (22)

where Eg is the edge strength on the pixel of the reference image calculated with (16), and σe is the constant determining the shape of Gaussian function. We use σe=1ⅹ103 by default. Thus, the effect of the regularization weighting factor, η, on the pixel is controlled according to )/||( 2

eEgexp σ− . To solve this equation, iterative re-

weighted least squares (IRLS) algorithm is used. IRLS poses the optimization as a sequence of least squares problems, while each least square problem re-weighted by solution at the previous step [10]. Ringing artifacts are suppressed more and image details are preserved better in the adaptively regularized image than the reference image. Furthermore, we can obtain better results by setting this regularized image as the second reference image. The second reference image is shock-filtered, edge information is extracted, and adaptive regularization is applied to the blurred image again. This procedure can be repeated for better results, but we confirmed that two steps are enough for the good result.

Although the adaptively regularized image shows well preserved edges and reduced ringing artifacts, its fine scale detail layer is suppressed. This detail layer is enhanced as follows. If we define the adaptively regularized image as Ia, then the fine scale detail layer is obtained by subtraction the bilateral filtered Ia from Ia. Bilateral filter is defined as:

∑∈

−−=)('

)'())'()(()'(1))((xWx

dx

xIxIxIGrxxGZ

xIF (23)

where Gd and Gr are Gaussian function, W(x) is a neighboring window and Zx is a normalization term. This detail layer, Ia-F(Ia), is added to the adaptively regularized

(a) (b) (c)

Fig. 5. (a) Shock-filtered image. (b) )/||( 2

eEgexp σ− map. (c) Final deconvloved image.

image to obtain the final deconvolved image.

The shock-filtered image at the first step, )/||( 2eEgexp σ− map on the second

step, and the adaptively regularized image at the second step are represented in Fig. 5.

5 Experimental Results

We applied our non-blind image deconvolution with adaptive regularization algorithm to various images. The kernels of all these images are estimated by Fergus et al’s single image method [2]. We compare our algorithm with the standard Richardson-Lucy method [7], Total variation (TV) regularization [9] and Levin et al’s method [10]. For the fare comparison, we set the regularization parameters of all algorithms such that the best results come out.

In figure 6, the estimated kernel size is 37ⅹ37, and the size of blurred image is 664ⅹ489. Richardson-Lucy method produces the most noticeable ringing. The Levin et al’s algorithm reduces ringing significantly, but image detail is also suppressed. The result of our algorithm suppresses ringing artifacts while preserving image details.

In figure 7, and figure 8, the estimated kernel sizes are 37ⅹ37 and 35ⅹ35 respectively. Comparing with other methods, our method shows the best quality considering image details and artifacts simultaneously.

Fig. 6. (a) Blurred image and estimated kernel (b) Richardson-Lucy method. (c) Levin et al’s method. (d) Our method. (e) Close-up views of (b)-(d).

(a) (b)

(c) (d) (e)

Fig. 7. (a) Blurred image and estimated kernel. (b) Richardson-Lucy method. (c) TV regularization. (d) Our method. (e) Close-up views of (b)-(d).

Fig. 8. (a) Blurred image and estimated kernel. (b) TV regularization. (c) Levin et al’s method. (d) Our method. (e) Close-up views of (b)-(d).

(a) (b)

(c) (d) (e)

(a) (b)

(c) (d) (e)

6 Conclusion

The most notorious artifacts at image deconvolution are ringing artifacts and noise amplification. This problem is solved using the image prior that represents statistics of the image. However, strong regularization for reducing artifacts at image deconvolution does not preserve image details. In this paper, we have presented the adaptive regularization method with help of the reference image that separates the smoothed region from the textured region. The experimental results show that our approach suppresses ringing artifacts effectively, while preserving image details.

Acknowledgments. This research was supported by the MKE(The Ministry of Knowledge Economy), Korea, under the ITRC(Information Technology Research Center) support program supervised by the NIPA(National IT Industry Promotion Agency) (NIPA-2010-(C1090-1011-0003)).

References

1. Ben-Ezra, M.R., Nayar, S.K.: Motion Deblurring using Hybrid Imaging. In: Proceedings of Computer Vision and Pattern Recognition, vol. 1, pp. 657-664 (2003)

2. Fergus, R., Singh, B., Hertzmann, A., Roweis, S.T., Freeman, W.T.: Removing Camera Shake from a Single Photograph. ACM Trans. on Graph. (SIGGRAPH), vol. 25, pp. 787-794 (2006)

3. Yuan, L., Sun, J., Quan, L., Shum, H.Y.: Image Deblurring with Blurred/Noisy Image Pairs. ACM Trans. on Graph. (SIGGRAPH), vol. 26, pp. 1-10 (2007)

4. Shan, Q., Jia, J.Y., Agarwala, A.: High-quality Motion Deblurring from a Single Image. ACM Trans. on Graph. (SIGGRAPH), vol. 27 (2008)

5. Krishnan, D., Fergus, R.: Fast Image Deconvolution using Hyper-Laplacian Priors. Advances in Neural Information Processing Systems, vol. 22, pp 1-9 (2009)

6. Wiener, N.: Extrapolation, Interpolation, and Smoothing of Stationary Time Series, MIT Press (1964)

7. Lucy, L.: An Iterative Technique for the Rectification of Observed Distributions, vol. 79, pp. 745-754 (1974)

8. Donatelli, M., Estatico, C., Martinelli, A., Serra-Capizzano, S.: Improved Image Deblurring with Anti-reflective Boundary Conditions and Re-blurring. Inverse Problems, vol. 22, pp. 2035-2053 (2006)

9. Dey, N., Blanc-Fraud, L., Zimmer, C., Kam, Z., Roux, P., Olivo-Marin, J. Zerubia, J.: Richardson-Lucy Algorithm with Total Variation Regularization for 3D Confocal Microscope Deconvolution. Microscopy Research Technique, vol. 69, pp. 260-266 (2006)

10. Levin, A., Fergus, R. Durand, F., Freeman, W.T.: Image and Depth from a Conventional Camera with a Coded Aperture. ACM Trans. on Graph. (SIGGRAPH), vol. 26, pp. 70-77 (2007)

11. Jia, J.: Single Image Motion Deblurring using Transparency. In: Proceedings of Computer Vision and Pattern Recognition, pp. 1-8 (2007)

12. Liu, R., Jia, J.: Reducing Boundary Artifacts in Image Deconvolution. IEEE International Conference on Image Processing (2008)

13. Osher, S., Rudin, L.I.: Feature-oriented Image Enhancement using Shock Filters. SIAM Journal on Numerical Analysis, vol. 27, pp. 919-940 (1990)