Markov Chain Monte Carlo Super-resolution ImageReconstruction With Artifacts Suppression

Jing Tian and Kai-Kuang Ma*School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore 639798

* Email: ekkma@ntu.edu.sg

Abstract— Recently, the Markov chain Monte Carlo (MCMC) tech-nique has been proved as an effective approach to address thesuper-resolution image reconstruction problem. However, this approachusually requires a substantial amount of time to generate a sufficientlylarge number of samples for estimating the unknown high-resolutionimage. Limiting the simulation time could possibly lead to someartifacts presented in the reconstructed high-resolution image. Toeffectively mitigate these artifacts, an outlier-sensitive bilateral filteringis proposed in this paper, which contains a switching mechanism steeredby an outliers detection scheme. Only for those pixel positions thathave been identified containing outliers, our proposed bilateral filteringwill be applied; for the rest, the conventional bilateral filtering willbe exploited. Experimental results are presented to demonstrate thesuperior performance of the proposed method.

Keywords—super-resolution, Markov chain Monte Carlo

I. INTRODUCTION

The aim of super-resolution (SR) image reconstruction is tofuse a set of low-resolution images of the same scene to produce asingle higher-resolution image with more details. The SR problemhas enabled active research due to its potential in overcoming theinherent limitations of existing image acquisition systems withoutcalling for sophisticated hardware [1]–[3]. Motivated by the factthat the SR problem, in essence, is an ill-posed inverse problem [4],[5], the Markov chain Monte Carlo (MCMC) technique, togetherwith the Bayesian inference approach, was introduced in [6] andhas demonstrated attractive performance on addressing the SRproblem. Given the observed low-resolution images, the MCMC SRalgorithm produces a high-resolution image through the utilizationof the samples generated by the MCMC process according to theposterior probability distribution of the unknown high-resolutionimage.

However, one important practical issue of the above-mentionedMCMC SR process is that the MCMC-based approach alwaysrequires substantial amount of time to generate a sufficiently largenumber of samples for estimating the unknown high-resolutionimage. By limiting the simulation time (equivalently, the number ofsamples generated for the MCMC process), it is possible to resultin some artifacts in the reconstructed high-resolution image. Totackle this problem, our aim is to blend in a post-processing stagefor enhancing the reconstructed high-resolution image, especiallywhen the number of the samples generated is insufficient. Forthat, the bilateral filtering method [7] has been speculated as aneffective approach in mitigating image artifacts while preservingthe edge information [8]. In this paper, the efficiency and perfor-mance improvement contributed by the post-processing approachis investigated. Furthermore, an outlier-sensitive bilateral filteringmethod is proposed to reduce the artifacts presented in the high-resolution image produced by the MCMC SR process.

The paper is organized as follows. In Section II, a Bayesianformulation of the SR problem is briefly introduced and theMCMC SR algorithm is described. The outlier-sensitive bilateral

filtering method is proposed in Section III. Experimental results arepresented in Section IV. Finally, Section V concludes this paper.

II. THE MCMC SUPER-RESOLUTION APPROACH

A. Observation Model

Given an original image, it is considered as the high-resolutionground truth, so that it can be compared with the reconstructedimage for performance evaluation. To simulate the observedlow-resolution observations, the original high-resolution image iswarped, convolved with a point spread function (PSF), downsam-pled to a lower resolution, and finally added with a zero-meanGaussian noise. These operations can be formulated as

y(k) = H(k)X + V(k), (1)

where y(k) and X represent the k-th low-resolution image and thehigh-resolution image represented in the lexicographic-ordered vec-tor form, with a size of L1L2×1 and M1M2×1, respectively. H(k)

is an L1L2 × M1M2 matrix, representing the above-mentionedwarping (e.g., shift and rotation), convolving and downsamplingprocess, and V(k) is an L1L2 × 1 vector, representing the additivenoise, which is generated from a Gaussian distribution with zeromean and variance σ2

v .With such establishment, the goal of the SR image reconstruc-

tion is to produce a single high-resolution image X based on afew (say, ρ) low-resolution observations, which are denoted asy(1),y(2), . . . ,y(ρ). To simplify the notation, we define Y =y(1),y(2), . . . ,y(ρ).

B. The MCMC Super-resolution Approach

In [6], the MCMC technique was first used to estimate the un-known high-resolution image from a set of observed low-resolutionimages.

Firstly, assume that the low-resolution images are obtainedindependently from the high-resolution image. According to (1),we have

p (Y|X) =

ρ∏k=1

p(y(k)|X

)

∝ exp

(− 1

2σ2v

ρ∑k=1

∥∥∥y(k) − H(k)X∥∥∥2

). (2)

On the other hand, the prior model of the image usually reflectsthe expectation that the image is locally smooth. Markov randomfield (MRF) [9] has been widely used to model local smoothness inimages and will be considered in this paper. The γ-neighbourhoodMRF model used in our works yields the form as

p(X|λ) =1

Z(λ)exp

−1

2λXT CX

, (3)

940

1-4244-0387-1/06/$20.00 c©2006 IEEE

where λ is the hyperparameter and C is an M ×M matrix, whoseentries are given by

C(i, j) =

γ, i = j;−1, i and j are adjacent in the γ-neighbourhood;0, otherwise.

(4)The partition function Z(λ) is a normalizing factor that is definedas Z(λ) =

∫χ

exp −λU(X) dX, where the energy functionU(X) is made of U(X) =

∑c∈C Vc(X), in which Vc(X) is the

potential function of a local group of pixels c (called the clique)and its value depends on the local configuration on the clique c.

Now, we present how to simultaneously find the estimationof the high-resolution image (denoted as X) as well as the hy-perparameter (denoted as λ), by maximizing their joint posteriorprobability density function (pdf) p (X, λ|Y). To compute theabove joint posterior pdf, we firstly apply the Bayes rule to rewritep (X, λ|Y) as

p (X, λ|Y) =p (X, λ,Y)

p (Y)∝ p (X, λ,Y) (5)

= p (Y|X, λ) p (X, λ) = p (Y|X) p (X|λ) p(λ).

By substituting (2) and (3) into (5), we have

p (X, λ|Y) (6)

∝ 1

Z(λ)exp

− 1

2σ2v

ρ∑k=1

∥∥∥y(k) − H(k)X∥∥∥2

− 1

2λXT CX

p(λ).

In (6), the uniform distribution is chosen for p(λ) as the prior pdfof the hyperparameter λ; that is, p(λ) = 1

λmax−λmin, for λ ∈

[λmin, λmax], where [λmin, λmax] represents the dynamic rangeof λ. However, the direct assessment of (6) is difficult due tothe functional dependence of the partition function (i.e., Z(λ)) onthe hyperparameter λ. The computation of this partition functionis computationally intractable, since it needs to integrate over allpossible values of X. Note that the size of the space χ is NM1×M2

for the N -bit gray-level image with the size of M1 × M2. Hence,we resort to the MCMC method, which provides an efficient wayto address this problem.

The goal of the MCMC SR algorithm is to generateN samples from p(X, λ|Y); these samples are denoted asZ(1),Z(2), . . . ,Z(N) (each of which has the same size as that of X)and λ(1), λ(2), . . . , λ(N), where N is large enough to guarantee theconvergence of the MCMC method. Then all the samples generatedbefore the convergence (i.e., up to the sample number T ) areconsidered unreliable and should be discarded. Then X is obtainedby computing the mean of the rest (N − T ) samples; that is,

X =1

N − T

N∑i=T+1

Z(i), (7)

in which the parameter T is derived in [6]

T =ln

(ε

M(θmax−θmin)

)ln

(1 − 1

M

ρH2min

ρH2min+λmaxγσ2

v

) , (8)

where ε is the user-defined tolerance, θmin and θmax indicate thedynamic range of the image gray-level, M is the dimension of thehigh-resolution image, λmax and γ are the parameters used in theMRF, ρ is the number of low-resolution observations, σ2

v is the vari-ance of the additive white noise and Hmin represents the minimumvalue of all non-zero entries of the matrices H(1),H(2), . . . ,H(ρ).

III. THE POST-PROCESSING METHOD — THE PROPOSED

OUTLIER-SENSITIVE BILATERAL FILTERING

A. Bilateral Filtering

In this section, we provide a brief introduction to the bilateralfiltering [7]. The principle of the bilateral filtering method is toreplace each pixel by a weighted average of the pixels in itsneighborhood. The weighting function is based on the spatialdistance measured from the center pixel, called the domain filter,as well as the distance in the intensity, called the range filter.

More specifically, let u be the location of the pixel underconsideration, and let Ω = Ωu(M) be the pixels in a neighborhood(with a size of (2M + 1)× (2M + 1)) of u. The bilateral filteringreplaces the pixel value at u (denoted as f(u)) by

h(u) =

∫v∈Ω

f(v)c(v,u)s (f(v), f(u)) dv∫v∈Ω

c(v,u)s (f(v), f(u)) dv, (9)

where f(v) is the pixel value of the pixel v, c(v,u) ands (f(v), f(u)) are the domain filter and the range filter, re-spectively. These two filters are commonly modeled as Gaussianfunctions and yield the forms as

c(v,u) = exp

−|v − u|2

2σ2d

, (10)

s (f(v), f(u)) = exp

−|f(v) − f(u)|2

2σ2r

, (11)

where the parameters σ2d and σ2

r are the variances of the respectivefunctions.

The output of the domain filter (i.e., (10)) decreases as the spatialdistance between the pixels u and v increases, such that the blurringcan be effectively reduced, since for those pixels that are far awayfrom u should have lower influence on u. On the other hand, theoutput of the range filter (i.e., (11)) decreases as the differencebetween the pixel values of u and v increases, so that the edgesare better kept, since the pixels with significantly different valueswith that of u should have lower influence on u.

B. Outlier-sensitive Bilateral Filtering — The Proposed Method

In this section, a modified bilateral filtering method, called theoutlier-sensitive bilateral filtering, is proposed to reduce the SRartifacts, which are caused by insufficient number of samples beingused for the MCMC SR process as described in Section II-B.

Although the conventional bilateral filtering method can smooththe image while well preserving the edge information, it can noteffectively remove the outliers from the image. This is due to thefact that the range filter output will be small, if the reference pixelused in the range filter (i.e., f(u) in (11)) is an outlier, whichusually has much larger or smaller pixel value than its neighbouringpixels values. Consequently, the outliers tend to be kept in thefiltered image and fail to be removed.

To overcome this problem, the basic idea is to replace anyunreliable reference pixel used in the range filter by a reliableestimator, so that the modified filter can be adaptively changedaccording to the local statistics of the samples. The statistics usedin the paper is the variance of generated samples at each pixelposition throughout the estimation of the high-resolution image,since the estimates resulted from insufficient number of samplesusually yield high individual variance [10]. This local statistics isdetermined as follows.

Let u be the location of the pixel under consideration, andsuppose that the high-resolution image is estimated by averaging

APCCAS 2006 941

a set of (N − T ) samples according to (7). Then, the estimatorvariance computed at the position u can be obtained from

τ(u) =1

N − T

N∑i=T+1

(f (i)(u) − 1

N − T

N∑i=T+1

f (i)(u)

)2

,

(12)where f (i)(u) represents the pixel value of pixel u in the i-th imagesample Z(i).

If the estimator variance computed at the position u (defined in(12)) is the largest in its neighborhood Ω, then the pixel valueat this position is treated as unreliable; that is, it is estimatedusing insufficient number of samples. This unreliable pixel valuewill be replaced by a more reliable estimator — the median valuecomputed based on the neighborhood of pixel u, denoted as g(u).The estimator g(u) is then incorporated into the range filter ofthe conventional bilateral filtering method to arrive at an improvedfilter, as follows,

h(u) =

∫v∈Ω

f(v)c(v,u)s (f(v), g(u)) dv∫v∈Ω

c(v,u)s (f(v), g(u)) dv, (13)

where g(u) is determined by

g(u) = median f(v)v∈Ω . (14)

The proposed outlier-sensitive bilateral filtering method is ableto adaptively switch between the conventional bilateral filtering asin (9) and our developed filter as in (13). The switching mechanismis steered by the estimator variance computed at each pixel position,which is determined by (12). More specifically, the filter definedin (13) is applied at the outlier positions, where the pixels yieldthe largest estimator variance in their respective neighborhood Ω,while the conventional bilateral filtering is applied at the restpixel positions. By this way, the proposed outlier-sensitive bilateralfiltering can adaptively adjust according to the local statistics ofthe reconstructed high-resolution image, so that it can effectivelyremove the outliers artifacts, while smoothing the image withoutsacrificing the edges.

IV. EXPERIMENTAL RESULTS

In this section, we provide some experimental results to explorethe performance of the proposed post-processing method, usingthree test images that are illustrated in Figure 1. The low-resolutionimages are generated from these original images, respectively,through a series of operations as follows. We first apply a shift,which is drawn from a continuous uniform distribution over theinterval (−2, 2) in the units of pixels, and then apply a rotation,which is drawn from a continuous uniform distribution over theinterval [−4, 4] in the units of degrees. Next, the processed imagesare convoluted with the Gaussian low-pass filter with a size of4 × 4 and the standard derivation of 2, followed by a down-sampling operation with a factor of two in both the horizontal andthe vertical directions. Lastly, the processed image is added with azero-mean Gaussian noise such that the signal-to-noise ratio (SNR)equals to 20 dB. Here the SNR is defined as 10 × log10(σ

2s/σ2

v),where σ2

s and σ2v are the variances of the noise-free image and

the additive white noise, respectively. In our simulation, four low-resolution images are independently generated by applying theabove-mentioned operations on each test image, respectively.

The MCMC SR algorithm produces a high-resolution image asfollows. After the first sample is randomly initialized, the MCMCtechnique is applied to generate N samples. Besides, the minimumand maximum values of the hyperparameter λ (i.e., λmin and λmax

(a) (b) (c)

Fig. 1. The test images used in the experiments: (a) Boat; (b) Lena; and(c) Text.

defined in the prior pdf of λ) are set to be 1 and 50, respectively.Then, the first T samples are removed and the rest (N−T ) samplesare used to produce the high-resolution image via (7).

To test the proposed post-processing method and compare itsperformance with that of the conventional bilateral filtering method[7], we apply methods to enhance the high-resolution image thatis estimated by the MCMC SR algorithm, respectively. All theparameters (i.e., σ2

d and σ2r ) used in the above two methods are

experimentally selected. Table I compares their PSNR performance.For subjective performance evaluation, Figure 2 presents the resultsof applying the above two methods on the reconstructed high-resolution Text image using N = 3×T and N = 10×T samples,respectively. As can be seen from Table I and Figure 2, the proposedoutlier-sensitive bilateral filtering method constantly yields the bestperformance, particularly in the case when a smaller number ofsamples are generated for the MCMC SR process.

V. CONCLUSIONS

The MCMC technique was successfully introduced for address-ing the SR problem in [6]. However, it is possible to result in someartifacts in the reconstructed high-resolution image, if the numberof samples generated for the MCMC SR process is insufficient. Toremedy this drawback, an outlier-sensitive bilateral filtering methodis proposed in this paper to effectively reduce the artifacts presentedin the reconstructed high-resolution image. The proposed methodachieves better performance than the conventional bilateral filteringmethod, as evidenced by our experimental results.

REFERENCES

[1] S. Chaudhuri, Super-Resolution Imaging. Boston: Kluwer AcademicPublishers, 2001.

[2] M. G. Kang and S. Chaudhuri, “Super-resolution image reconstruc-tion,” IEEE Signal Processing Mag., vol. 20, pp. 19–20, Mar. 2003.

[3] N. K. Bose, R. H. Chan, and M. K. Ng, Special Issue on HighResolution Image Reconstruction, International Journal of ImagingSystems and Technology, vol. 14, no. 2-3, Aug. 2004.

[4] R. R. Schultz and R. L. Stevenson, “Extraction of high-resolutionframes from video sequences,” IEEE Trans. Image Processing, vol. 5,pp. 996–1011, Jun. 1996.

[5] R. C. Hardie, K. J. Barnard, and E. E. Armstrong, “Joint MAPregistration and high-resolution image estimation using a sequenceof undersampled images,” IEEE Trans. Image Processing, vol. 6, pp.1621–1633, Dec. 1997.

[6] J. Tian and K.-K. Ma, “A MCMC approach for Bayesian super-resolution image reconstruction,” in Proc. IEEE Int. Conf. on ImageProcessing. Genoa, Italy, 2005, pp. 45–48.

[7] C. Tomasi and R. Manduchi, “Bilateral filtering for gray and colorimages,” in Proc. IEEE Int. Conf. on Computer Vision. Bombay,India, Jan. 1998, pp. 839–846.

[8] R. Xu and S. N. Pattanaik, “A novel Monte Carlo noise reductionoperator,” IEEE Computer Graphics and Applications, vol. 25, pp.31–35, March-April 2005.

[9] S. Z. Li, Markov Random Field Modeling in Computer Vision. NewYork: Springer-Verlag, 1995.

[10] M. D. McCool, “Anisotropic diffusion for Monte Carlo noise reduc-tion,” ACM Trans. Graphics, vol. 18, pp. 171–194, Apr. 1999.

942 APCCAS 2006

TABLE ITHE PSNR PERFORMANCE COMPARISON.

Test imageBi-cubic spline Number of samples

MCMC SRMCMC SR + conventional MCMC SR + proposed

generated for the outlier-sensitive bilateralinterpolation MCMC SR bilateral filtering [7] filtering

Boat 24.00 dB

N = 3 × T 25.32 dB 26.27 dB 26.92 dBN = 6 × T 27.10 dB 27.35 dB 27.76 dBN = 10 × T 28.02 dB 28.12 dB 28.21 dB

Lena 23.33 dB

N = 3 × T 24.56 dB 25.73 dB 26.51 dBN = 6 × T 26.75 dB 27.05 dB 27.37 dBN = 10 × T 27.35 dB 27.41 dB 27.56 dB

Text 15.61 dB

N = 3 × T 17.53 dB 18.11 dB 18.92 dBN = 6 × T 18.95 dB 19.37 dB 19.82 dBN = 10 × T 19.58 dB 19.83 dB 20.12 dB

(a) (b) (c)

(d) (e) (f)

(g) (h) (i)

Fig. 2. Various reconstructed results of Text image: (a) the original (or ground-truth) image; (b) a synthesized low-resolution image (with pixel duplicationfor presentation); (c) the bi-cubic spline interpolation approach; (d) the MCMC SR algorithm using N = 3×T samples; (e) applying the bilateral filteringmethod [7] on (d); (f) applying the proposed outlier-sensitive bilateral filtering method on (d); (g) the MCMC SR algorithm using N = 10 × T samples;(h) applying the bilateral filtering method [7] on (g); and (i) applying the proposed outlier-sensitive bilateral filtering method on (g).

APCCAS 2006 943