Inverse Halftoning via Nonlocal Regularization

Xin Li

West Virginia University

This work is partially supported by NSF CCF-0914353

What is Inverse Halftoning?

X: continuous-tone original

Y: halftoned (B/W)

halftoning

inversehalftoning

^X: continuous-tone estimated

Evolutionary Path of Inverse Halftoning

Inverse Halftoning

Inverse Problems

ImageRestoration

Regularizationtheory

ImagePrior

What is State-of-the-ArtMethod PSNR(dB)

Inverse halftoning and kernel estimation for error diffusion (PW Wong TIP’1996)

31dB (MMSE projection)32dB (MAP projection)

Inverse halftoning using wavelets (Xiong et al., TIP’1997)

31.67dB

Look-up table (LUT) method for inverse halftoning (Mese and Vaidyanathan TIP’2001)

31.50dB

A fast, high-quality inverse halftoning algorithm for error diffused halftones (Kite et al. TIP’2000)

31.30dB

Hybrid LMS-MMSE inverse halftoning technique(Chang et al. TIP’2001)

31.39dB

A Tantalizing Hypothesis

Wavelet-based(Xiong et al.)

LUT-based(Mese et al.)

Iterative filtering-based(Wong)

Hybrid LMS-MMSE(Chang et al.)

Gradient-based(Kite et al.)

Are they fundamentally equivalent? – all based on the local models (singularities in images are characterized by local intensity variations).

Hierarchy of Mathematical SpacesHilbert-space: a completeInner-product space

Quantum mechanics

Fourier/waveletanalysis

Learning theory

PDE(e.g., Total-Variation)

Mathematical formalism(Hilbert, Ackermann, Von Neumann …)

Metric space: a set witha notion of distance

General relativity

Fixed-point theorems

Game theory

Dynamic systems

Mathematical constructivism(Poincare, Brouwer, Weyl …)

Filtering as Projection

• Examples– Linear filtering (low-pass vs. high-pass)– Nonlinear filtering/diffusion – Bilateral filtering– Wavelet/DCT shrinkage– Nonlocal filtering (BM3D, nonlocal TV)

'')',';,(

'')','()',';,(

dydxyxyxw

dydxyxfyxyxwfNLFP

“Phase Space” of Image SignalsSA-DCT TV BM3D Nonlocal-TV

Local filters Nonlocal filters

Alternating Projections

X0

X1

X2

X∞

Projection-Onto-Convex-Set (POCS) Theorem: If C1,…,Ck areconvex sets, then alternating projection P1,…,Pk will convergeto the intersection of C1,…,Ck if it is not empty

C1

C2

C1 : observation constraint set

C2 : regularization constraint set

Graduated Nonconvexity (GNC)

temperature of deterministic annealing threshold or Lagrangian multiplier

Summary of Algorithm

Key messages:

1.From local to nonlocal regularization thanks to the fixed-point formulation in the metric space (PNLF depends on the clustering result or similarity matrix)2.From convex to nonconvex optimization: deterministic annealing (also-called graduated nonconvexity) is the ``black magic” behind

Experimental Results

MATLAB codes accompanying this work are availableFrom my homepage: http://www.csee.wvu.edu/~xinl/

“o” – lena“+” – peppers

Image Comparison Results (I)

This work(PSNR=32.90

dB)

wavelet-based (PSNR=31.95dB)

TV-based (PSNR=30.91dB)

This work(PSNR=32.64

dB)

wavelet-based (PSNR=31.03dB)

TV-based (PSNR=30.92dB)

Beyond Inverse Halftoning• Image denoising

– W. Dong, X. Li, L. Zhang and G. Shi, "Sparsity-based image denoising via dictionary learning and structural clustering" , CVPR'2011 (oral paper), June 2011

• Image deblurring– Xin Li , "Fine-Granularity and Spatially-Adaptive Regularization for Projection-based Image

Deblurring,"IEEE Trans. on Image Processing, Vol. 20, No. 4., pp. 971-983, Apr. 2011.– Weisheng Dong, Xin Li, Lei Zhang, and Guangming Shi, “Sparsity-based image deblurring with

locally adaptive and nonlocally robust regularization,” accept to Proc. IEEE International Conference on Image Processing (ICIP), 2011

• Image coding– X. Li, "Collective sensing: a fixed-point approach in the metric space," SPIE Conf. on Visual

Comm. and Image. Proc. (VCIP), Jul. 2010• Super-resolution

– Weisheng Dong, Guangming Shi, Lei Zhang, and Xiaolin Wu, “Super-resolution with nonlocal regularized sparse representation,” in Proc. SPIE Visual Communications and Image Processing (VCIP), July 2010

• Compressed sensing– X. Li, “The magic of nonloca Perona-Malik diffusion”, IEEE Signal Processing Letter, vol. 18, no.

9, pp. 533-534, Sep. 2011

Source code collection for reproducible research http://www.csee.wvu.edu/~xinl/source.html