SINGLE IMAGE SUPER-RESOLUTION USINGDEFORMABLE PATCHESYu Zhu1, Yanning Zhang1, Alan L. Yuille2

1School of Computer Science, Northwestern Polytechnical University, China2Department of Statistics, UCLA, USA

MOTIVATION

The dictionary may not containthe desired HR patch exactly.

A basic patch can be deformedto a potential patch to fit the inputLR patch.

The dictionary could expressmore patterns using finite basicpatches via deformation.

CONTRIBUTION

Propose a deformable patches model for single image SR problem,making the dictionary more expressive.

Develop an effective patch matching strategy to select the best ba-sic patch for deformation.

Extend the model of single patch to a weighted combination of sev-eral deformed candidates for more reliable HR estimation.

DEFORMATION MODELDeformation of Basic Patch:

Br = αφ(Bh) + β

Taylor Expansion:φ(Bh) ≈ Bh(x+ u,y + v)

= Bh + diag(Bhx)u+ diag(Bhy)v

Energy Function:E(u,v) = ||DHBr − Pl||2 + ψ(u,v)

METHODDeformation Similarity for Patch Matching:

Bh = argminBh

P>d Pd − P>d G(G>G+ Γ)−1G>Pd

Optimizing for Energy Function:

E(M) = ||Pd +GM ||2 +M>ΓM

M = −(G>G+ Γ)−1G>Pd

Weighted Combination:

Bkd =

M∑i=1

ωki B

kdi

Bd = φ(Bh) ωki =

1

Zexp(−(Bk

di − µk)2/2σ2

k)

Patch Reconstruction[αβ

]= (A>A)A>Pl A =

[DHBd 1

]Br = αBd + β

Where:

Pd = DHBh −Pl − βα

Px = DHdiag(Bhx)

Py = DHdiag(Bhy)

M =

[uv

]G =

[Px Py

]Γ = µ+ λ

[−∇2 00 −∇2

]+ η

[(∇2)2 0

0 (∇2)2

]

EVALUATIONS ON DEFORMATION3×, dictionary size 30000 and overlap 0.

Undeformed Patch(UP), Deformed Patch(DP), Weighted combined UndeformedPatch(UP+W) and Weighted combined Deformed Patch(DP+W)

PSNR(dB) 3×, dictionary size 30000 and overlap 0.Image UP DP UP+W DP+Wlena 29.3259 29.6104 30.4901 30.8557zebra 22.5464 23.0735 24.4748 25.0028

cameraman 24.539 24.7518 25.4206 25.6633oldman 27.914 28.1613 29.2855 29.6448child 28.1921 28.4638 29.5217 29.8793

UP: undeformed patch, DP: deformed patch, W: weighted combination

0 5000 10000 15000 20000 25000 50000 75000 1000029.8

30

30.2

30.4

30.6

30.8

31

PSNR(dB)

SCDL[23]

BPJDL[12]

Proposed

0 5000 10000 15000 20000 25000 50000 75000 1000023.5

24

24.5

25

25.5

26

PSNR(dB)

SCDL[23]

BPJDL[12]

Proposed

EVALUATIONS ON VISUAL QUALITYPSNR(dB) of the final super-resolved test images (3×, dictionary size 30000 and overlap 6)

Image Bicubic Glasner Freedman SCDL BPJDL Proposedlena 30.0986 30.3197 30.6928 31.6493 31.6755 31.7536zebra 23.6214 25.7724 26.8935 27.2387 27.5010 27.7907

cameraman 25.1935 25.9155 25.5409 26.2110 26.2032 26.2221oldman 29.4678 28.9615 30.2424 30.5824 30.6059 30.6666child 29.3479 29.4468 29.7914 30.9166 30.9433 30.9467

Freedman SCDL In-place example BPJDL Proposed

Glasner Freedman SCDL BPJDL Proposed

REFERENCE AND CODE-G. Freedman and R. Fattal. Image and video upscaling from local self-examples. ACM Trans. Graph., 30(2):12:1-12:11, 2011.-D. Glasner, S. Bagon, and M. Irani. Super-resolution from a single image.In ICCV, pages 349-356, 2009.-L. He, H. Qi, and R. Zaretzki. Beta process joint dictionary learning forcoupled feature spaces with application to single image superresolution. InCVPR, pages 345-352, 2013.-J. Yang, J. Wright, T. S. Huang, and Y. Ma. Image super-resolution viasparse representation. IEEE TIP, 19(11):2861-2873, 2010.-J. Yang, Z. Lin, and S. Cohen. Fast image super-resolution based on in-place example regression. In CVPR, pages 1059-1066, 2013.

http://www.stat.ucla.edu/∼zhu.yu/