Embed Size (px)
DESCRIPTION
Single Image Super-Resolution: A Benchmark Chih-Yuan Yang 1 , Chao Ma 2 , Ming- Hsuan Yang 1 UC Merced 1 , Shanghai Jiao Tong University 2. Motivation. We would like to figure out some questions. Which is the best super-resolution algorithm? What is the influence of blur kernel width? - PowerPoint PPT Presentation
Citation preview
Single Image Super-Resolution:
A Benchmark
Chih-Yuan Yang1, Chao Ma2, Ming-Hsuan Yang1
UC Merced1, Shanghai Jiao Tong University2
Motivation
• We would like to figure out some questions.• Which is the best super-resolution algorithm?• What is the influence of blur kernel width?• What metric should be used?
Approach (step 1)
We collect 11 state-of-the-art super-resolution algorithms 1. Bicubic interpolation2. Back projection (Irani 93 : IP)3. Fast image/video (Shan 07 : SLJT)4. Gradient profile (Sun 08 : SSXS)5. Self example (Glasner 09 : GBI)6. Sparse regression (Kim 10 : KK)7. Sparse representation (Yang 10 : YWHM)8. Local self example (Freedman 11 : FF)9. Adaptive regularization (Dong 11 : DZSW)10. Simple function (Yang 13 : YY)11. Anchored neighborhood regression (Timofte 13 : TSG)
Approach (step 2)
We set 2 parameters• Scaling factors as 6 values• 2 3 4 5 6 8
• Blurring kernel width as 9 values• 0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0
to generate super-resolution images from 2 datasets• Berkeley segmentation dataset (200 images)• LIVE dataset (29 images)
Approach (step 3)
We conduct a human subject study to collect perceptual scoresand compute the ranked correlation coefficient between the perceptual scores and 8 metric indices1. PSNR2. Weighted PSNR3. SSIM4. Multi-scale SSIM5. VIF (visual information fidelity)6. UIQI (universal image quality index)7. IFC (information fidelity criterion)8. NQM (noise quality measure)
Flow chart
4.25 5.416.24 3.335.78 4.953.77 5.12
29.77 28.4127.23 28.0126.44 25.9826.66 27.34
Ground Truth
Low-resolution
High-resolution
downsample
image quality assessment
scaling factorblur kernel width
metric indices
perceptual scores
metric
(1)(2)
(3)
(4)
(5)
Prepare ground truth images
Flow chart
4.25 5.416.24 3.335.78 4.953.77 5.12
29.77 28.4127.23 28.0126.44 25.9826.66 27.34
Ground Truth
Low-resolution
High-resolution
downsample
image quality assessment
scaling factorblur kernel width
metric indices
perceptual scores
metric
(1)(2)
(3)
(4)
(5)
Generate low-resolutionimages
Flow chart
4.25 5.416.24 3.335.78 4.953.77 5.12
29.77 28.4127.23 28.0126.44 25.9826.66 27.34
Ground Truth
Low-resolution
High-resolution
downsample
image quality assessment
scaling factorblur kernel width
metric indices
perceptual scores
metric
(1)(2)
(3)
(4)
(5)Generate super-resolution
images
Flow chart
4.25 5.416.24 3.335.78 4.953.77 5.12
29.77 28.4127.23 28.0126.44 25.9826.66 27.34
Ground Truth
Low-resolution
High-resolution
downsample
image quality assessment
scaling factorblur kernel width
metric indices
perceptual scores
metric
(1)(2)
(3)
(4)
(5)
Compute metric indices
Flow chart
4.25 5.416.24 3.335.78 4.953.77 5.12
29.77 28.4127.23 28.0126.44 25.9826.66 27.34
Ground Truth
Low-resolution
High-resolution
downsample
image quality assessment
scaling factorblur kernel width
metric indices
perceptual scores
metric
(1)(2)
(3)
(4)
(5)
Compute correlation coefficients
Averaged Metric Indices
BSD dataset (200 images) LIVE dataset (29 images)
s=2
s=3
s=4
s=5
s=6
s=8
We find
BSD dataset (200 images)
s=2
s=3
s=4
the SLJT, FF, DZSW methods generate misaligned super-resolution results and low metric indices
We find
BSD dataset (200 images)
s=2
s=3
s=4
the best Gaussian kernel width isproportional to the scaling factor
Reason
Information remained in a low-resolution image is determined by 2 factors1. blurring2. subsamplingWhen a subsampling ratio is larger, a larger kernel maximizes the remained information in low-resolution images.
We find
index / PSNR
all algorithms work well for smooth imagesbut poorly for highly textured images.
Easiest
Mostchallenging
Reason
• All test algorithms use appearance features and statistical approaches.• Thus they effectively handle smooth regions but
difficultly reconstruct textures.
Perceptual correlations
Best: IFC0.8434
Worst: VIF0.3874
PSNR0.4760
SSIM0.6203
Reason
• IFC is a metric modelled by natural image priors based on high-frequency features• Our test images are all natural images• The perceptual scores are determined by the
reconstructed high-frequency details
Conclusions
• IFC metric shows higher correlation with perceptual scores than PSNR and SSIM• Existing algorithms have difficulty to reconstruct
high-frequency textures• A scaling factor of 4 is already challenging
Future Work
• How to overcome the limitation of visual features and statistical approaches?• How to evaluate super-resolution results without a
ground truth image?
Code and datasets available• https://eng.ucmerced.edu/people/cyang35• 11 algorithms on MATLAB• 4 of our implementation (IP, SSXS, GBI, FF)• 7 of original release
• 400 Perceptual scores• 130,000 super-resolution images• 1M evaluation values
Thank you for your attention.
