Upload
donhi
View
222
Download
1
Embed Size (px)
Citation preview
Blind Image Deblurring Using Dark Channel Prior
Jinshan Pan1,2,3, Deqing Sun2,4, Hanspeter Pfister2, and Ming-Hsuan Yang3
1Dalian University of Technology 2Harvard University
3UC Merced 4 NVIDIA
Overview
Blurred image captured in low-light conditions 2
Overview
Restored image 3
Overview
Blurred image 4
Overview
Restored image 5
Overview
• Our goal – A generic method
• state-of-the-art performance on natural images • great results on specific scenes (e.g. saturated images,
text images, face images)
– No edge selection for natural image deblurring – No engineering efforts to incorporate domain
knowledge for specific scenario deblurring
6
Blur Process
• Blur is uniform and spatially invariant
Blurred image Sharp image Noise Blur kernel
7
Blur Process
• Blur is uniform and spatially invariant
Blurred image Sharp image Noise Blur kernel
Convolution operator
8
Challenging
• Blind image deblurring is challenging
? ?
9
Ill-Posed Problem
10
Ill-Posed Problem
11
Related Work
• Probabilistic approach
𝑝 𝑘, 𝐼 𝐵 ∝ 𝑝 𝐵 𝐼, 𝑘 𝑝 𝐼 𝑝 𝑘 Posterior distribution Likelihood Prior on 𝐼 Prior on 𝑘
Blur kernel 𝑘 Latent image 𝐼 Blurred image 𝐵
12
Related Work
• Blur kernel prior – Positive and sparse
0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.10
10
20
30
40
50
60
70
b
p(b)
Most elements near zero
A few can be large
k
P(k)
Shan et al., SIGGRAPH 2008 13
Related Work
• Sharp image statistics – Fergus et al., SIGGRAPH 2006, Levin et al, CVPR
2009, Shan et al., SIGGRAPH 2008… Histogram of image gradients
Log
# pixe
ls
14
Related Work
log ( ) , 1iip I I α α= − ∇ <∑
Derivative distributions in natural images are sparse:
Parametric models
I
Log
prob
I
Gaussian: -I2
Laplacian: -|I| -|I|0.5
-|I|0.25
Levin et al., SIGGRAPH 2007, CVPR 2009 15
Related Work
• MAPI,k framework 𝑝 𝑘, 𝐼 𝐵 ∝ 𝑝 𝐵 𝐼, 𝑘 𝑝 𝐼 𝑝 𝑘
argmax𝑘,𝐼𝑝 𝑘, 𝐼 𝐵
(𝐼, 𝑘) = argmin𝑘,𝐼𝑙 𝐵 − 𝐼 ∗ 𝑘 + 𝜑 𝐼 + 𝜙 𝑘
16
Related Work
• The MAPI,k paradox [Levin et al., CVPR 2009]
P( , )>P( , )
Latent image kernel
Latent image kernel
17
Related Work
• The MAPI,k paradox [Levin et al., CVPR 2009]
sharp blurred
α∑ ∇i
α∑ ∇i
<? 18
Related Work
• The MAPI,k paradox [Levin et al., CVPR 2009]
Red windows = [ p(sharp I) >p(blurred I) ]
15x15 windows 25x25 windows 45x45 windows
simple derivatives [-1,1],[-1;1]
FoE filters
(Roth&Black)
19
Related Work
• The MAPI,k paradox [Levin et al., CVPR 2009]
1|| =d5.0|| 1 =d
5.0|| 2 =d
P(blurred step edge)
sum of derivatives: cheaper
11 5.0 = 41.15.05.0 5.05.0 =+
P(blurred impulse) P(impulse)
5.01 =d 5.02 =d11 =d 12 =d
211 5.05.0 =+ 41.15.05.0 5.05.0 =+sum of derivatives: cheaper
P(step edge)
<
k=[0.5,0.5]
20
<
Related Work
• The MAPI,k paradox [Levin et al., CVPR 2009]
P(blurred real image) P(sharp real image)
cheaper 0.5 5.8ii
I∇ =∑ 0.5 4.5iiI∇ =∑
Noise and texture behave as impulses - total derivative contrast reduced by blur 21
<
Related Work
• The MAPI,k paradox [Levin et al., CVPR 2009] – Maximum marginal probability estimation
• Marginalized probability [Levin et al., CVPR 2011] • Variational Bayesian [Fergus et al., SIGGRAPH 2006]
– MAPI,k
𝑝 𝑘 𝐵 ∝ 𝑝 𝐵 𝑘 𝑝 𝑘 = ∫ 𝑝 𝐵, 𝐼 𝑘 𝑝 𝑘 𝑑𝐼𝐼 = ∫ 𝑝 𝐵 𝐼,𝑘 𝑝(𝐼)𝑝 𝑘 𝑑𝐼𝑙
Marginalizing over 𝐼
𝑝 𝑘, 𝐼 𝐵 ∝ 𝑝 𝐵 𝐼,𝑘 𝑝(𝐼)𝑝 𝑘 22
Related Work
• The MAPI,k paradox [Levin et al., CVPR 2009] – Maximum marginal probability estimation
• Marginalized probability [Levin et al., CVPR 2011] • Variational Bayesian [Fergus et al., SIGGRAPH 2006]
– Computationally expensive
Variational Bayes
Optimization surface for a single variable
Maximum a-Posteriori (MAP)
Pixel intensity
Scor
e
23
Related Work
• Priors favor clear images – [Krishnan et al., CVPR 2011, Pan et al., CVPR 2014,
Michaeli and Irani, ECCV 2014] – Effective for some specific images, such as natural
images or text images – Cannot be generalized well
E(Clear image) < E(Blurred image)
24
Related Work
• MAPI,k with Edge Selection – Main idea
• E(clear) < E(blurred) in sharp edge regions [Levin et al., CVPR 2009]
– [Cho and Lee SIGGRAPH Asia 2009, Xu and Jia ECCV 2010, …]
– Advantages and Limitations • Fast and effective in practice • Explicitly try to recover sharp edges using heuristic
image filters and usually fail when sharp edges are not available
25
Related Work
• MAPI,k with Edge Selection (Extension) – Exemplar based methods [Sun et al., ICCP 2013,
HaCohen et al., ICCV 2013, Pan et al., ECCV 2014]
• Computationally expensive
26
Our Work
• Dark channel prior • Theoretical analysis • Efficient numerical solver • Applications
27
Convolution and Dark Channel
• Dark channel [He et al., CVPR 2009]
• Compute the minimum intensity in a patch of an image
( ) , , ( )( ) min ( )c
y N x c r g bD I x I y
∈∈
=
∑
28
Convolution and Dark Channel
• Convolution
z
s(x) (x+[ ] - z) (z)2
k
B I k∈Ω
= ∑
kΩ : the domain of blur kernel s : the size of blur kernel [ ] : the rounding operator
z(z) 0 (z)=1
k
k k∈Ω
≥ ∑,29
Convolution and Dark Channel
• Proposition 1: Let N(x) denote a patch centered at pixel x with size the same as the blur kernel. We have:
y N(x)(x) min (y)B I
∈≥
72 63 0 0 35 183
73 0 54
9 73 81 0 103 142
89 49 141
149 255 0 0 18 32 0 27 86
146 0 163 0 29 0 0 7 9
1/9 1/9 1/9
1/9 1/9 1/9
1/9 1/9 1/9
0 30 ≥ 0
A toy example 30
Convolution and Dark Channel
• Property 1: Let D(B) and D(I) denote the dark channel of the blurred and clear images, we have:
• Property 2: Let Ω denote the domain of an image I. If there exist some pixels x ∈ Ω such that I(x) = 0, we have:
D( )(x) D( )(x)B I≥
0 0||D( )|| > ||D( )||B I
31
Convolution and Dark Channel
0
20000
40000
60000
0 0.01 0.02 0.03 0.04 0.05Aver
age
num
ber o
f dar
k pi
xels
Intensity
Blurred images Clear images
The statistical results on the dataset with 3,200 examples 32
Convolution and Dark Channel
Clear Blurred Clear Blurred Blurred images have less sparse dark channels than clear images
33
Proposed Method
• Our model – Add the dark channel prior into standard
deblurring model
– How to solve? • L0 norm and non-linear min operator
2 22 2 0 0,
min || * || + || || || || | ||( )|I k
I k B I D Ikγ µ λ− + ∇ +
34
Optimization
• Algorithm skeleton – L0 norm
• Half-quadratic splitting method
– Non-linear min operator • Linear approximation
2 22 2min || * || || ||
kI k B kγ− +
22 0 0min || * || || || + || ( ) ||
II k B I D Iµ λ− + ∇
35
Optimization
• Update latent image I: – Alternative minimization
22 0 0min || * || || || || ( ) ||
II k B I D Iµ λ− + ∇ +
2 22 2 0 0
22, ,
|| || + || ( )min || * || || || || ||||I u g
I g D I uI k B g uα β µ λ∇ − − + +− +
2 2 22 2 2min || * || || ( ) || || ||
II k B D I u I gβ µ− + − + ∇ −
2 20 0,
|| || || ( ) || || || || |min |u g
I g D I u g uα β µ λ∇ − + − + +
Half-quadratic splitting [Xu et al., SIGGRAPH Asia 2011, Pan et al., CVPR 2014]
36
2 0 02 |min | | || || ( ||| * || )
IIB II k Dµ λ− + ∇ +
Optimization
• Update latent image I: – u, g sub-problem
2 20 0,
min || || + || ( ) || || || || ||u g
I g D I u g uα β µ λ∇ − − + +
20
20
min || ( ) || || ||
min || || || ||u
g
D I u u
x g g
β λ
α µ
− +
∇ − +
2( ),| ( ) |
0,otherwise
D I D Iu
λβ
≥=
2,| |
0,otherwise
I Ig
µα
∇ ∇ ≥=
Related papers: [Xu et al., SIGGRAPH Asia 2011, Xu et al., CVPR 2013, Pan et al., CVPR 2014]
u and g are independent!
37
Optimization
• Update latent image I: – I sub-problem
– Our observation
• Let y = argminz∈𝑁 x 𝐼(z), we have
2 2 22min || * || || ( ) || || ||
II k B D I u I gβ µ− + − + ∇ −
( )=MID I
1, z=y,M(x, z)=
0, otherwise.
38
min operator
Optimization
• I sub-problem – Compute M
Intermediate image I D(I)
Visualization of 𝐌Tu u
𝐌
𝐌T
Toy example 39
Experimental Results
• Natural image deblurring • Specific scenes
– Text images – Face images – Low-light images
• Non-uniform image deblurring
40
Natural Image Deblurring Results
• Quantitative evaluation – Levin et al., CVPR 2009 – Köhler et al. ECCV 2012 – Sun et al., ICCP 2013
41
Natural Image Deblurring Results
20
40
60
80
100
1.5 2 2.5 3 3.5 4
Succ
ess r
ate
(%)
Error ratios
OursXu et al.Xu and JiaPan et al.Levin et al.Cho and LeeMichaeli and IraniKrishnan et al.
Quantitative evaluations on the dataset by Levin et al., CVPR 2009
100%
42
Natural Image Deblurring Results
18
21
24
27
30
33
im01 im02 im03 im04 Average
Aver
age
PSN
R Va
lues
Blurred imagesFergus et al.Shan et al.Cho and LeeXu and JiaKrishnan et al.Hirsch et al.Whyte et al.Pan et al.Ours
Quantitative evaluations on the dataset by Köhler et al. ECCV 2012 43
Natural Image Deblurring Results
0
20
40
60
80
100
1 2 3 4 5 6
Succ
ess r
ate
(%)
Error ratios
OursXu and JiaPan et al.Michaeli and IraniSun et al.Xu et al.Levin et al.Krishnan et al.Cho and Lee
Quantitative evaluations on the dataset by Sun et al. ICCP 2013 44
Natural Image Deblurring Results
Blurred image Cho and Lee SIGGRAPH Asia 2009 Xu and Jia, ECCV 2010
Krishnan et al., CVPR 2011 Ours without D(I) Ours 45
Natural Image Deblurring Results
Blurred image Krishnan et al., CVPR 2011 Xu et al., CVPR 2013
Pan et al., CVPR 2014 Ours without D(I) Ours
Our real captured example 46
Text Image Deblurring Results
Average PSNRs
Cho and Lee 23.80
Xu and Jia 26.21
Krishnan et al. 20.86
Levin et al. 24.90
Xu et al. 26.21
Pan et al. 28.80
Ours 27.94
Quantitative evaluations on the text image dataset by Pan et al., CVPR 2014
Natural image debluring methods
47
Text Image Deblurring Results
Blurred image Xu et al., CVPR 2013
Pan et al., CVPR 2014 Ours Real captured example 48
Saturated Image Deblurring Results
Pan et al., CVPR 2014 Ours
Blurred image Xu et al., CVPR 2013
Real captured example 49
Face Image Deblurring Results
Blurred image Pan et al., ECCV 2014 Ours Xu et al., CVPR 2013
50
Real captured example
Non-Uniform Deblurring
Blurred image Krishnan et al., CVPR 2011 Whyte et al., IJCV 2012
Xu et al., CVPR 2013 Ours Our estimated kernels 51
Convergence
0.76
0.78
0.8
0.82
1 12 23 34 45
Aver
age
Kern
el S
imila
rity
Iterations
40
80
120
160
1 12 23 34 45Av
erag
e En
ergi
es
Iterations
Kernel similarity plot Objective function value plot
52
Running Time
Method 255 x 255 600 x 600 800 x 800
Xu et al. (C++) 1.11 3.56 4.31
Krishnan et al. (Matlab) 24.23 111.09 226.58
Levin et al. (Matlab) 117.06 481.48 917.84
Ours without D(I) (Matlab) 2.77 15.65 28.94
Ours with naive implementation (Matlab) 134.31 691.71 964.90
Ours (Matlab) 17.07 115.86 195.80
Running time (/s) comparisons (obtained on the same PC).
53
Analysis and Discussions
• Effectiveness of dark channel prior
Results on the dataset by Köhler et al. ECCV 2012
Results on the dataset by Levin et al. CVPR 2009
75
85
95
105
1.5 2 2.5 3
Succ
ess r
ate
(%)
Error ratios
Ours without dark channel Ours
25
27
29
31
33
im01 im02 im03 im04 Average
Aver
age
PSN
R Va
lues
Ours without dark channel Ours
54
Analysis and Discussions
• Existing prior favors clear images [Krishnan et al. CVPR 2011]
1
2
|| ||( )|| ||
Ip II
∇=
∇
0
20000000
40000000
60000000
80000000
1 14 27 40 53 66 79 92 105 118
Ener
gy V
alue
s of p
(I)
Image index
Blurred imagesClear images
Statistics of different priors on the text image deblurring by Pan et al., CVPR 2014. The normalized sparsity sometimes favors blurred text images 55
Analysis and Discussions
• Dark channel prior favors clear images
Statistics of different priors on the text image deblurring by Pan et al., CVPR 2014. The dark channel prior favors clear text images
0
20000
40000
60000
80000
0 0.2 0.4 0.6
Aver
age
num
ber o
f dar
k pi
xels
Intensity
Blurred imagesClear images
56
Limitations
• The dark channel of clear image does not contain zero-elements – Property 2 does not hold – Dark channel prior has no effect on image
deblurring
0 0|| ( ) || || ( ) ||D B D I=
57
Limitations
• The dark channel of clear image does not contain zero-elements
Blurred image Without D(I) Ours
Dark channel of clear image Dark channel of blurred image Estimated dark channel
Dark channel prior has no effect on image deblurring 58
Limitations
• Images containing noise
Blurred image 59
Limitations
• Images containing noise
Without D(I) 60
Limitations
• Images containing noise
With D(I) 61
Take Home Message
• The change in the sparsity of the dark channel is an inherent property of the blur process!
Code and datasets will be available at the authors’ websites.
62
More Results
63 Real captured image
More Results
64 Our result
More Results
65 Real captured image
More Results
66 Our result
Our Related Deblurring Work
• Outlier deblurring (CVPR 2016) – http://vllab.ucmerced.edu/~jinshan/projects/outli
er-deblur/
• Object motion deblurring (CVPR 2016) – http://vllab.ucmerced.edu/~jinshan/projects/obje
ct-deblur/
• Text image deblurring and beyond (TPAMI 2016) – http://vllab.ucmerced.edu/~jinshan/projects/text-
deblur/ 67