Blind Image Deblurring Using Dark Channel...

Blind Image Deblurring Using Dark Channel Prior

Jinshan Pan1,2,3, Deqing Sun2,4, Hanspeter Pfister2, and Ming-Hsuan Yang3

1Dalian University of Technology 2Harvard University

3UC Merced 4 NVIDIA

Overview

Blurred image captured in low-light conditions 2

Overview

Restored image 3

Overview

Blurred image 4

Overview

Restored image 5

Overview

• Our goal – A generic method

• state-of-the-art performance on natural images • great results on specific scenes (e.g. saturated images,

text images, face images)

– No edge selection for natural image deblurring – No engineering efforts to incorporate domain

knowledge for specific scenario deblurring

6

Blur Process

• Blur is uniform and spatially invariant

Blurred image Sharp image Noise Blur kernel

7

Blur Process

• Blur is uniform and spatially invariant

Blurred image Sharp image Noise Blur kernel

Convolution operator

8

Challenging

• Blind image deblurring is challenging

? ?

9

Ill-Posed Problem

10

Ill-Posed Problem

11

Related Work

• Probabilistic approach

𝑝 𝑘, 𝐼 𝐵 ∝ 𝑝 𝐵 𝐼, 𝑘 𝑝 𝐼 𝑝 𝑘 Posterior distribution Likelihood Prior on 𝐼 Prior on 𝑘

Blur kernel 𝑘 Latent image 𝐼 Blurred image 𝐵

12

Related Work

• Blur kernel prior – Positive and sparse

0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.10

10

20

30

40

50

60

70

b

p(b)

Most elements near zero

A few can be large

k

P(k)

Shan et al., SIGGRAPH 2008 13

Related Work

• Sharp image statistics – Fergus et al., SIGGRAPH 2006, Levin et al, CVPR

2009, Shan et al., SIGGRAPH 2008… Histogram of image gradients

Log

# pixe

ls

14

Related Work

log ( ) , 1iip I I α α= − ∇ <∑

Derivative distributions in natural images are sparse:

Parametric models

I

Log

prob

I

Gaussian: -I2

Laplacian: -|I| -|I|0.5

-|I|0.25

Levin et al., SIGGRAPH 2007, CVPR 2009 15

Related Work

• MAPI,k framework 𝑝 𝑘, 𝐼 𝐵 ∝ 𝑝 𝐵 𝐼, 𝑘 𝑝 𝐼 𝑝 𝑘

argmax𝑘,𝐼𝑝 𝑘, 𝐼 𝐵

(𝐼, 𝑘) = argmin𝑘,𝐼𝑙 𝐵 − 𝐼 ∗ 𝑘 + 𝜑 𝐼 + 𝜙 𝑘

16

Related Work

• The MAPI,k paradox [Levin et al., CVPR 2009]

P( , )>P( , )

Latent image kernel

Latent image kernel

17

Related Work

• The MAPI,k paradox [Levin et al., CVPR 2009]

sharp blurred

α∑ ∇i

α∑ ∇i

<? 18

Related Work

• The MAPI,k paradox [Levin et al., CVPR 2009]

Red windows = [ p(sharp I) >p(blurred I) ]

15x15 windows 25x25 windows 45x45 windows

simple derivatives [-1,1],[-1;1]

FoE filters

(Roth&Black)

19

Related Work

• The MAPI,k paradox [Levin et al., CVPR 2009]

1|| =d5.0|| 1 =d

5.0|| 2 =d

P(blurred step edge)

sum of derivatives: cheaper

11 5.0 = 41.15.05.0 5.05.0 =+

P(blurred impulse) P(impulse)

5.01 =d 5.02 =d11 =d 12 =d

211 5.05.0 =+ 41.15.05.0 5.05.0 =+sum of derivatives: cheaper

P(step edge)

<

k=[0.5,0.5]

20

<

Related Work

• The MAPI,k paradox [Levin et al., CVPR 2009]

P(blurred real image) P(sharp real image)

cheaper 0.5 5.8ii

I∇ =∑ 0.5 4.5iiI∇ =∑

Noise and texture behave as impulses - total derivative contrast reduced by blur 21

<

Related Work

• The MAPI,k paradox [Levin et al., CVPR 2009] – Maximum marginal probability estimation

• Marginalized probability [Levin et al., CVPR 2011] • Variational Bayesian [Fergus et al., SIGGRAPH 2006]

– MAPI,k

𝑝 𝑘 𝐵 ∝ 𝑝 𝐵 𝑘 𝑝 𝑘 = ∫ 𝑝 𝐵, 𝐼 𝑘 𝑝 𝑘 𝑑𝐼𝐼 = ∫ 𝑝 𝐵 𝐼,𝑘 𝑝(𝐼)𝑝 𝑘 𝑑𝐼𝑙

Marginalizing over 𝐼

𝑝 𝑘, 𝐼 𝐵 ∝ 𝑝 𝐵 𝐼,𝑘 𝑝(𝐼)𝑝 𝑘 22

Related Work

• The MAPI,k paradox [Levin et al., CVPR 2009] – Maximum marginal probability estimation

• Marginalized probability [Levin et al., CVPR 2011] • Variational Bayesian [Fergus et al., SIGGRAPH 2006]

– Computationally expensive

Variational Bayes

Optimization surface for a single variable

Maximum a-Posteriori (MAP)

Pixel intensity

Scor

e

23

Related Work

• Priors favor clear images – [Krishnan et al., CVPR 2011, Pan et al., CVPR 2014,

Michaeli and Irani, ECCV 2014] – Effective for some specific images, such as natural

images or text images – Cannot be generalized well

E(Clear image) < E(Blurred image)

24

Related Work

• MAPI,k with Edge Selection – Main idea

• E(clear) < E(blurred) in sharp edge regions [Levin et al., CVPR 2009]

– [Cho and Lee SIGGRAPH Asia 2009, Xu and Jia ECCV 2010, …]

– Advantages and Limitations • Fast and effective in practice • Explicitly try to recover sharp edges using heuristic

image filters and usually fail when sharp edges are not available

25

Related Work

• MAPI,k with Edge Selection (Extension) – Exemplar based methods [Sun et al., ICCP 2013,

HaCohen et al., ICCV 2013, Pan et al., ECCV 2014]

• Computationally expensive

26

Our Work

• Dark channel prior • Theoretical analysis • Efficient numerical solver • Applications

27

Convolution and Dark Channel

• Dark channel [He et al., CVPR 2009]

• Compute the minimum intensity in a patch of an image

( ) , , ( )( ) min ( )c

y N x c r g bD I x I y

∈∈

=

∑

28

Convolution and Dark Channel

• Convolution

z

s(x) (x+[ ] - z) (z)2

k

B I k∈Ω

= ∑

kΩ : the domain of blur kernel s : the size of blur kernel [ ] : the rounding operator

z(z) 0 (z)=1

k

k k∈Ω

≥ ∑，29

Convolution and Dark Channel

• Proposition 1: Let N(x) denote a patch centered at pixel x with size the same as the blur kernel. We have:

y N(x)(x) min (y)B I

∈≥

72 63 0 0 35 183

73 0 54

9 73 81 0 103 142

89 49 141

149 255 0 0 18 32 0 27 86

146 0 163 0 29 0 0 7 9

1/9 1/9 1/9

1/9 1/9 1/9

1/9 1/9 1/9

0 30 ≥ 0

A toy example 30

Convolution and Dark Channel

• Property 1: Let D(B) and D(I) denote the dark channel of the blurred and clear images, we have:

• Property 2: Let Ω denote the domain of an image I. If there exist some pixels x ∈ Ω such that I(x) = 0, we have:

D( )(x) D( )(x)B I≥

0 0||D( )|| > ||D( )||B I

31

Convolution and Dark Channel

0

20000

40000

60000

0 0.01 0.02 0.03 0.04 0.05Aver

age

num

ber o

f dar

k pi

xels

Intensity

Blurred images Clear images

The statistical results on the dataset with 3,200 examples 32

Convolution and Dark Channel

Clear Blurred Clear Blurred Blurred images have less sparse dark channels than clear images

33

Proposed Method

• Our model – Add the dark channel prior into standard

deblurring model

– How to solve? • L0 norm and non-linear min operator

2 22 2 0 0,

min || * || + || || || || | ||( )|I k

I k B I D Ikγ µ λ− + ∇ +

34

Optimization

• Algorithm skeleton – L0 norm

• Half-quadratic splitting method

– Non-linear min operator • Linear approximation

2 22 2min || * || || ||

kI k B kγ− +

22 0 0min || * || || || + || ( ) ||

II k B I D Iµ λ− + ∇

35

Optimization

• Update latent image I: – Alternative minimization

22 0 0min || * || || || || ( ) ||

II k B I D Iµ λ− + ∇ +

2 22 2 0 0

22, ,

|| || + || ( )min || * || || || || ||||I u g

I g D I uI k B g uα β µ λ∇ − − + +− +

2 2 22 2 2min || * || || ( ) || || ||

II k B D I u I gβ µ− + − + ∇ −

2 20 0,

|| || || ( ) || || || || |min |u g

I g D I u g uα β µ λ∇ − + − + +

Half-quadratic splitting [Xu et al., SIGGRAPH Asia 2011, Pan et al., CVPR 2014]

36

2 0 02 |min | | || || ( ||| * || )

IIB II k Dµ λ− + ∇ +

Optimization

• Update latent image I: – u, g sub-problem

2 20 0,

min || || + || ( ) || || || || ||u g

I g D I u g uα β µ λ∇ − − + +

20

20

min || ( ) || || ||

min || || || ||u

g

D I u u

x g g

β λ

α µ

− +

∇ − +

2( ),| ( ) |

0,otherwise

D I D Iu

λβ

≥=

2,| |

0,otherwise

I Ig

µα

∇ ∇ ≥=

Related papers: [Xu et al., SIGGRAPH Asia 2011, Xu et al., CVPR 2013, Pan et al., CVPR 2014]

u and g are independent!

37

Optimization

• Update latent image I: – I sub-problem

– Our observation

• Let y = argminz∈𝑁 x 𝐼(z), we have

2 2 22min || * || || ( ) || || ||

II k B D I u I gβ µ− + − + ∇ −

( )=MID I

1, z=y,M(x, z)=

0, otherwise.

38

min operator

Optimization

• I sub-problem – Compute M

Intermediate image I D(I)

Visualization of 𝐌Tu u

𝐌

𝐌T

Toy example 39

Experimental Results

• Natural image deblurring • Specific scenes

– Text images – Face images – Low-light images

• Non-uniform image deblurring

40

Natural Image Deblurring Results

• Quantitative evaluation – Levin et al., CVPR 2009 – Köhler et al. ECCV 2012 – Sun et al., ICCP 2013

41

Natural Image Deblurring Results

20

40

60

80

100

1.5 2 2.5 3 3.5 4

Succ

ess r

ate

(%)

Error ratios

OursXu et al.Xu and JiaPan et al.Levin et al.Cho and LeeMichaeli and IraniKrishnan et al.

Quantitative evaluations on the dataset by Levin et al., CVPR 2009

100%

42

Natural Image Deblurring Results

18

21

24

27

30

33

im01 im02 im03 im04 Average

Aver

age

PSN

R Va

lues

Blurred imagesFergus et al.Shan et al.Cho and LeeXu and JiaKrishnan et al.Hirsch et al.Whyte et al.Pan et al.Ours

Quantitative evaluations on the dataset by Köhler et al. ECCV 2012 43

Natural Image Deblurring Results

0

20

40

60

80

100

1 2 3 4 5 6

Succ

ess r

ate

(%)

Error ratios

OursXu and JiaPan et al.Michaeli and IraniSun et al.Xu et al.Levin et al.Krishnan et al.Cho and Lee

Quantitative evaluations on the dataset by Sun et al. ICCP 2013 44

Natural Image Deblurring Results

Blurred image Cho and Lee SIGGRAPH Asia 2009 Xu and Jia, ECCV 2010

Krishnan et al., CVPR 2011 Ours without D(I) Ours 45

Natural Image Deblurring Results

Blurred image Krishnan et al., CVPR 2011 Xu et al., CVPR 2013

Pan et al., CVPR 2014 Ours without D(I) Ours

Our real captured example 46

Text Image Deblurring Results

Average PSNRs

Cho and Lee 23.80

Xu and Jia 26.21

Krishnan et al. 20.86

Levin et al. 24.90

Xu et al. 26.21

Pan et al. 28.80

Ours 27.94

Quantitative evaluations on the text image dataset by Pan et al., CVPR 2014

Natural image debluring methods

47

Text Image Deblurring Results

Blurred image Xu et al., CVPR 2013

Pan et al., CVPR 2014 Ours Real captured example 48

Saturated Image Deblurring Results

Pan et al., CVPR 2014 Ours

Blurred image Xu et al., CVPR 2013

Real captured example 49

Face Image Deblurring Results

Blurred image Pan et al., ECCV 2014 Ours Xu et al., CVPR 2013

50

Real captured example

Non-Uniform Deblurring

Blurred image Krishnan et al., CVPR 2011 Whyte et al., IJCV 2012

Xu et al., CVPR 2013 Ours Our estimated kernels 51

Convergence

0.76

0.78

0.8

0.82

1 12 23 34 45

Aver

age

Kern

el S

imila

rity

Iterations

40

80

120

160

1 12 23 34 45Av

erag

e En

ergi

es

Iterations

Kernel similarity plot Objective function value plot

52

Running Time

Method 255 x 255 600 x 600 800 x 800

Xu et al. (C++) 1.11 3.56 4.31

Krishnan et al. (Matlab) 24.23 111.09 226.58

Levin et al. (Matlab) 117.06 481.48 917.84

Ours without D(I) (Matlab) 2.77 15.65 28.94

Ours with naive implementation (Matlab) 134.31 691.71 964.90

Ours (Matlab) 17.07 115.86 195.80

Running time (/s) comparisons (obtained on the same PC).

53

Analysis and Discussions

• Effectiveness of dark channel prior

Results on the dataset by Köhler et al. ECCV 2012

Results on the dataset by Levin et al. CVPR 2009

75

85

95

105

1.5 2 2.5 3

Succ

ess r

ate

(%)

Error ratios

Ours without dark channel Ours

25

27

29

31

33

im01 im02 im03 im04 Average

Aver

age

PSN

R Va

lues

Ours without dark channel Ours

54

Analysis and Discussions

• Existing prior favors clear images [Krishnan et al. CVPR 2011]

1

2

|| ||( )|| ||

Ip II

∇=

∇

0

20000000

40000000

60000000

80000000

1 14 27 40 53 66 79 92 105 118

Ener

gy V

alue

s of p

(I)

Image index

Blurred imagesClear images

Statistics of different priors on the text image deblurring by Pan et al., CVPR 2014. The normalized sparsity sometimes favors blurred text images 55

Analysis and Discussions

• Dark channel prior favors clear images

Statistics of different priors on the text image deblurring by Pan et al., CVPR 2014. The dark channel prior favors clear text images

0

20000

40000

60000

80000

0 0.2 0.4 0.6

Aver

age

num

ber o

f dar

k pi

xels

Intensity

Blurred imagesClear images

56

Limitations

• The dark channel of clear image does not contain zero-elements – Property 2 does not hold – Dark channel prior has no effect on image

deblurring

0 0|| ( ) || || ( ) ||D B D I=

57

Limitations

• The dark channel of clear image does not contain zero-elements

Blurred image Without D(I) Ours

Dark channel of clear image Dark channel of blurred image Estimated dark channel

Dark channel prior has no effect on image deblurring 58

Limitations

• Images containing noise

Blurred image 59

Limitations

• Images containing noise

Without D(I) 60

Limitations

• Images containing noise

With D(I) 61

Take Home Message

• The change in the sparsity of the dark channel is an inherent property of the blur process!

Code and datasets will be available at the authors’ websites.

62

More Results

63 Real captured image

More Results

64 Our result

More Results

65 Real captured image

More Results

66 Our result

Our Related Deblurring Work

• Outlier deblurring (CVPR 2016) – http://vllab.ucmerced.edu/~jinshan/projects/outli

er-deblur/

• Object motion deblurring (CVPR 2016) – http://vllab.ucmerced.edu/~jinshan/projects/obje

ct-deblur/

• Text image deblurring and beyond (TPAMI 2016) – http://vllab.ucmerced.edu/~jinshan/projects/text-

deblur/ 67