Image Decomposition, Inpainting, and Impulse Noise Removal by Sparse & Redundant Representations Michael Elad The Computer Science Department The Technion

Image Decomposition, Inpainting, and Impulse Noise Removal by

Sparse & Redundant Representations

Michael EladThe Computer Science DepartmentThe Technion – Israel Institute of technologyHaifa 32000, Israel

Sparse representations for Image Decomposition

2

General

Sparsity and over-completeness have important roles in analyzing and representing signals.

We have already discussed the image separation task (image=cartoon + texture) and inpainting based on it, adopting a global treatment of these problems.

We have seen in the context of denoising that migration from global to local processing led to far better results.

We Now return to these tasks (and beyond), with a patch-based approach, with the hope to do better.

Before – Lets recall the global treatment


3

Global Separation &

Inpainting


4

Decomposition via Sparsity

yx

00,

s.t.sArgMinˆˆ

y

x

= s+

The dictionaries x y should be chosen such that they•sparsify the desired content, while •giving poor service to the opposite content


5

Noise Considerations

2

2yx11,

sArgMinˆˆ

yx

11,

s.t.sArgMinˆˆ

Forcing exact representation is sensitive to additive noise

and model mismatch


6

Original ‘Barbara’ image

Separated texture using local overlapped DCT (32×32 blocks)

Separated Cartoon using Curvelets (5 resolution layers)

Results – Good old ‘Barbara’


7

Results – Zoom in

Zoom in on the result

shown in the previous slide

(the texture part)

Zoom in on the results

shown in the previous slide

(the cartoon part)

The same part taken from Vese’s et. al.

The same part taken from Vese’s et. al.


8

Application - Inpainting

2

2yx11,

sArgMinˆˆ

For separation

What if some values in s are unknown (with known locations!!!)?

2x y1 1 2,

ˆArgMin s

ˆM

The image will be the inpainted outcome. Interesting comparison to [Bertalmio et.al.

’02]

βΦαΦ yx


9

Results - Inpainting

Source

Cartoon Part

Texture Part

Outcome


10


Source

Cartoon Part

Texture Part

Outcome


11


Outcome

Source


12

Patch-Based (Local)

Separation


13

MCA for Patches

CXTX T CY X X V

Lets impose the MCA model on each and every patch-set in these images

kk T CT Ck T Ck k 2k 2

YX X V

R D DR D D


14

Start by Denoising

kT C k

kand qD D D

k k

22kk k2 0 2X, , q k

min X Y q q XD

D R

We define ,

and formulate a patch-based MAP denoising problem:

We are familiar with this expression: it aims to remove the noise V from Y, bypassing altogether the separation task.

k kq Compute

Sparse Coding of patches

Compute D K-SVD Update

Compute X Patch-

averaging

kk T C

k 2

YR D D


15

Decompose the Dictionary

The denoising process leads to a dictionary D, assumed to be composed of DC and DT.

We need to divide the content of D into these two “sub-dictionaries”, and we can do this by measuring the Total-Variation.

Any atom with “activity” below a threshold is assigned as a cartoon atom, and similarly, to the texture one if above the threshold.

DC & DT


16

Final Image Separation

Now that DC and DT are defined, we can separate.

Starting with

the coefficients in qk are split between the two dictionaries.

We obtain the system:

22kk k2 0 2X k

min X Y q q XD R

22

T C T CT C k kk k2 2X kmin X X Y X XD D R R

22

T CT k C kk k2 2X XD R D R

T Tk k k T kT

k kT Tk k k C kC k k

YX

YX

I R R I R D

I I R R R D


17

Results

The obtained results are of similar quality (and slightly better) than the

global approach


18

Results

Zoom in on the texture

global result shown before

Zoom in on the cartoon

global result shown before

The same part taken from the local approach

The same part taken from the local approach


19

Patch-Based (Local)

Inpainting


20

Lets Start with a Simple Experiment

We assume that the image is corrupted by additive noise (=20), and z% missing pixels (the Mask is given)

Lets break the image into a set of non-overlapping patches, and inpaint each separately by (D: 2D-DCT)

Once done, the patches are put into position to create the inpainted image.

0

2 2k k k2

min

s.t. Y cnM D R


21

Lets Start with a Simple Experiment

25% missing 50% missing 75% missing

RMSE for 75%missing

RMSE for 50% missing


Algorithm

29.70 19.61 14.55 No-overlap


22

How About Overlapped Patches?

RMSE for 75%missing



Algorithm

29.70 19.61 14.55 No-overlap

18.18 11.55 9.00 Overlap

25% missing 50% missing 75% missing


23

Formulating This Problem: K-SVD

k k

2 2G kk k0 22X, , k

min X Y XD

M D R

k k Compute

Sparse Coding of patches

Compute D K-SVD Update

Compute X Patch-

averaging

2 2

k k k0 2min s.t. Y cnM D R

Initialize with X=MGY

Sparse coding:

Dictionary Update:

Image Update:

2k k 2

kmin YD

M D R

2 2G k 22X k

min X Y XM D R


24

Results

RMSE for 75%missing



Alg.

29.70 19.61 14.55 No-overlap

18.18 11.55 9.00 Overlap

17.74 10.05 8.1 K-SVD

For the Peppers image

This is a more challenging case, where the DCT is not a suitable dictionary. • For Redundant DCT we get

RMSE=16.13, and • For K-SVD (15 iterations) we

get RMSE=12.74


25

Inpainting Color Images


26

Inpainting Video


27

Patch-Based Impulse-Noise

Removal


28

Introduction

In the inpainting problem discussed above, the mask is KNOWN. What do we do if the mask (missing pixels) is unknown – this is the impulse noise problem.

Definition: we obtain an image with p% of the pixels in random locations corrupted by adding ±50 to the gray-level. This is known as “salt-and-pepper” noise.

This is an example with 10% corrupted pixels:

Most common method for filtering this noise: Median filtering


29

Sparse-Land Alternative

We will use the median filter as a simple (and primitive) detector of missing pixels (there are better ways!):

Once the mask is available, we shall proceed with regular inpainting

… BUT …

Applying the inpainting ONLY for the masked pixels

For a fair comparison, we will adopt the same mask for the median filter., i.e., modify the pixels having Mask=0 only.

k0 Y Median Y 27

Mask1 Otherwise


30

Results

The image with the

detected mask

Masked median filter,

RMSE=6.52

Inpainting by DCT, RMSE=5.86

Regular Median filter, RMSE=7.42

Documents

Image Decomposition, Inpainting, and Impulse Noise Removal by Sparse & Redundant Representations Michael Elad The Computer Science Department The Technion