CV Restoration

Computer Vision -Computer Vision -RestorationRestoration

Hanyang University

Jong-Il Park

Department of Computer Science and Engineering, Hanyang University

Restoration vs. EnhancementRestoration vs. Enhancement

Restoration

Objective process

A priori knowledge on degradation model

Modeling the degradation and applying the inverse process to recover the original

To improve an image in some predefined sense

Enhancement

Subjective process


Restoration processRestoration process


Noise modelsNoise models

Assume noise is independent of spatial coordinates and it isuncorrelated w.r.t. the image.

• Gaussian: electronic circuit noise, sensor noise• Rayleigh: range images• Exponential and gamma: laser images• impulse(salt-and-pepper): faulty switching


Eg. Sample noisy imagesEg. Sample noisy images


Eg. Sample noisy images(cont.)Eg. Sample noisy images(cont.)


Periodic noisePeriodic noise

Spatially dependent noise Periodic noise can be

reduced significantly via frequency domain filtering


Estimation of noise parametersEstimation of noise parameters

PDF from small patches


When the only degradation is noiseWhen the only degradation is noise

Periodic noise subtraction gives a good result Random noise mean filter, order-statistics filter,…

),(),(),(

and

),(),(),(

vuNvuFvuG

yxyxfyxg


Mean filtersMean filters

Arithmetic mean filters For Gaussian or uniform noise

Geometric mean filters For Gaussian or uniform noise

Harmonic mean filters Work well for salt noise but fail for pepper noise

Contraharmonic mean filters Suited for impulse noise but require identification(salt

or pepper)


Arithmetic & Geometric mean filterArithmetic & Geometric mean filter


Contraharmonic filtersContraharmonic filters

Q<0 : eliminates salt noiseQ=-1 harmonic mean filter

Q=0 : arithmetic mean filter Q>0: eliminates pepper noise

xy

xy

Sts

Q

Sts

Q

tsg

tsg

yxf

),(

),(

1

),(

),(

),(ˆ


Eg. Contraharmonic filtersEg. Contraharmonic filters


Wrong sign in contraharmonic filtersWrong sign in contraharmonic filters

Disaster!


Order-Statistics filtersOrder-Statistics filters

Median filter Max filter Min filter Midpoint filter Alpha-trimmed mean filter


Median filtersMedian filters

33x3x3medianmedian

33x3x3medianmedian

33x3x3medianmedian

blurred


Max and Min filterMax and Min filter

• Max filter • Min filterRemoves pepper noiseRemoves dark pixels

Removes salt noiseRemoves light pixelsMakes dark objects larger


Eg. ComparisonEg. Comparison

(a) Additive uniformnoise

(b) (a)+additive S&P

5x5 arithmetic mean 5x5 geometric mean

5x5 median 5x5 alpha-trimmedMean(d=5)


Adaptive filtersAdaptive filters

Behavior changes locally based on statistical characteristics of local support

Simple adaptive filter based on mean and variance1. If global_var is zero, then f(x,y)=g(x,y)

2. If local_var>global_var, then f(x,y)=g(x,y) (high local var edge should be preserved)

3. If local_var==global_var, then arithmetic mean filtering


Eg. Adaptive filterEg. Adaptive filter


Adaptive median filterAdaptive median filter

Cope with impulse noise with large probability Preserve detail while smoothing non-impulse noise

Level A:A1=zmed-zmin

A2=zmed-zmax

If A1>0 AND A2<0, go to level BElse increase the window sizeIf window size<=Smax repeat level AElse output zxy

Level B:B1=zxy-zmin

B2=zxy-zmax

If B1>0 AND B2<0, output zxy

Else output zmed

Algorithm


Eg. Adaptive median filterEg. Adaptive median filter

median adaptive median


Periodic noise reductionPeriodic noise reduction

By frequency domain filtering Band reject filter


Eg. Periodic noise reductionEg. Periodic noise reduction


Noise extractionNoise extraction

By bandpass filter

Help understanding noise pattern


Notch filtersNotch filters


Eg. Notch filteringEg. Notch filtering

Removing sensor scan-line patterns


Optimum notch filteringOptimum notch filtering

First isolating the principal contributions of the interference pattern

Then subtracting weighted portion of the pattern from the corrupted image


Eg. Periodic interference(1/3)Eg. Periodic interference(1/3)

Noisy image



Extraction of noise interference pattern



Restored image by subtracting weighted portion of periodic interference (Refer to the derivation of weights in pp.250-252)


Linear, Position-Invariant DegradationLinear, Position-Invariant Degradation

),(),(),(),(

:DomainFrequency

),(),(),(),(

:Domain Spatial

vuNvuFvuHvuG

yxyxfyxhyxg


Degradation knowledgeDegradation knowledge

Degradation knowledge about 1. A priori (known)

2. A posteriori (unknown) blind restoration or blind deconvolution

fg

H

Restoration:Restoration:determine the original image ,

given the observed image and

knowledge about the degradation (H).


Fundamental issueFundamental issue

Restoration problem

restoration is to find , such that

but, 1. does not exist: singular

2. may exist, but not be unique: ill-conditioned

3. may exist and unique, but there exists ,

which can be made arbitrarily small, such that

which is not negligible

Image restoration is ill-conditioned at best and Image restoration is ill-conditioned at best and singular at worstsingular at worst

gfT }{1T fgT }{1

1T1T1T

,,}{1 fgT


Estimation of degradation functionEstimation of degradation function

Approaches

Observation

Experimentation

Mathematical modeling


Estimation by observationEstimation by observation

Looking at a small section of the image containing simple structures and then obtaining degradation function

Observed sub-image:

Estimate of original image:

),( yxg s),(ˆ yxf s

),(ˆ),(

),(vuF

vuGvuH

s

ss


Estimation by experimentationEstimation by experimentation

Possible only if equipment similar to the equipment used to acquire the degraded images is available

Eg. Use an impulse


Estimation by modelingEstimation by modeling Based on either physical characteristics or basic

principles

Eg.1. Physical characteristics: atmospheric turbulence

Eg.2. Math derivation: motion blur Starting from

After some manipulation(p.259)

Setting the motion model, we obtain the degradation func.

6/5)( 22

),( vukevuH

dtevuHT tvytuxj 0

)]()([2 00),(

T

dttyytxxfyxg0 00 )](),([),(


Eg.1. Physical modelEg.1. Physical model

Atmospheric turbulence


Eg.2. Math modelingEg.2. Math modeling

Motion blur


Restoration methodsRestoration methods

Inverse filtering Wiener filtering Constrained least square filtering Geometric mean filtering Etc..


Inverse filteringInverse filtering

Poor performance! Very sensitive to noise

),(),(),(),( vuNvuFvuHvuG

),(

),(),(),(ˆ

vuH

vuNvuFvuF

),(

),(),(ˆ

vuH

vuGvuF

Noise amplificationwhen H(u,v) is small


Eg. Inverse filteringEg. Inverse filtering


Minimum mean-square error filterMinimum mean-square error filter

Necessary to handle noise explicitly Statistical characteristics of noise should be

incorporated into the restoration process MMSE filter

To find an estimate of the uncorrupted image such that the mean square error between them is minimized:

Assume: the noise and the image are uncorrelated The one or the other has zero mean The gray levels in the estimate are a linear function of

the levels in the degraded image Derivation: Homework

})ˆ{( 22 ffEe

f̂ f


MMSE filter (cont.)MMSE filter (cont.)

Frequency domain expression:

Approximation of the Wiener filter

),(),(/),(|),(|

),(*

),(),(|),(|),(

),(),(*),(ˆ

2

2

vuGvuSvuSvuH

vuH

vuGvuSvuHvuS

vuSvuHvuF

f

f

f

Wiener filter

),(|),(|

|),(|

),(

1),(ˆ

2

2

vuGKvuH

vuH

vuHvuF

PS of noise PS of image f


Eg. Wiener filteringEg. Wiener filtering

Using the approximation K is chosen interactively


Eg. Restoration by Wiener filterEg. Restoration by Wiener filter motion blurmotion blur

Severe noise

Moderate noise

Negligible noise


Constrained Least Square FilteringConstrained Least Square Filtering

Difficulty in Wiener filter The power spectra of the undegraded image and noise

must be known Minimization in a statistical sense

The constrained LS filtering requires knowledge of

Mean of the noise Variance of the noise

Optimal result for each image


Vector-matrix form of convolutionVector-matrix form of convolution

g: MN-vector (lexicographical order of an image) f: MN-vector H: MNxMN matrix

ηHfg


Formulation: Constrained LS filterFormulation: Constrained LS filter

To find the minimum of a criterion function C defined as

subject to the constraint

where is the Euclidean vector norm

1

0

1

0

22 ),(M

x

N

y

yxfC

22 ||||||ˆ|| ηfHg

www T2||||


Freq. Domain Sol.Freq. Domain Sol.

: adjustable parameter

: Fourier transform of the Laplacian operator

),(|),(||),(|

),(* ),(ˆ

22vuG

vuPvuH

vuHvuF

),( vuP


Eg. Constrained LS filterEg. Constrained LS filter

Significant improvement over Wiener filter


Procedure for computing Procedure for computing

Define a residual vector

Adjust so that

Calculation

a 22 ||||||||)( ηr

fHgr ˆ

][|||| 222 mMN η

In general, automatically determined restoration filteryields inferior results to manual adjustment of filter parameters

iteration

Documents

CV Restoration