Upload
others
View
15
Download
0
Embed Size (px)
Citation preview
1
Image Restoration
Image Processing with Biomedical Applications
ELEG-475/675Prof. Barner
Image ProcessingImage Restoration
Prof. Barner, ECE Department, University of Delaware 2
Image Restoration
Image enhancement is subjective Heuristic and ad hoc
Image restoration is more theoretically motivated
A priori knowledge of image degradation utilizedOptimality criteria used to formulate restoration
Image ProcessingImage Restoration
Prof. Barner, ECE Department, University of Delaware 3
Preliminaries:Correlation and Power Spectrum
Note self-convolution
Correlation results if we do not reflect one term
Note maximum at Τ=0; also
Power Spectral Density (PSD)
Cross correlation and PSD
Assume ergotic signals (RVs) – interchange time/ensemble averages, e.g.,
( ) ( ) ( ) ( )f t f t f t f t dtτ∞
−∞∗ = −∫
( ) ( ) ( ) ( ) ( )fR f t f t f t f t dtτ τ∞
−∞= ∗ − = +∫
2
( ) ( )fR d f t dtτ τ∞ ∞
−∞ −∞
⎡ ⎤= ⎢ ⎥⎣ ⎦∫ ∫
{ } { } 2( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) | ( ) |f fP s R f t f t F s F s F s F s F sτ ∗= ℑ = ℑ ∗ − = − = =
( ) ( ) ( ) ( ) ( )fgR f t g t f t g t dtτ τ∞
−∞= ∗ − = +∫
( ) { ( )}fg fgP s R τ= ℑ
{ ( )} ( )x t x t dtε∞
−∞= ∫
Image ProcessingImage Restoration
Prof. Barner, ECE Department, University of Delaware 4
Degradation and Restoration Model
Degradation is taken to be a linear spatially invariant operator
( , ) ( , ) ( , ) ( , )g x y h x y f x y x yη= ∗ +
( , ) ( , ) ( , ) ( , )G u v H u v F u v N u v= +
2
Image ProcessingImage Restoration
Prof. Barner, ECE Department, University of Delaware 5
Noise Properties
Arises in acquisition, digitization, and transmission/storage processesCCD cameras are affected by:
Light levelsSensor temperatureBad sensors
Transmission noise can be due to interferenceWireless transmission interferenceLost networking packets
Statistics depend on sourceCommon assumption: White Spectrum
Correlation: R(Τ)=Aδ(Τ)Power spectrum: P(s)=A
Image ProcessingImage Restoration
Prof. Barner, ECE Department, University of Delaware 6
Noise Probability Density Functions
Gaussian (normal) PDF
Typically models electronics and sensor noiseRayleigh PDF
Skewed distribution typically models range imaging noise
2 2( ) / 21( )2
zp z e μ σ
πσ− −=
2( ) /2 ( ) for
0 for ( )
z a bz a e z ab
z ap z
− −− ≥
<
⎧= ⎨⎩
2 (4 )/ 4, 4
ba b πμ π σ −= + =
Image ProcessingImage Restoration
Prof. Barner, ECE Department, University of Delaware 7
Noise Probability Density Functions II
Gamma PDF
Exponential PDF
Both appropriate for laser imagingHeavy-tailed distributions
samples contain frequent outliers
1( ) for 0
( 1)!
0 for 0( )
b baza z z a e z
b
zp z
−−− ≥
−
<
⎧⎪= ⎨⎪⎩
22, b b
a aμ σ= =
{ for 00 for 0( )
azae zzp z
− ≥<=
22
1 1, a a
μ σ= =
Image ProcessingImage Restoration
Prof. Barner, ECE Department, University of Delaware 8
Noise Probability Density Function III
Uniform PDF
Impulsive (salt and pepper) PDF
Shot or spike notice appropriate faulty sensor or electronics, transmission error/drop
1 if
0 otherwise( )
a z bb ap z
≤ ≤−
⎧= ⎨⎩
22 ( ),
2 12a b b aμ σ+ −
= =
for ( ) for
0 otherwise
a
b
P z ap z P z b
=⎧⎪= =⎨⎪⎩
3
Image ProcessingImage Restoration
Prof. Barner, ECE Department, University of Delaware 9
PDF Plots
Gaussian distribution is most widely used
Central Limit TheoremDesirable propertiesIndependence and correlated
Other distributions appropriate for specific cases
Simplicity of uniform enables derivation of results
Image ProcessingImage Restoration
Prof. Barner, ECE Department, University of Delaware 10
Test Pattern Corruption Example
Image ProcessingImage Restoration
Prof. Barner, ECE Department, University of Delaware 11
Test Pattern Corruption Example II
Image ProcessingImage Restoration
Prof. Barner, ECE Department, University of Delaware 12
Noise Parameter Determination
Generate statistics from uniform regionHistogram matchingQuantile-Quantile (Q-Q) plot
Determined defining statistics, e.g.2 2( ), ( ) ( )
i i
i i i iz S z S
z p z z p zμ σ μ∈ ∈
= = −∑ ∑
4
Image ProcessingImage Restoration
Prof. Barner, ECE Department, University of Delaware 13
Q-Q Plot ExampleAre samples x1, x2,…, xN, and y1, y2,…, yN governed by the same distribution?
Order the samples: x(1), x(2),…, x(N), and y(1), y(2),…, y(N)
Plot (x(1), y(1)), (x(2), y(2)),…,(x(N), y(N))Samples governed by the same distribution will lie (approximately) along a line
Image ProcessingImage Restoration
Prof. Barner, ECE Department, University of Delaware 14
Additive Noise Degradation Case
Gaussian case: mean (averaging) filter is ML optimal (previously derived)Laplacian (double exponential) case: median filter is ML optimal (previously derived)Both assume iid noise, constant signal
See book for Sample filtering resultsAdditional simple spatial filters
We consider arbitrary noise PSD
Image ProcessingImage Restoration
Prof. Barner, ECE Department, University of Delaware 15
The Wiener Estimator
Assume no channel distortionDerived in one dimension for simplicityMean Square Error (MSE) optimality criteria:
h(t) ( )y t( )s t ( )x t
( )n t
2 2MSE { ( )} ( )
( ) ( ) ( )
e t e t dt
e t s t y t
ε∞
−∞= =
= −∫
Image ProcessingImage Restoration
Prof. Barner, ECE Department, University of Delaware 16
Rearrangement of MSE
Calculus of variations approach:Obtain a functional expression of MSE in terms of h(t)Obtain an expression for the optimal impulse response, ho(t), in terms of known power spectraDevelop an expression for the MSE that results when ho(t) is used
Rewriting the MSE:2 2 2 2
2 21 2 3
MSE { ( )} {[ ( ) ( )] } { ( ) 2 ( ) ( ) ( )}MSE { ( )} 2 { ( ) ( )} { ( )}
e t s t y t s t s t y t y ts t s t y t y t T T T
ε ε ε
ε ε ε
= = − = − +
= − + = + +
5
Image ProcessingImage Restoration
Prof. Barner, ECE Department, University of Delaware 17
MSE Components
Consider the first component:
Express the second component by writing the output as a convolution
2 21 { ( )} ( ) (0)sT s t s t dt Rε
∞
−∞= = =∫
{ }{ }
2
2
2
2 ( ) ( ) ( )
2 ( ) ( ) ( )
2 ( ) ( )xs
T s t h x t d
T h s t x t d
T h R d
ε τ τ τ
τ ε τ τ
τ τ τ
∞
−∞
∞
−∞
∞
−∞
= − −
= − −
= −
∫
∫∫
Image ProcessingImage Restoration
Prof. Barner, ECE Department, University of Delaware 18
MSE Components II
Express the third component as the product of two convolutions:
utilizing change of variables: v=t-u
which is the autocorrelation evaluated at u-Τ
Final result:
Function of: filter impulse response, known autocorrelation and crosscorrelation
{ }{ }
3
3
( ) ( ) ( ) ( )
( ) ( ) ( ) ( )
T h x t d h u x t u du
T h h u x t x t u d du
ε τ τ τ
τ ε τ τ
∞ ∞
−∞ −∞
∞ ∞
−∞ −∞
= − −
= − −
∫ ∫
∫ ∫
{ } { }( ) ( ) ( ) ( )x t x t u x v u x vε τ ε τ− − = + −
3 ( ) ( ) ( )xT h h u R u d duτ τ τ∞ ∞
−∞ −∞= −∫ ∫
MSE (0) 2 ( ) ( ) ( ) ( ) ( )s xs xR h R d h h u R u d duτ τ τ τ τ τ∞ ∞ ∞
−∞ −∞ −∞= − + −∫ ∫ ∫
Image ProcessingImage Restoration
Prof. Barner, ECE Department, University of Delaware 19
MSE Minimization (I)
Define arbitrary impulse response:
inserting into MSE expression
First three terms: optimal MSE (MSEo)Autocorrelation Rx(u-Τ) is an even function
Fourth and fifth terms are equalCombine fourth, fifth, and sixth terms
( ) ( ) ( )oh t h t g t= +
MSE (0) 2 [ ( ) ( )] ( ) [ ( ) ( )][ ( ) ( )] ( )s o xs o o xR h g R d h g h u g u R u d duτ τ τ τ τ τ τ τ∞ ∞ ∞
−∞ −∞ −∞= − + + + + −∫ ∫ ∫
MSE (0) 2 ( ) ( ) ( ) ( ) ( )
( ) ( ) ( ) ( ) ( ) ( )
2 ( ) ( ) ( ) ( ) ( )
s o xs o o x
o x o x
xs x
R h R d h h u R u d du
h g u R u d du h u g R u d du
g R d g g u R u d du
τ τ τ τ τ τ
τ τ τ τ τ τ
τ τ τ τ τ τ
∞ ∞ ∞
−∞ −∞ −∞
∞ ∞ ∞ ∞
−∞ −∞ −∞ −∞
∞ ∞ ∞
−∞ −∞ −∞
= − + −
+ − + + −
− + −
∫ ∫ ∫∫ ∫ ∫ ∫∫ ∫ ∫
Image ProcessingImage Restoration
Prof. Barner, ECE Department, University of Delaware 20
MSE Minimization (II)
Show T5 is nonnegativeExpanding autocorrelation
Define z(t)= g(t)*x(t)
Independent of ho
4 5
MSE MSE 2 ( ) ( ) ( ) ( ) ( ) ( ) ( )
MSE
o o x xs x
o
g u h R u d R u du g g u R u dud
T T
τ τ τ τ τ τ∞ ∞ ∞ ∞
−∞ −∞ −∞ −∞
⎡ ⎤= + − − + −⎢ ⎥⎣ ⎦= + +
∫ ∫ ∫ ∫
5
5
( ) ( ) ( )
( ) ( ) ( ) ( )
T g g u x t dtdud
T g x t u du g x t dtd
τ τ τ
τ τ τ τ
∞ ∞ ∞
−∞ −∞ −∞
∞ ∞ ∞
−∞ −∞ −∞
= −
= − −
∫ ∫ ∫∫ ∫ ∫
25 ( ) 0T z t dt
∞
−∞= ≥∫
6
Image ProcessingImage Restoration
Prof. Barner, ECE Department, University of Delaware 21
MSE Minimization (III)
Expression to minimize
Necessary and sufficient condition:Make term in brackets 0 for all u
Note that for linear systems: [equation 48]Since the above is a convolution
In terms of PSD
5MSE MSE 2 ( ) ( ) ( ) ( )o o x xsg u h R u d R u du Tτ τ τ∞ ∞
−∞ −∞
⎡ ⎤= + − − +⎢ ⎥⎣ ⎦∫ ∫
( ) ( ) ( )xs o xR h u R u duτ τ∞
−∞= −∫
( ) ( ) ( ) ( )xs o x xyR h u R u Rτ τ= ∗ =
( ) ( ) ( ) ( )xs o x xyP s H s P s P s= =( )( )( )
xso
x
P sH sP s
=
Image ProcessingImage Restoration
Prof. Barner, ECE Department, University of Delaware 22
Wiener Filter Design
To determine
Obtain an observation signal sample (x(t))Determine autocorrelation and PSD
Obtain a signal sampled without noise (s(t))Cross correlate x(t) and s(t); determine PSD
Implement filter in the frequency domain or take inverse transform to get impulse response
If statistics are unknown or can’t be measured, assume functional form
Example: white PSD
( )( )( )
xso
x
P sH sP s
=
Image ProcessingImage Restoration
Prof. Barner, ECE Department, University of Delaware 23
Uncorrelated Signal and Noise Case
In this case: Crosscorrelation:
Autocorrelation:
Optimal filter:
{ } { } { }( ) ( ) ( ) ( )s t n t s t n tε ε ε=
{ } [ ]{ }( ) ( ) ( ) ( ) ( ) ( )xsR x t s t s t n t s tτ ε τ ε τ= + = + +
{ } { }( ) ( ) ( ) ( ) ( )xsR s t s t n t s tτ ε τ ε τ= + + +
{ } { }( ) ( ) ( ) ( ) ( ) ( ) ( )xs s sR R n t s t R n t dt s t dτ τ ε ε τ τ τ τ∞ ∞
−∞ −∞= + + = + +∫ ∫
( ) ( ) (0) (0)xs sR R N Sτ τ= +
( ) ( ) ( ) 2 (0) (0)x s nR R R N Sτ τ τ= + +
( ) (0) (0) ( )( )( ) ( ) 2 (0) (0) ( )
so
s n
P s N S sH sP s P s N S s
δδ
+=
+ +( )( )
( ) ( )s
os n
P sH sP s P s
=+
Image ProcessingImage Restoration
Prof. Barner, ECE Department, University of Delaware 24
Analysis of Optimal MSE
Recall: For the uncorrelated case (see previous)
Further assuming zero means
Substituting for the optimal filter (even function)
MSE (0) ( ) ( )o s o xsR h R dτ τ τ∞
−∞= − ∫
( ) ( ) (0) (0)xs sR R N Sτ τ= +
{ }1MSE (0) ( ) ( )o s o sR h P s dτ τ∞ −
−∞= − ℑ∫
2MSE (0) ( ) ( ) j so s s oR P s h e d dsπ ττ τ
∞ ∞
−∞ −∞= − ∫ ∫
MSE ( ) ( ) ( )o s s oP s ds P s H s ds∞ ∞
−∞ −∞= − −∫ ∫
( )MSE ( ) ( )( ) ( )
so s s
s n
P sP s ds P s dsP s P s
∞ ∞
−∞ −∞= −
+∫ ∫( ) ( )MSE ( ) ( )
( ) ( )s n
o n os n
P s P s ds P s H s dsP s P s
∞ ∞
−∞ −∞= =
+∫ ∫
7
Image ProcessingImage Restoration
Prof. Barner, ECE Department, University of Delaware 25
Wiener Filter Examples (I)
The Wiener filter transfer function Bandlimited signal
Image ProcessingImage Restoration
Prof. Barner, ECE Department, University of Delaware 26
Wiener Filter Examples (II)
Separable signal and noise case
MSE is zero
Image ProcessingImage Restoration
Prof. Barner, ECE Department, University of Delaware 27
Wiener Deconvolution
Combine deconvolution (system inversion) and noise filteringCombination of two linear operations
Spectrum of observed signal: Input to Wiener filter: For uncorrelated signal and noise (previous result)
Wiener (MSE optimal) deconvolution filter:
( ) ( ) ( ) ( )X s F s S s N s= +( )( ) ( ) ( ) ( )( )
N sY s S s S s K sF s
= + = +
2
22
( ) | ( ) |( )( ) ( ) ( )| ( ) |
( )
so
s k
P s S sH sP s P s N sS s
F s
= =+
+
( ) ( ) ( ) ( )1( )( ) ( ) ( ) ( ) | ( ) | ( ) ( )o s s
s k s n
H s P s F s P sG sF s F s P s P s F s P s P s
∗⎡ ⎤= = =⎢ ⎥+ +⎣ ⎦
( )F s1( )F s
( )oH s( )s t ( )w t ( )x t
( )n t
( )y t ( )z t
( )G s
Image ProcessingImage Restoration
Prof. Barner, ECE Department, University of Delaware 28
Wiener Deconvolution Examples
Wiener deconvolutionexample: (a) signal and noise power spectra; (b) blurring function; (c) Wiener deconvolutiontransfer function
Example of two-dimensional Wiener deeconvolution: (a) signal power spectrum; (b) noise power spectrum; (c) blurring function; (d) transfer function
8
Image ProcessingImage Restoration
Prof. Barner, ECE Department, University of Delaware 29
Wiener Generalization (I)
The two-dimensional Wiener filter:
Often used approximation:
2
2
1 | ( , ) |ˆ ( , ) ( , )( , ) | ( , ) | ( , ) / ( , )f
H u vF u v G u vH u v H u v S u v S u vη
⎡ ⎤= ⎢ ⎥
+⎢ ⎥⎣ ⎦
2
2
1 | ( , ) |ˆ ( , ) ( , )( , ) | ( , ) |
H u vF u v G u vH u v H u v K⎡ ⎤
= ⎢ ⎥+⎣ ⎦
Image ProcessingImage Restoration
Prof. Barner, ECE Department, University of Delaware 30
Wiener Generalization (II)
Geometric Mean Filter generalization:
Special casesα=1: inverse filterα=0: parametric of Wiener filter (standard if β=1)α=1/2; β=1: referred to as the Spectrum equalization filter
The product of two quantities raised to the same powerDefinition of the geometric mean
Trade-off between inverse filtering and Wiener filtering controlled by α
1
22
( , ) ( , )ˆ ( , ) ( , )| ( , ) | ( , )
| ( , ) |( , )f
H u v H u vF u v G u vH u v S u v
H u vS u v
α
α
ηβ
−
∗ ∗
⎡ ⎤⎢ ⎥
⎡ ⎤ ⎢ ⎥= ⎢ ⎥ ⎢ ⎥⎡ ⎤⎣ ⎦ ⎢ ⎥+ ⎢ ⎥⎢ ⎥⎢ ⎥⎣ ⎦⎣ ⎦
Image ProcessingImage Restoration
Prof. Barner, ECE Department, University of Delaware 31
Degradation Function Estimation
Recall: In the no (low) noise case
Estimation by observation:
Estimation by experimentation:Input impulse of strength A
Estimation by modeling:Example: atmospheric interference
( , ) ( , ) ( , ) ( , )G u v H u v F u v N u v= +
( ,. )( , ) ˆ ( , )s
ss
G u vH u vF u v
=
( , )( , ) G u vH u vA
=
2 2 5 / 6( )( , ) k u vH u v e− +=
Image ProcessingImage Restoration
Prof. Barner, ECE Department, University of Delaware 32
Estimation by Experiment Example
Experimentally determined point spread function (PSF)
9
Image ProcessingImage Restoration
Prof. Barner, ECE Department, University of Delaware 33
Modeling Image Motion
Systems most often consider motionCamera and/or subject motion
If T is the exposure duration:
Blurred image: g(x,y)Planar motion trajectories: xo(t) and yo(t)
Fourier transform evaluation yields
where
[ ]0 00( , ) ( ), ( )
Tg x y f x x t y y t dt= − −∫
( , ) ( , ) ( , )G u v H u v F u v=
[ ]0 02 ( ) ( )
0( , )
T j ux t vy tH u v e dtπ− += ∫Image ProcessingImage Restoration
Prof. Barner, ECE Department, University of Delaware 34
Motion Example
If xo(t)=at/T and yo(t)=0
Two dimensional linear (xo(t)=at/T and yo(t)=bt/T) motion
02 ( ) 2 /
0 0( , ) sin( )
T Tj ux t j uat T j uaTH u v e dt e dt ua eua
π π πππ
− − −= = =∫ ∫
[ ] ( )( , ) sin ( )( )
j ua vbTH u v ua vb eua vb
πππ
− += ++
Image ProcessingImage Restoration
Prof. Barner, ECE Department, University of Delaware 35
Atmospheric Turbulence Degradation
Image ProcessingImage Restoration
Prof. Barner, ECE Department, University of Delaware 36
Inverse Filtering Example
Direct application of
Amplifies noisePossible solution:
Limit cutoff frequency
( , )ˆ ( , )( , )
G u vF u vH u v
=
10
Image ProcessingImage Restoration
Prof. Barner, ECE Department, University of Delaware 37
Inverse and Wiener Filtering Comparison
Wiener parameter K set experimentallyWiener result much sharper than the band limited inverse filterFull inverse filter results is useless
Image ProcessingImage Restoration
Prof. Barner, ECE Department, University of Delaware 38
Motion Blur with Additive Noise Example
Image ProcessingImage Restoration
Prof. Barner, ECE Department, University of Delaware 39
Mean in Geometric Mean Example
Image ProcessingImage Restoration
Prof. Barner, ECE Department, University of Delaware 40
Multiple Applications of the Median
11
Image ProcessingImage Restoration
Prof. Barner, ECE Department, University of Delaware 41
Mixed NoiseExample
Image corrupted byUniform noiseUniform and salt and pepper noise
Filtering methodsArithmetic meanGeometric meanMedianAlpha-trimmed mean
Image ProcessingImage Restoration
Prof. Barner, ECE Department, University of Delaware 42
Mean, Arithmetic Mean, and Adaptive Approach
Image ProcessingImage Restoration
Prof. Barner, ECE Department, University of Delaware 43
Median and Adaptive Median
Image ProcessingImage Restoration
Prof. Barner, ECE Department, University of Delaware 44
Periodic Interference
Band reject filtersStraightforward extension from a high/low pass case
Ideal, Butterworth, Gaussian
12
Image ProcessingImage Restoration
Prof. Barner, ECE Department, University of Delaware 45
Sinusoidal Corruption Example
Image ProcessingImage Restoration
Prof. Barner, ECE Department, University of Delaware 46
Periodic Scan LineInterference
Note horizontal scan line “striping”
Vertical spectral comment
Notch pass filter and spatial interferenceImage with interference rejected
Image ProcessingImage Restoration
Prof. Barner, ECE Department, University of Delaware 47
NASA Image, Spectrum, and Filtering Result