Spatial Domain LinearSpatial Domain Linear Filteringeeweb.poly.edu/~yao/EL5123/lecture5_filtering.pdf · Spatial Domain LinearSpatial Domain Linear Filtering Yao Wang ... Averaging

Spatial Domain LinearSpatial Domain Linear Filtering

Yao WangPolytechnic University, Brooklyn, NY 11201

With contribution from Zhu Liu, Onur Guleryuz, andGonzalez/Woods, Digital Image Processing, 2ed

Outline

• IntroductionNoise remo al sing lo pass filters• Noise removal using low-pass filters

• Sharpening by edge enhancement– Edge detection using high-pass filters– Edge enhancement by high emphasis filters

• Edge detection– First order gradientg– Second order gradient

• Summary

Yao Wang, NYU-Poly EL5123: Spatial Filtering 2

• Summary

Problems

• SmoothingNoise removal– Noise removal

– Detail preserving image smoothingSh i• Sharpening– Edge enhancement– Detail focusing

• Edge detection


Approaches

• Spatial domain operation or filtering (the processed value for the current pixelprocessed value for the current pixel depends on both itself and surrounding pixels)pixels)– Linear filtering

Non linear filtering– Non-linear filtering• Rank order filtering including median• Morphological filtering• Morphological filtering

– Adaptive filtering


Noise Removal (Image Smoothing)• An image may be “dirty” (with dots, speckles,stains)• Noise removal:

– To remove speckles/dots on an image– Dots can be modeled as impulses (salt-and-pepper or speckle)

or continuously varying (Gaussian noise)or continuously varying (Gaussian noise)– Can be removed by taking mean or median values of

neighboring pixels (e.g. 3x3 window)Equivalent to low pass filtering– Equivalent to low-pass filtering

• Problem with low-pass filtering– May blur edges– More advanced techniques: adaptive, edge preserving


Example


Averaging Filter: An Intuitive Approach

• Replace each pixel by the average of pixels in a square window surrounding thispixels in a square window surrounding this pixel

)1,1(),1()1,1((91),( nmfnmfnmfnmg

))1,1(),1()1,1()1,(),()1,(

9

nmfnmfnmfnmfnmfnmf

• Trade-off between noise removal and detail preserving:

)),(),(),( fff

detail preserving:– Larger window -> can remove noise more

effectively, but also blur the details/edges


y, g

Example: 3x3 average

100 100 100 100 100

100 200 205 203 100100 200 205 203 100

100 195 200 200 100

100 200 205 195 100

100 100 100 100 100

100 100 100 100 100100 144 167 145 100100 167 200 168 100100 167 200 168 100

100 144 166 144 100100 100 100 100 100


Example


Weighted Averaging Filter• Instead of averaging all the pixel values in the

window, give the closer-by pixels higherwindow, give the closer by pixels higher weighting, and far-away pixels lower weighting.

)(*)()()()(1 1

nmhnmflnkmflkhnmgk l

• This type of operation is in fact 2-D linear

),(*),(),(),(),(0 0

nmhnmflnkmflkhnmgkk ll

This type of operation is in fact 2 D linear convolution of f(m,n) by a filter h(m,n).

• Weighted average filter retains low frequency g g q yand suppresses high frequency = low-pass filter


Example Weighting Mask

1 1 1 1 2 1

1 1 1

1 1 1

2 4 2

1 2 1

91

161

1 1 1 1 2 1

All weights must sum to one


Example: Weighted Average

100 100 100 100 100100 200 205 203 100100 195 200 200 100100 200 205 195 100100 100 100 100 100 1 2 1

11 1 1

100 100 100 100 1002 4 21 2 1

161

1 1 11 1 1

91

100 100 100 100 100100 156 176 158 100

100 100 100 100 100100 144 167 145 100

100 174 201 175 100100 156 175 156 100100 100 100 100 100

100 167 200 168 100100 144 166 144 100100 100 100 100 100


100 100 100 100 100100 100 100 100 100

Example

Original image Average filtered image Weighted Average filtered image


Symmetrical Filter• The smoothing filter is generally symmetrical in both

horizontal and vertical directions, as well as diagonal.

• Simplified calculation.),(),(),(),( nmhnmhnmhnmh

k lHhk=0

-W

),(*),(),(),(),(1

0

1

0

nmhnmflnkmflkhnmgk

kk

l

ll

0

W-W Wl=0

W

nlmfnlmflnmflnmflh

nmfhnmgWllkk 0101

))()()()()(0(

),()0,0(),(

W

k

W

l

l

lnkmflnkmflnkmflnkmflkh

nlmfnlmflnmflnmflh

1 1

1

)),(),(),(),()(,(

)),(),(),(),()(,0(


k l1 1

Computation Cost• General (2W+1) by (2W+1) filter

– (2W+1)2 multiplications per processed pixel(2W+1) multiplications per processed pixel

W

k

W

llnkmflkhnmg ),(),(),(

• Symmetrical filter– 1+W+W2 ~=(W+1)2 multiplications

Wk Wl

1+W+W (W+1) multiplications• Saving

– W = 2 and 512x512 imageW 2, and 512x512 image– Saving 512*512*(52-32)=222

– More savings for larger W


More savings for larger W

Common Smoothing Filters

11121111 b.1,

11

11

)2(1;

121242121

161;

111121111

101 2

2

bwith

bbbb

b

b

• Are any of these separable?

11121111 b

Are any of these separable?• Criteria for designing a smoothing filter

h(k l)≥0 (function as averaging)– h(k,l)≥0 (function as averaging)– (the mean value is preserved)1),(

1

0

1

0

k

kk

l

lllkh


Smoothing Filter is a Low-pass Filter• The 2D Fourier Transform of a smoothing filter

• Image signal usually varies slowly, and noise is usually a wide band signal.A l i l filt th hi h f t f th• Applying a low pass filter removes the high-frequency part of the noise.

• Image edges and other details are blurred


Interpretation in Freq Domain

Filter response

Low-passed image spectrum

Filter response

Noise spectrum

Original image spectrum

Noise spectrum

f0


Spectrum of Example Low Pass Filterrs

;11111

91111

111

91H

22

2 ;111

)2(1

11

)2(1 bb

bbbb

b

bH

9/)2cos(21)2cos(21),(1

9111

9

vuvuH

2)2/()2cos(2)2cos(2),(

1)2(11)2(

bvbubvuH

bbb

11

0.2

0.4

0.6

0.8

-0.5

0

0.5

05

1015

2025

0

10

20

300

05

1015

2025

0

10

20

30


b=2

Directional Smoothing• Smoothing filter tend to blur details• Challenging problem: detail preserving noise removal!

• Di the i-th directional neighborhood• Usually 4 or 8 directions D

3

y• 3 core steps

– Find σi2, the variance in Di

– Find i0, so that σi02 ≤σi

2D1

Find i0, so that σi0 σi

– Filtering in i0 direction

ioDlk

io

lnkmfN

nmg),(

),(1),(

})sincosexp{(),(

),(*),(),(2

0

iii

i

nmanmh

nmhnmfnmg


})p{(),( iii

Noise Removal by Averaging Multiple ImagesImages


Noise Removal by Averaging Multiple ImagesImages

• Assume y = x + n, where n ~ N(0, σ2).• Observe one image, the noise is n• Observe N images: yi = x + ni i=1 N where ni ~ N(0 σ2) and niObserve N images: yi x ni, i 1,…,N, where ni N(0, σ ), and ni

are independent.

NNN

nxnxyy 1)(11

i

ii

ii

i nN

xnxN

yN

y111

)(

)/,0(~, 2 NNnwherenxy

• Averaging over N images reduces the noise variance by 1/N.


Edge Detection

• What is an edge?Difference bet een edge and line and• Difference between edge and line and point


Characterization of EdgesIdeal step edge

Real edge has a slope

First order derivative:Maximum at edge location

Second order derivative:Second order derivative:Zero crossing at edge location


Edge Detection Based on First Order DerivativesDerivatives

• Edge High-pass filtering across edgeLow-pass filtering along edge

111

111h

Non-Edge

g Yes

No

Non Edge

H |g|>Tf g Yes Edge


What if we don’t know edge direction?

Edge Detection Based on Gradients in Two Orthogonal DirectionsTwo Orthogonal Directions

• Combine results from directional edge detectors in two orthogonal directions anddetectors in two orthogonal directions and determine the magnitude and direction of the edgethe edge.

H f 2+f 2 |g|>Tg

fx Edge

Non-Edge

Hx fx +fy |g|>T

f

Hy tan-1(fx/fy) theta

f


Hy tan (fx/fy)fy

Directional Edge Detector • High-pass (or band-pass) in one direction (simulating

first order derivative)• Low pass in the orthogonal direction (smooth noise)• Prewitt edge detector LP BP

101111

31

101101101

31;111

101

31

111000111

31

yx HH

• Sobel edge detector11011111

11011121 101

121

41

101202101

41;121

101

41

121000121

41

yx HH


The sobel filter provides better smoothing along the edge

Freq. Response of Sobel Filter

1121:Edges Horizontalfor Filter Sobel

H x

(DTFT)RF

121101

41

121000121

41

Hh

ujeeHh

vjvj

ujujxx

211211]2[112sin2- 1] 0 [-1

:(DTFT)ResponseFrequency

22

22

veeHh vjvjyy 2cos1

22

4- 1] 2 [1

422

|Hx(u)| |Hy(v)|


0 1/4 1/2 u 0 1/4 1/2 v

Band-pass Low-pass

Spectrum of the Sobel Filter

10

2040

2040

0

5

xH

020

020

10

0

5

10

yH

020

40

020

40

L l th d b d th d


Low pass along the edge, band pass cross the edge

Example of Sobel Edge Detector

Original image Filtered image by H Filtered image by HyOriginal image Filtered image by Hx Filtered image by Hy


How to set threshold?

• Trial and errorAccording to edge magnit de distrib tion• According to edge magnitude distribution– E.g assuming only 5% pixels should be edge

pixels then the threshold should be the 95%pixels, then the threshold should be the 95% percentile of the edge magnitudeIllustrate on board– Illustrate on board


gy gmgx gy gmgx

T=100 T=50 T=20

2000

2500

Histogram of gm

500

1000

1500

0

0 100 200


Problems of previous approach• Cannot locate edges precisely• Ramp edges can lead to many edge pixels• Ramp edges can lead to many edge pixels

detected depending on the threshold T– T too high: may not detect weak edgesg y g– T too small: detected edges too think, noisy points

falsely detected• Remedy:

– Detecting local maximum of |g| in the normal direction of the edge or try all possible 8 direction in a 3x3of the edge, or try all possible 8 direction in a 3x3 neighbor

– Only consider pixels with |g| > T


y p |g|

Edge Detection with Many Directional FiltersFilters

• Instead of using two orthogonal directions, can design multiple directional filterscan design multiple directional filters– 0, 45, 90, 135

S hi h i th hi h t• See which one gives the highest response in the normal direction


Edge Detection Based on Second Order DerivativeDerivative

• Convolve an image with a filter corresponding to taking second ordercorresponding to taking second order derivative (e.g. Laplacian or LoG operator)Locate zero crossing in the filtered image• Locate zero-crossing in the filtered image


Laplacian Operator

2

2

2

22

yf

xf

f

),(),1(

2 f

yxfyxfxf

)1,(),(2)1,(

),1(),(2),1(

2

2

2

yxfyxfyxff

yxfyxfyxfxf

),(4)1,()1,(),1(),1(

)1,(),(2)1,(

2

2

yxfyxfyxfyxfyxf

yxfyxfyxfy

f

;010141

010;

010141010


010010

Fourier Transform of Laplacian Operator

141010

181111

010

141

111181

6

8

10

12

20

300

2

4

300

2

4

6

8

10

05 10

1520 25

0

10

05

1015

2025

0

10

20


More isotropic

Isotropic Edge Detector

• Laplacian operator can detect changes in all directionsall directions


Pros and Cons

• Can locate edges more accuratelyCan detect edges in ario s direction• Can detect edges in various direction

• But more prone to noise• Remedy:

– Smoothing before applying Laplaciang pp y g p


Laplacian of Gaussian (LoG)

• To suppress noise, smooth the signal using a Gaussian filter firstusing a Gaussian filter first– F(x,y)* G(x,y)

Th l L l i filtG• Then apply Laplacian filter– F(x,y)*G(x,y)*L(x,y) = F(x,y)* (L(x,y)*G(x,y))

G

• Equivalent filter: LoG– H(x,y)=L(x,y)*G(x,y)


Derivation of LoG Filter• Analog form

22 yx

2

22

2

222222

2

2

),(

yx

y

yxGG

eyxG

22422

2 2),(

eyx

yG

xGyxG

• Take samples to create filter maskSi f k > 6 dd– Size of mask nxn, n>=6, odd

– Exn


LoG Filter


Laplacian filtered LOG filtered

Note that each strong edge in the original image corresponds to a thin stripe g g g g p pwith high intensity in one side and low intensity in the other side.


How to detect zero crossing?

• For each pixel that has low filtered value, check a 3x3 neighbor to see whether itscheck a 3x3 neighbor, to see whether its two neighbors in opposite direction have opposite sign and their difference exceedsopposite sign and their difference exceeds a threshold (Marr-Hildreth edge detection method)method)


Example


Summary of Edge Detection Method• First order gradient based:

– Using edge detectors in two orthogonal directionsUsing edge detectors in two orthogonal directions• For each direction: low-pass along edge, high-pass across

edge

Using edge detectors in multiple (>2) directions– Using edge detectors in multiple (>2) directions– Use threshold or detect local maximum across the

edge directiong• Second order gradient based

– Laplacian is noise-prone, LoG is betterp p– Detect zero crossing– Isotropic


More on Edge Detection• Methods discussed so far generally cannot yield

connected thin edge maps• Need sophisticated post processing

– Thinning– Connecting broken linesConnecting broken lines

• Noise can lead to many false edge points• Even with many years of research, no perfect

d d t t i t!edge detectors exist!– Canny edge detector: Gaussian smoothing along

edges, high pass in each possible edge direction• For more on edge detection, See Gonzalez Sec.

10.1, 10.2


Results using MATLAB “edge( )” functionSobel,T=0.14 LOG, T=0.0051 canny,T=[0.0313,0.0781]

Sobel,T=0.1 LOG, T=0.01 canny,T=[0.1,0.15]


Image Sharpening• Sharpening : to enhance line structures or other

details in an image• Enhanced image = original image + scaled

version of the line structures and edges in the imageimage

• Line structures and edges can be obtained by applying a high pass filter on the imageapplying a high pass filter on the image

• Combined operation is still a weighted averaging operation, but some weights can be negative.

• In frequency domain, the filter has the “high-emphasis” character


Interpretation in Freq DomainFilter response (high emphasis)

sharpened image spectrum

Original image spectrum

f0


Designing Sharpening Filter Using High Pass FiltersPass Filters

• The desired image is the original plus an appropriately scaled high-passed imageappropriately scaled high-passed image

• Sharpening filter

hs fff

),(),(),( nmhnmnmh hs


Interpretation in Freq Domainhigh emphasis=allpass+highpass

all pass

high pass

f0


How to design high pass filter• High pass filter tries to detect changes • Difference between a current pixel and its• Difference between a current pixel and its

neighbors– Some coefficients should be negative g

• Output should be zero if the image is flat

k l

0),(1

0

1

0

k

kk

l

ll

lkh

• More rigorous design would start with desired frequency response and take inverse FT.

0 0kk ll


frequency response and take inverse FT.

Using Laplacian Operator as Highpass FilterFilter

141010

181111

010

141

111181

6

8

10

12

20

300

2

4

300

2

4

6

8

10

05 10

1520 25

0

10

05

1015

2025

0

10

20


More isotropic

Examplef(x) fs(x)=f(x)+ag(x)g(x)=f(x)*hh(x)

x xx

0101

0101

.1010181

41

010141

41

withHH sh

.11161111

81181

111

81

withHH sh


1118

1118

Example of Sharpening Using Laplacian OperatorOperator

010

010181

010

41

hH


Example of Sharpening Using Laplacian OperatorOperator

4

8


Designing Sharpening Filtering Using Low Pass Filters (Unsharp Masking)( p g)

• The sharpened = original – a*lowpass

1


10,1

1

ls fff


1 ),(),(1

1),( nmhnmnmh ls


Interpretation in Freq Domainhigh emphasis=allpass- a*lowpass

all pass

low pass

f0

low pass


Example

f(x) fl(x)=f(x)*hl(x) fs(x)=f(x)- fl(x)

x x x

1121

1121

1

.41

1212602

481

121242

161

withHH sl

Criteria for designing sharpening filters: There are both positive and negativeCoefficients h(0 0) > 0 and ∑h(k l)=1 (to keep the same mean value)


Coefficients, h(0,0) > 0, and ∑h(k, l) 1 (to keep the same mean value).

Example

Original image Sharpened image


Challenges of Noise Removal and Image Sharpeningp g

• How to smooth the noise without blurring the details too much?the details too much?

• How to enhance edges without amplifying noise?noise?

• Still a active research area– Wavelet domain processing


Wavelet-Domain Filtering


Courtesy of Ivan Selesnick

Feature Enhancement by SubtractionSubtraction

Taking an image without injecting a contrast agent first. Then take the image again after the organ is injected some special contrast agent (which go into the bloodstreams only). Then subtract the two images --- A popular technique in medical imaging


Then subtract the two images --- A popular technique in medical imaging

Summary• Noise removal using low-pass filters

– Averaging and weighted averaging filter• Edge detection

– Using isotropic high-pass filters: eg. 2D Laplacian– Using two orthogonal high-pass filters: eg. Sobel– Using many directional high-pass filters

• Image sharpening by high emphasis filters– High pass filtering using Laplacian operatorsHigh pass filtering using Laplacian operators– Sharpening using high pass filter (low pass+ a* high pass)– Sharpening using low pass filter (unsharp masking)(original-

a*lowpass)p )• Given a filter, you should be able to tell what does it do

(smoothing, edge detection, sharpening?) by looking at its coefficients and also through DTFT


its coefficients, and also through DTFT

Reading

• R. Gonzalez, “Digital Image Processing”, Section 3 4 ~ 3 8 10 1 10 2Section 3.4 ~ 3.8, 10.1, 10.2– Note that in Gonzalez book: Edge detection

filtering are referred to sharpeningfiltering are referred to sharpening.• A. K. Jain, “Fundamentals of Digital Image

Processing ” Section 7 4 (except medianProcessing, Section 7.4 (except median filter).


Written Assignment1. For each filter given below, compute its Fourier transform, and illustrate its magnitude

response. Determine what is its function (smoothing, edge enhancement or edge detection?) based on the filter coefficients as well as its frequency response. For each filter, determine whether it is separable? If yes compute the FT separately and explain the function of eachwhether it is separable? If yes, compute the FT separately and explain the function of each 1D filter. If not, compute the FT directly.

121 111

121242121

161

1H ;111181111

2

H

010151

010

3H


Written Assignment (cnt’d)2. Recall that the Laplacian of Gaussian filter is the Laplacian of a Gaussian function. Let the

Gaussian function be given by2

22

2),( yx

eyxG

show that the LOG filter can be written as

3 Define the a discrete Laplacian of Gaussian (LoG) filter of size 7x7 by sampling the above

2),( 2

22

24

222

yx

eyxyxL

3. Define the a discrete Laplacian of Gaussian (LoG) filter of size 7x7 by sampling the above continuous LoG function, assuming =1. Determine its discrete space Fourier transform. Illustrate its magnitude response. Explain its function. You can use the “freqz2( )” function to determine and illustrate its DTFT.


Computer Assignment1. Write a Matlab or C-program to simulate noise removal. First create a noisy image, by adding

zero mean Gaussian random noise to your image (In matlab, the noise can be generated using “imnoise()”). You can specify the noise variance in “imnoise( )”). Then apply an averaging filter to the noise added image. For a chosen variance of the added noise, you need to try different window sizes (from 3x3 to 7x7) to see which one gives you the best trade-off between noisewindow sizes (from 3x3 to 7x7) to see which one gives you the best trade-off between noise removal and blurring. Hand in your program, the original noise-added images at two different noise levels (0.01 and 0.1) and the corresponding filtered images with the best window sizes. Write down your observation. For the filtering operation, don’t use the matlab “conv2()” function. Rather, write your own function for the averaging operation. Your program should allow the user to specify the window size as an input parameter.

2. Write a Matlab or C-program for implementing the following simple edge detection algorithm: For every pixel: i) find the horizontal and vertical gradients, gx and gy, using the Sobel operator; ii) calculate the magnitude of the gradient using ; and iii) For a chosen

22yxm ggg

threshold T, consider the pixel to be an edge pixel if gm ≥ T. Save the resulting edge map (Use a gray level of 255 to indicate an edge pixel, and a gray level of 0 for a non-edge pixel). Apply this program to your test image, and observe the resulting edge map with different T values, until you find a T that gives you a good result. Hand in your program and the edge maps u t you d a t at g es you a good esu t a d you p og a a d t e edge apsgenerated by two different threshold values. Write down your observation. Hint: one automatic way to determine T is by sorting the pixels based on the gradient magnitudes, and choose T such that a certain percentage (say 5%) of pixels will be edge pixels. You can use either the


matlab conv2( ) function or write your own code for the filtering part.

3. Using the matlab “edge( )” function, evaluate the edge detection results using the following methods on an image: “sobel”, “log”,and “canny”. For each method, first use the default threshold value. Then try to use a threshold that is higher and lower, to see the effect. Comments on the advantages and disadvantages of each method.

For problems 2 and 3, try to use an image that has a lot of edges. A good set of test images can be found under the matlab directory …./toolbox/images/imdemosimages can be found under the matlab directory …./toolbox/images/imdemos


Documents

Spatial Domain LinearSpatial Domain Linear Filteringeeweb.poly.edu/~yao/EL5123/lecture5_filtering.pdf · Spatial Domain LinearSpatial Domain Linear Filtering Yao Wang ... Averaging