CSC508

Convolution Operators

CSC508

Convolution

• Arguably the most fundamental operation of computer vision

• It’s a neighborhood operator– Similar to the median and outlier processes we discussed

last week

• Utilizes a pattern of weights defined over the neighborhood– Also known as

• A filter

• A kernel

CSC508

Convolution

• Mathematically defined as

vuvuvjuiij FHR

,,,

Rij

H vjui ,

F vu,

The resultant image

The input image neighborhood

The kernel or filter

u x v The size of the kernel• H is convolved with F yielding R

CSC508

Convolution

• But what does it really mean?– It’s a “multiply/accumulate” operation

238 237 234 227 223 216

229 227 224 220 225 221

205 212 221 220 225 220

177 192 213 207 212 217

164 180 211 208 209 215

190 194 220 212 210 219

i

j

0 1 2

3 4 5

6 7 8

220*2224*1227*0jiR 220*5221*4212*3 207*8213*7192*6

Kernel

Image

CSC508

Consider 1-Dimensional Input

Input Pixels (1 Line)

0

50

100

150

200

250

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35

CSC508

A 1-Dimensional Kernel

Convolution Filter (1 Line)

-0.2

0

0.2

0.4

0.6

0.8

1

1.2

0 2 4 6 8 10 12 14 16

CSC508

Convolution in 1-DimensionInput Pixels (1 Line)

0

50

100

150

200

250

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35

Convolution Filter (1 Line)

-0.2

0

0.2

0.4

0.6

0.8

1

1.2

0 2 4 6 8 10 12 14 16

Convolution Filter (1 Line)

-0.2

0

0.2

0.4

0.6

0.8

1

1.2

0 2 4 6 8 10 12 14 16

Convolution Filter (1 Line)

-0.2

0

0.2

0.4

0.6

0.8

1

1.2

0 2 4 6 8 10 12 14 16

Convolution Filter (1 Line)

-0.2

0

0.2

0.4

0.6

0.8

1

1.2

0 2 4 6 8 10 12 14 16

Convolution Filter (1 Line)

-0.2

0

0.2

0.4

0.6

0.8

1

1.2

0 2 4 6 8 10 12 14 16

Convolution Filter (1 Line)

-0.2

0

0.2

0.4

0.6

0.8

1

1.2

0 2 4 6 8 10 12 14 16

Convolution Filter (1 Line)

-0.2

0

0.2

0.4

0.6

0.8

1

1.2

0 2 4 6 8 10 12 14 16

Convolution Filter (1 Line)

-0.2

0

0.2

0.4

0.6

0.8

1

1.2

0 2 4 6 8 10 12 14 16

Convolution Filter (1 Line)

-0.2

0

0.2

0.4

0.6

0.8

1

1.2

0 2 4 6 8 10 12 14 16

Convolution Filter (1 Line)

-0.2

0

0.2

0.4

0.6

0.8

1

1.2

0 2 4 6 8 10 12 14 16

Convolution Filter (1 Line)

-0.2

0

0.2

0.4

0.6

0.8

1

1.2

0 2 4 6 8 10 12 14 16

Convolution Filter (1 Line)

-0.2

0

0.2

0.4

0.6

0.8

1

1.2

0 2 4 6 8 10 12 14 16

Convolution Filter (1 Line)

-0.2

0

0.2

0.4

0.6

0.8

1

1.2

0 2 4 6 8 10 12 14 16

Convolution Filter (1 Line)

-0.2

0

0.2

0.4

0.6

0.8

1

1.2

0 2 4 6 8 10 12 14 16

Convolution Filter (1 Line)

-0.2

0

0.2

0.4

0.6

0.8

1

1.2

0 2 4 6 8 10 12 14 16

Convolved Output (1 Line)

-250

-200

-150

-100

-50

0

50

100

150

200

250

0 5 10 15 20 25 30 35

CSC508

1-Dimension Output

Convolved Output (1 Line)

-250

-200

-150

-100

-50

0

50

100

150

200

250

0 5 10 15 20 25 30 35

CSC508

Convolution

• We “slide” the kernel over the entire image– In two dimensions we just slide the kernel top to

bottom and left to right

• Eventually it gets centered over every pixel– Note, we typically do this operation so that all

pixels are handled in parallel (simultaneously)– It’s a fairly “expensive” operation

• What about the edges of the image?– Various techniques can be employed

CSC508

Convolution at the Edges

• Just ignore the edges– Initialize the resultant image to 0 (or something

else)

• Use only the parts of the kernel that are within the image

• Enlarge the input image on all sides by reflection

CSC508

Enlarging by Reflection

238 238 237 234 227 223 216 216

238 238 237 234 227 223 216 216

229 229 227 224 220 225 221 221

205 205 212 221 220 225 220 220

177 177 192 213 207 212 217 217

164 164 180 211 208 209 215 215

190 190 194 220 212 210 219 219

190 190 194 220 212 210 219 219

Original image

Assume a 3x3 kernel

A larger kernel requires additional reflection

CSC508

Why Reflect?

• Why not just pad with 0 (or some other value)?

• We’ll see later when we look at edge detection

CSC508

Some Interesting Kernels

• Neighborhood averaging– A simple blurring function

– Note that the results of the convolution may go out of range

– Solution is to normalize the kernel

– Various techniques• Divide each kernel by the sum of all

values

• Divide the result by the sum of all values

1 1 1 1 1

1 1 1 1 1

1 1 1 1 1

1 1 1 1 1

1 1 1 1 1

CSC508

Neighborhood Averaging

CSC508

Some Interesting Kernels

• Emboss– Not really useful for

computer vision but interesting none the less

– Why don’t we need to normalize this kernel?

– But, we do need to make sure the output does not go below 0 (since a kernel value is negative)

-1 0 0

0 0 0

0 0 1

CSC508

Emboss

CSC508

Some Interesting Kernels

• Laplacian– A simple gradient detection mask

– What is the range of the result?• [-1020..1020]

• Need to do something about this (clamp or scale)

0 1 0

1 -4 1

0 1 0

CSC508

Laplacian

CSC508

Some Interesting Kernels

• You can set the kernel weights to any values you want– Anything that will give you the desired effect– Selecting “meaningful” weights is an art

CSC508

Gaussian Kernel (Filter)

• The Gaussian filter is a smoothing or blurring filter

• Width is the number of pixels covered by the filter• Sigma is the standard deviation of the Gaussian curve

in pixels

eGyx

yx 2

22

22

1, 2

2,2widthyxwidth

Coding the Gaussian

• Use odd number of rows and columns in the kernel (e.g. 3x3, 5x5, 7x7…)

• Loop over every location in the kernel matrix– Translate integer indices from [0..width-1] to floating point

[–width/2..+width/2]

– This makes the floating point coordinate of the central value (0.0, 0.0)

• Perform the calculation given with the translated loop indices as the x and y values

• When done, normalize the kernel coefficients by dividing each coefficient by the sum of all coefficients

CSC508

CSC508

Gaussian Filter

• Sigma 1.0, Filter Width 7

1 2 3 4 5 6 7S1

S3

S5S7

0.00E+00

2.00E-02

4.00E-02

6.00E-02

8.00E-02

1.00E-01

1.20E-01

1.40E-01

1.60E-01

CSC508

Gaussian Filter

• Sigma 1.6, Filter Width 7

1 2 3 4 5 6 7S1

S3

S50

0.01

0.02

0.03

0.04

0.05

0.06

0.07

CSC508

Gaussian Filter

• Sigma 1.6, Filter Width 15

1 3 5 7 9

11

13

15

S1

S6

S11

0.00E+00

1.00E-02

2.00E-02

3.00E-02

4.00E-02

5.00E-02

6.00E-02

7.00E-02

CSC508

Gaussian is Separable

• Two 1D Gaussians produces the same results a one 2D Gaussian– First convolve the original image horizontally – Then convolve the horizontal results vertically – This speeds things up dramatically

• Reduces the total number of multiplies and additions

CSC508

Gaussian Filter – 1D

eGx

x 2

2

22

12

22widthxwidth

CSC508

Gaussian Filter – 1D

• Sigma 1.6, Filter Width 15

0.00E+00

1.00E-02

2.00E-02

3.00E-02

4.00E-02

5.00E-02

6.00E-02

7.00E-02

1 3 5 7 9 11 13 15

CSC508

Gaussian Filter

• Why would we want to blur an image?

• It will help us to extract gradient features (edges)

CSC508

Edge Detection

• What is an edge?– An intensity gradient– That is, a change of intensity within a localized region of the image (a

neighborhood)

The edge

CSC508

Edge Detection

• Is there an edge here?

• There is definitely a gradient but no edge– At least not over a small neighborhood

CSC508

Edge Detection

• Edges are described by magnitude which is related to the intensity on either side of the edge

Weaker edge

Strong edge

CSC508

Edge Detection

• Edges are described by orientation (direction)

– Orientation is determined by the angle of the gradient and the intensity on either side of the gradient

CSC508

Edge Detection

• Various techniques– Differential operator

• Sobel

– Templates• Nevatia-Babu

– Procedural• Marr-Hildreth (Laplacian-Gaussian)

• Canny

CSC508

Sobel Edge Operator

• Starts with two convolution kernels

-1 0 1

-2 0 2

-1 0 1

-1 -2 -1

0 0 0

1 2 1

• Perform two convolutions on the original image resulting in two intermediate images

CSC508

Sobel Edge Operator

• From the two convolved images you can now compute edge magnitude and direction

GG yxjiMag

22

GG

x

yjiDir arctan

– The magnitude will have to be scaled to 0..255 • Unscaled values will be both + and -

– The direction is typically scaled to a small number of bins• i.e. 0..8 or 0..16

CSC508

Sobel Edge Operator

Input Magnitude EncodedDirection

If we were to zoom in on the corners we’d see other edge orientations present

This edge is not missing, it’s just the same color as the background

CSC508

Nevatia-Babu Template Operator

• Six edge-oriented convolution kernels (templates)

-100 -100 0 100 100

-100 -100 0 100 100

-100 -100 0 100 100

-100 -100 0 100 100

-100 -100 0 100 100

-100 32 100 100 100

-100 78 92 100 100

-100 -100 0 100 100

-100 -100 -92 78 100

-100 -100 -100 -32 100

100 100 100 100 100

-32 78 100 100 100

-100 -92 0 92 100

-100 -100 -100 -78 32

-100 -100 -100 -100 -100

100 100 100 100 100

100 100 100 100 100

0 0 0 0 0

-100 -100 -100 -100 -100

-100 -100 -100 -100 -100

100 100 100 100 100

100 100 100 78 -32

100 92 0 -92 -100

32 -78 -100 -100 -100

-100 -100 -100 -100 -100

100 100 100 32 -100

100 100 92 -78 -100

100 100 0 -100 -100

100 78 -92 -100 -100

100 -32 -100 -100 -100

0-deg 30-deg 60-deg

90-deg 120-deg 150-deg

CSC508

Nevatia-Babu Template Operator

• Convolve each image pixel with all six kernels• Select the mask that produces the maximum output

– Assign the magnitude to the output of the maximal mask

– Assign the direction to the orientation of the maximal mask

– Direction is further modified by the sign of the maximal mask

• As you might imagine, this is a very time consuming operation

CSC508

Things To Do• Programming homework assignment

– Gaussian Convolution• Reflect the edges of the image• Implement as both a 2D convolution and 2 1D convolutions• “prove” that they are equivalent through code demonstrations

– Sobel Edge Operator • You may write in any programming language you choose• Deliverables:

– Zipped images in email– Email the source code to reinhart@clunet.edu with the subject line

CSC508 PROGRAM 2• Due beginning of class in two weeks

– (late assignments will be penalized 10%)– I will post test images

• Reading for Next Week – Still in chapter 8– We’ll talk about

• Non-maximal edge suppression• Edge following• Hysteresis

– We’ll experiment with two edge detection algorithms• Marr-Hildreth (Laplacian-Gaussian/Difference of Gaussians)• Canny (Differential)