34
BIL721: Computational Photography Spring 2015, Lecture 3 Aykut Erdem Hacettepe University Computer Vision Lab (HUCVL) Image Processing Basics Photo from Why Cats Paint by H. Busch and B. Silver Today Digital images Light and Perception Image operations Linear filtering Non-linear filtering 2 Image Formation 3 slide by Steven Seitz Sampling and Reconstruction 4 slide by Steven Seitz

Please share your knowledge Today - Hacettepeaykut/classes/spring2015/bil721/... · Please share your knowledge BIL721: Computational Photography! Spring 2015, Lecture 3! Aykut Erdem!

Embed Size (px)

Citation preview

Please share your knowledge

BIL721: Computational Photography !Spring 2015, Lecture 3 !

Aykut Erdem !Hacettepe University !

Computer Vision Lab (HUCVL)!

Image Processing Basics

Photo from W

hy Cats Paint by H. Busch and B. Silver

Today •  Digital images!•  Light and Perception!•  Image operations!•  Linear filtering!•  Non-linear filtering!!

2!

Image Formation!

3!

slide by Steven Seitz!

Sampling and Reconstruction!

4!

slide by Steven Seitz!

Sampled representations •  How to store and compute with continuous

functions?!•  Common scheme for representation: samples !

–  write down the function’s values at many points !

5!

slide by Steven Marschner!

Reconstruction •  Making samples back into a continuous function !

–  for output (need realizable method)!–  for analysis or processing (need mathematical method) !–  amounts to “guessing” what the function did in between !

6!

slide by Steven Marschner!

7!

slide by Derek Hoiem!

A Digital Image is a Matrix of Pixels 0.92% 0.93% 0.94% 0.97% 0.62% 0.37% 0.85% 0.97% 0.93% 0.92% 0.99%0.95% 0.89% 0.82% 0.89% 0.56% 0.31% 0.75% 0.92% 0.81% 0.95% 0.91%0.89% 0.72% 0.51% 0.55% 0.51% 0.42% 0.57% 0.41% 0.49% 0.91% 0.92%0.96% 0.95% 0.88% 0.94% 0.56% 0.46% 0.91% 0.87% 0.90% 0.97% 0.95%0.71% 0.81% 0.81% 0.87% 0.57% 0.37% 0.80% 0.88% 0.89% 0.79% 0.85%0.49% 0.62% 0.60% 0.58% 0.50% 0.60% 0.58% 0.50% 0.61% 0.45% 0.33%0.86% 0.84% 0.74% 0.58% 0.51% 0.39% 0.73% 0.92% 0.91% 0.49% 0.74%0.96% 0.67% 0.54% 0.85% 0.48% 0.37% 0.88% 0.90% 0.94% 0.82% 0.93%0.69% 0.49% 0.56% 0.66% 0.43% 0.42% 0.77% 0.73% 0.71% 0.90% 0.99%0.79% 0.73% 0.90% 0.67% 0.33% 0.61% 0.69% 0.79% 0.73% 0.93% 0.97%0.91% 0.94% 0.89% 0.49% 0.41% 0.78% 0.78% 0.77% 0.89% 0.99% 0.93%

8!

slide by Derek Hoiem!

A Digital Image is a Matrix of Pixels

9!

slide by Charless Fowlkes!

Pixels Measure Brightness over an Area

slide by Chuck Dyer! 10!

Light matter, eyes •  Light from sources is reflected by objects and

reaches the eye. !•  The amount of light from the source gets

multiplied by the object reflectance !–  on a per-wavelength basis !

11!

slide by Fredo Durand!

Human perception •  Our eyes have an uncanny ability to discount

the illumination!–  Only objects really matter for survival!–  Light is only useful to understand if you’re a

photographer or to choose your sun lotion !

12!

slide by Fredo Durand!

Illusion by Adelson

13!

•  A & B have exactly the same tone !

Perception of Intensity

slide by Fredo Durand! 14!

•  A & B have exactly the same tone !

Demo"Simultaneous Contrast Effect

15!

http://web.mit.edu/persci/gaz/gaz-teaching/flash/contrast-movie.swf!!

Mechanism to discount light •  Light adaptation

– We recenter our neural response around the current average brightness!

– neural + chemical + pupil!!

•  Chromatic adaptation– eliminate color cast due to light sources !

•  e.g. Daylight is white but tungsten is yellowish !– Related to white balance !

16!

slide by Fredo Durand!

Perception of Color: Munker-White Illusion

All of the small bars within each pattern are actually the same color (i.e., all of the blue segments are the same color blue, all of the red segments are the same color red, and all of the small gray bars are the same color gray). The lightness differences you are experiencing are illusions. !

slide by Chuck Dyer! 17!

White balance & "Chromatic adaptation •  Different illuminants have

different color temperature!•  Our eyes adapt to this: !

Chromatic adaptation!– We actually adapt better in

brighter scenes!–  This is why candlelit scenes

still look yellow!

18!

slide by Fredo Durand!

White balance problem •  When watching a picture on screen

or print, we adapt to the illuminant of the room, not that of the scene in the picture!

•  The eye cares more about objects’ intrinsic color, not the color of the light leaving the objects !

•  We need to discount the color of the light source!

19!

Same object, !different illuminants!

slide by Fredo Durand!

Von Kries adaptation •  Multiply each channel by a gain

factor!– R’=R*kr!– G’=G*kg!– B’=B*kb!

20!www.shortcourses.com/guide/guide2-27.html!

slide by Fredo Durand!

Von Kries adaptation •  Multiply each channel by a gain !

factor!•  Note that the light source could have a

more complex effect !– Arbitrary 3x3 matrix!– More complex spectrum transformation!

21!www.shortcourses.com/guide/guide2-27.html!

slide by Fredo Durand!

White balance challenge •  How do we find the scaling factors for r, g,

and b?!

22!

slide by Fredo Durand!

Best way to do white balance •  Grey card: !•  Take a picture of a neutral object !

(white or gray) !•  Deduce the weight of each channel !•  If the object is recoded as rw, gw, bw !

use weights k/rw, k/gw, k/bw!where k controls the exposure !

23!

slide by Fredo Durand!

Party name tags •  Provide excellent white references! !

24!

imwrite(im/im[300, 214])

slide by Fredo Durand!

Without grey cards •  We need to “guess” which pixels

correspond to white objects !

25!

slide by Fredo Durand!

Grey world assumption •  Assume average color in the image is grey !

•  Use weights proportional to !

•  Usually assumes 18% grey to set exposure !

26!

slide by Fredo Durand!

Brightest pixel assumption •  Highlights usually have the color of the light

source !–  At least for dielectric materials !

•  White balance by using the brightest pixels !–  Plus potentially a bunch of heuristics!–  In particular use a pixel that is not saturated/clipped !

27!

slide by Fredo Durand!

Light with color control •  http://www.robgalbraith.com/

content_page4c65.html?cid=7-11666-11979!

•  Gel on flash!

28!

slide by Fredo Durand!

http://digital-photography-school.com/how-to-color-balance-your-flash-with-gels/!

Color Image R

G

B

29!

slide by Derek Hoiem!

Images in Matlab •  Images represented as a matrix!•  Suppose we have an n x m RGB image called “im” !

–  im(1,1,1) = top-left pixel value in R channel!–  im(y, x, b) = y pixels down, x pixels to right in the bth channel!–  im(n, m, 3) = bottom-right pixel in B channel!

•  imread(filename)%returns a%uint8%image (values 0 to 255)!–  Convert to double format (values 0 … 1) with%im2double

0.92% 0.93% 0.94% 0.97% 0.62% 0.37% 0.85% 0.97% 0.93% 0.92% 0.99%0.95% 0.89% 0.82% 0.89% 0.56% 0.31% 0.75% 0.92% 0.81% 0.95% 0.91%0.89% 0.72% 0.51% 0.55% 0.51% 0.42% 0.57% 0.41% 0.49% 0.91% 0.92%0.96% 0.95% 0.88% 0.94% 0.56% 0.46% 0.91% 0.87% 0.90% 0.97% 0.95%0.71% 0.81% 0.81% 0.87% 0.57% 0.37% 0.80% 0.88% 0.89% 0.79% 0.85%0.49% 0.62% 0.60% 0.58% 0.50% 0.60% 0.58% 0.50% 0.61% 0.45% 0.33%0.86% 0.84% 0.74% 0.58% 0.51% 0.39% 0.73% 0.92% 0.91% 0.49% 0.74%0.96% 0.67% 0.54% 0.85% 0.48% 0.37% 0.88% 0.90% 0.94% 0.82% 0.93%0.69% 0.49% 0.56% 0.66% 0.43% 0.42% 0.77% 0.73% 0.71% 0.90% 0.99%0.79% 0.73% 0.90% 0.67% 0.33% 0.61% 0.69% 0.79% 0.73% 0.93% 0.97%0.91% 0.94% 0.89% 0.49% 0.41% 0.78% 0.78% 0.77% 0.89% 0.99% 0.93%

0.92% 0.93% 0.94% 0.97% 0.62% 0.37% 0.85% 0.97% 0.93% 0.92% 0.99%0.95% 0.89% 0.82% 0.89% 0.56% 0.31% 0.75% 0.92% 0.81% 0.95% 0.91%0.89% 0.72% 0.51% 0.55% 0.51% 0.42% 0.57% 0.41% 0.49% 0.91% 0.92%0.96% 0.95% 0.88% 0.94% 0.56% 0.46% 0.91% 0.87% 0.90% 0.97% 0.95%0.71% 0.81% 0.81% 0.87% 0.57% 0.37% 0.80% 0.88% 0.89% 0.79% 0.85%0.49% 0.62% 0.60% 0.58% 0.50% 0.60% 0.58% 0.50% 0.61% 0.45% 0.33%0.86% 0.84% 0.74% 0.58% 0.51% 0.39% 0.73% 0.92% 0.91% 0.49% 0.74%0.96% 0.67% 0.54% 0.85% 0.48% 0.37% 0.88% 0.90% 0.94% 0.82% 0.93%0.69% 0.49% 0.56% 0.66% 0.43% 0.42% 0.77% 0.73% 0.71% 0.90% 0.99%0.79% 0.73% 0.90% 0.67% 0.33% 0.61% 0.69% 0.79% 0.73% 0.93% 0.97%0.91% 0.94% 0.89% 0.49% 0.41% 0.78% 0.78% 0.77% 0.89% 0.99% 0.93%

0.92% 0.93% 0.94% 0.97% 0.62% 0.37% 0.85% 0.97% 0.93% 0.92% 0.99%0.95% 0.89% 0.82% 0.89% 0.56% 0.31% 0.75% 0.92% 0.81% 0.95% 0.91%0.89% 0.72% 0.51% 0.55% 0.51% 0.42% 0.57% 0.41% 0.49% 0.91% 0.92%0.96% 0.95% 0.88% 0.94% 0.56% 0.46% 0.91% 0.87% 0.90% 0.97% 0.95%0.71% 0.81% 0.81% 0.87% 0.57% 0.37% 0.80% 0.88% 0.89% 0.79% 0.85%0.49% 0.62% 0.60% 0.58% 0.50% 0.60% 0.58% 0.50% 0.61% 0.45% 0.33%0.86% 0.84% 0.74% 0.58% 0.51% 0.39% 0.73% 0.92% 0.91% 0.49% 0.74%0.96% 0.67% 0.54% 0.85% 0.48% 0.37% 0.88% 0.90% 0.94% 0.82% 0.93%0.69% 0.49% 0.56% 0.66% 0.43% 0.42% 0.77% 0.73% 0.71% 0.90% 0.99%0.79% 0.73% 0.90% 0.67% 0.33% 0.61% 0.69% 0.79% 0.73% 0.93% 0.97%0.91% 0.94% 0.89% 0.49% 0.41% 0.78% 0.78% 0.77% 0.89% 0.99% 0.93%

R

G

B

row column

30!

slide by Derek Hoiem!

Basic types of operations

31!

Basic types of operationsPoint operations:

range only

Domain operations

Neighborhood operations:

domain and range

Thursday, February 9, 12

Basic types of operationsPoint operations:

range only

Domain operations

Neighborhood operations:

domain and range

Thursday, February 9, 12

Basic types of operationsPoint operations:

range only

Domain operations

Neighborhood operations:

domain and range

Thursday, February 9, 12

Neighborhood !operations: !

domain and range!

Basic types of operationsPoint operations:

range only

Domain operations

Neighborhood operations:

domain and range

Thursday, February 9, 12

Point operations: !range only!

g(x, y) = f(tx

(x, y), ty

(x, y))

Basic types of operationsPoint operations:

range only

Domain operations

Neighborhood operations:

domain and range

Thursday, February 9, 12

Domain !operations!

g(x, y) = f(tx,y

(x, y))

slide by Fredo Durand!

Point Operations •  Map each pixel’s value to a new value !•  Neighborhood is 1 x 1!•  g(i,j) = h(f(i,j)) where f is the input image, g is the

output (i.e., transformed) image, and h is the point operator / transformation!

•  Examples!–  g(i,j) = af(i,j) + b where a > 0 is a gain parameter and b

controls the brightness!–  Mapping one color space to another, e.g., RGB → HSV!–  Image rotation, translation, scale change, … !

32!

slide by Derek Hoiem!

Instagram Filters •  How do they make those Instagram filters?!

“It's really a combination of a bunch of different methods. In some cases we draw on top of images, in others we do pixel math. It really depends on the effect we're going for.” -- Kevin Systrom, co-founder of Instagram!!

slide by Chuck Dyer! 33!

Example Instagram Steps 1.  Perform an independent RGB color point

transformation on the original image to increase contrast or make a color cast !

slide by Chuck Dyer! 34!

Example Instagram Steps 2.  Overlay a circle background image to

create a vignette effect !

slide by Chuck Dyer! 35!

Example Instagram Steps 3.  Overlay a background image as

decorative grain!slide by Chuck Dyer! 36!

Example Instagram Steps 4.  Add a border or frame!

slide by Chuck Dyer! 37!

Result

Javascript library for creating Instagram-like effects, see:! http://alexmic.net/filtrr/!

slide by Chuck Dyer! 38!

Histogram •  For each value (e.g. 0-255) !

how many pixels have this value? !•  Cumulative histogram: for each value x, how many

pixels have a value smaller than x? !

39!

Histogram

• For each value (e.g. 0-255) how many pixels have this value?

• Cumulative histogram: for each value x, how many pixels have a value smaller than x

46

pixel value

#pixels

http://en.wikipedia.org/wiki/Histogram_equalizationThursday, February 9, 12

Histogram

• For each value (e.g. 0-255) how many pixels have this value?

• Cumulative histogram: for each value x, how many pixels have a value smaller than x

46

pixel value

#pixels

http://en.wikipedia.org/wiki/Histogram_equalizationThursday, February 9, 12

slide by Fredo Durand!

Very useful on cameras •  Allows you to tell if you use the dynamic range

entirely.!

40!

Very useful on camera

• Allows you to tell if you use the dynamic range entirely

47

Thursday, February 9, 12

slide by Fredo Durand!

41!

Bad: bright values under-used Bad: bright values saturate !(underexposure) (overexposure)!

48http://www.luminous-landscape.com/tutorials/understanding-series/understanding-histograms.shtml

Bad: bright values under-used(underexposure)

Bad: bright values saturate(overexposure)

Thursday, February 9, 12

http://www.luminous-landscape.com/tutorials/understanding-series/understanding-histograms.shtml!!

slide by Fredo Durand!

Histogram equalization

• Given image with given histogram• monotonic remapping to get a flat histogram

50

pixel value

#pixels

old pixel value

new pixel value

http://en.wikipedia.org/wiki/Histogram_equalizationThursday, February 9, 12

Histogram equalization

• Given image with given histogram• monotonic remapping to get a flat histogram

50

pixel value

#pixels

old pixel value

new pixel value

http://en.wikipedia.org/wiki/Histogram_equalizationThursday, February 9, 12

Histogram equalization •  Given image with given histogram !•  monotonic remapping to get a flat histogram !

42!

Histogram equalization

• Given image with given histogram• monotonic remapping to get a flat histogram

50

pixel value

#pixels

old pixel value

new pixel value

http://en.wikipedia.org/wiki/Histogram_equalizationThursday, February 9, 12

Histogram equalization

• Given image with given histogram• monotonic remapping to get a flat histogram

50

pixel value

#pixels

old pixel value

new pixel value

http://en.wikipedia.org/wiki/Histogram_equalizationThursday, February 9, 12

slide by Fredo Durand!

Histogram Equalization!

43!

slide by Derek Hoiem!

Goal: Given n x m image f with 8 bpp (brightness values 0 to 255), create a new image g that has about nm/255 pixels of each brightness value !!•  Compute f ’s histogram: H(i), 0 ≤ i ≤ 255!•  Compute f ’s cumulative histogram: !•  Compute mapping and output image: !

C(i) = H ( j)j=0

i

g(i, j) = round 255C( f (i, j))−Cminnm−Cmin

"

#$

%

&'

where Cmin = minimum non-zero value in C !44!

slide by Derek Hoiem!

Histogram Equalization Algorithm

Color Transfer •  Goal: Change the colors of a given source

image to match the color palette of another image in order to transfer the mood or style of one image to another !

•  One simple method: Shift and scale the pixel values of the target image to match the mean and standard deviation of the source image!

•  Color space representation of images affects quality of results. One good one: Lab color space.!

45!

Color Transfer Let I = Source image, i.e., one being recolored !Let J = Palette image, i.e., one colors taken from !1.  Convert I and J to Lab color space; call result

images S and P respectively!2.  Compute L channel output matrix:!

3.  Similarly, compute a and b channels !4.  Combine L, a and b, and convert back to RGB!

PSSS

Pout LL µµ

σσ

+−= )(

46!

Color Transfer

Pale1e%

47!

Local / Neighborhood Operations •  Value of pixel in output image is a function of

the corresponding pixel in the input image plus other nearby pixels (usually defined by a square or rectangular window centered on the given pixel) !

48!

slide by Derek Hoiem!

•  Basic idea: define a new function by averaging over a sliding window!

•  A simple example to start off: smoothing !

Linear Filtering

49!

slide by Steven Marschner!

Linear Filtering •  Same moving average operation, expressed

mathematically:!

50!

Image Filtering •  Modify the pixels in an image based on some

function of a local neighborhood of the pixels !!

Local image data!

7"

Modified image data!

Some function!

•  Simplest: linear filtering– Replace each pixel by a linear combination

of its neighbors!

51!

5" 1"4"1" 7"1"

5" 3"10"slide by Derek Hoiem!

Linear Functions

•  Simplest: linear filtering– Replace each pixel by a linear

combination of its neighbors !

5" 1"4"1" 7"1"

5" 3"10"

0.5"0.5"0"0"1"0"

0" 0"0"

Local image data! kernel!

7"

Modified image data!52!

slide by Derek Hoiem!

1"1"1"

1"1"1"

1"1"1"

],[g ⋅⋅

2D Example: Box Filter !

53!

slide by David Lowe!

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

h[m,n]= g[k, l]k,l∑ f [m+ k,n+ l]

[.,.]h[.,.]fImage Filtering!

1"1"1"

1"1"1"

1"1"1"

],[g ⋅⋅

54!

slide by Steven Seitz!

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 10

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

[.,.]h[.,.]fImage Filtering!

1"1"1"

1"1"1"

1"1"1"

],[g ⋅⋅

h[m,n]= g[k, l]k,l∑ f [m+ k,n+ l]

55!

slide by Steven Seitz!

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 10 20

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

[.,.]h[.,.]fImage Filtering!

1"1"1"

1"1"1"

1"1"1"

],[g ⋅⋅

h[m,n]= g[k, l]k,l∑ f [m+ k,n+ l]

56!

slide by Steven Seitz!

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 10 20 30

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

[.,.]h[.,.]fImage Filtering!

1"1"1"

1"1"1"

1"1"1"

],[g ⋅⋅

h[m,n]= g[k, l]k,l∑ f [m+ k,n+ l]

57!

slide by Steven Seitz!

0 10 20 30 30

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

[.,.]h[.,.]fImage Filtering!

1"1"1"

1"1"1"

1"1"1"

],[g ⋅⋅

h[m,n]= g[k, l]k,l∑ f [m+ k,n+ l]

58!

slide by Steven Seitz!

0 10 20 30 30

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

[.,.]h[.,.]fImage Filtering!

1"1"1"

1"1"1"

1"1"1"

],[g ⋅⋅

?"

h[m,n]= g[k, l]k,l∑ f [m+ k,n+ l]

59!

slide by Steven Seitz!

0 10 20 30 30

50

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

[.,.]h[.,.]fImage Filtering!

1"1"1"

1"1"1"

1"1"1"

],[g ⋅⋅

?"

h[m,n]= g[k, l]k,l∑ f [m+ k,n+ l]

60!

slide by Steven Seitz!

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 10 20 30 30 30 20 10

0 20 40 60 60 60 40 20

0 30 60 90 90 90 60 30

0 30 50 80 80 90 60 30

0 30 50 80 80 90 60 30

0 20 30 50 50 60 40 20

10 20 30 30 30 30 20 10

10 10 10 0 0 0 0 0

[.,.]h[.,.]fImage Filtering! 1"1"1"

1"1"1"1"1"1"],[g ⋅⋅

h[m,n]= g[k, l]k,l∑ f [m+ k,n+ l]

61!

slide by Steven Seitz!

What does it do? !•  Replaces each pixel with an

average of its neighborhood!

•  Achieves smoothing effect (i.e., removes sharp features)!

!•  Weaknesses:!

•  Blocky results!•  Axis-aligned streaks!

1"1"1"

1"1"1"

1"1"1"

],[g ⋅⋅

Box Filter!

62!

slide by David Lowe!

Smoothing with Box Filter!

63!

slide by Derek Hoiem!

Properties of Smoothing Filters

•  Smoothing!– Values all positive !– Sum to 1 ⇒ constant regions same as input !– Amount of smoothing proportional to mask size !– Removes high-frequency components!–  low-pass filter!

64!

slide by Derek Hoiem!

Practice with Linear Filters

0"0"0"

0"1"0"

0"0"0"

Original!

?"

65!

slide by David Lowe!

Practice with Linear Filters

0"0"0"

0"1"0"

0"0"0"

Original! Filtered !(no change)!

66!

slide by David Lowe!

Practice with Linear Filters

0"0"0"

1"0"0"

0"0"0"

Original!

?"

67!

slide by David Lowe!

Practice with Linear Filters

0"0"0"

1"0"0"

0"0"0"

Original! Shifted leftby 1 pixel!

68!

slide by David Lowe!

Practice with Linear Filters

Original!

1"1"1"1"1"1"1"1"1"

0"0"0"0"2"0"0"0"0" #" ?"

(Note that filter sums to 1) !

69!

slide by David Lowe!

Practice with Linear Filters

Original!

1"1"1"1"1"1"1"1"1"

0"0"0"0"2"0"0"0"0" #"

Sharpening filter•  Sharpen an out of focus image by subtracting a

blurred version!

70!

slide by David Lowe!

Sharpening

71!

slide by Fredo Durand!

Sharpening by Unsharp Masking •  h = f – k(f * g) where k is a small positive constant

and g =!

•  Why does it work?!•  Say f is a blurred image produced from an ideal

image p by convolving it with a box filter s= !•  p * g ∝ p – (p * s) = p - f!•  h = f – k(f * g) ≈ f – k(-1/5(p – f)) ≈ p!•  Simulates Mach Band effect in human vision!•  Called unsharp masking in photography!

0" 1" 0"

1" *4" 1"

0" 1" 0"

called a Laplacian mask!

0 1/5 0

1/5 1/5 1/5

0 1/5 0

72!

slide by Derek Hoiem!

Unsharp masking •  Sharpening is often called “unsharp mask” because

photographers used to sandwich a negative with a blurry positive film in order to sharpen !

73!

slide by Fredo Durand!

Sharpening using Unsharp Mask Filter

Original! Filtered result!

74!

slide by Derek Hoiem!

Unsharp Masking!

75!

slide by Derek Hoiem!

Mach Bands

Actual brightness!

Perceived by you!

76!

slide by Chuck Dyer!

Demo"Mach Bands and "

the Cornsweet Illusion

77!

http://web.mit.edu/persci/gaz/gaz-teaching/flash/craik-movie.swf!!

Center/Surround Cells in Retina Organization of ganglion cells in retina respond to discontinuities of light (S. Kuffler, 1953) !

78!

slide by Chuck Dyer!

E. Lingelbach’s version of “Hermann Grid”

79!

slide by Chuck Dyer!

Lateral Inhibition Explanation !

Inhibited less by white stripes!

Inhibited more by white stripes!

80!

slide by Chuck Dyer!

Lateral Inhibition in HVS!

81!

slide by Chuck Dyer!

82!

slide by Chuck Dyer!

Simultaneous Contrast

In Van Gogh’s cafes, the contrast between yellow & blue, and red & green, cause both colors to seem more vivid. Color perception always depends on the other colors nearby !

slide by Chuck Dyer!

Other Filters: Edge Detection

*1"0"1"

*2"0"2"

*1"0"1"

Vertical edges!(absolute value)!

Sobel!

84!

slide by Derek Hoiem!

Other Filters: Edge Detection

*1"*2"*1"

0"0"0"

1"2"1"

Horizontal edges!(absolute value)!

Sobel!

85!

slide by Derek Hoiem!

Cross-Correlation vs. Convolution

•  2D filtering / cross-correlation!– h=filter2(g,f); or h=imfilter(f,g);

%•  2D convolution!

– h=conv2(g,f);

h[m,n]= g[k, l]k,l∑ f [m− k,n− l]

g = filter!

h[m,n]= g[k, l]k,l∑ f [m+ k,n+ l]

f = image!

86!

slide by Derek Hoiem!

Convolution •  Convolution: !

–  Flip the filter in both dimensions (bottom to top, right to left)!

–  Then apply cross-correlation !

Notation for convolution operator!

g"

h[m,n]= g[k, l]k,l∑ f [m− k,n− l]

h+=+g+*+f"

87!

slide by Kristen Graum

an!

Key Properties of Linear Filters !Linearity: !filter(f1 + f2) = filter(f1) + filter(f2) !Shift invariance: same behavior regardless of pixel location!filter(shift(f)) = shift(filter(f))

!Any linear, shift-invariant operator can be represented as a convolution operation !

88!

slide by Svetlana Lazebnik!

More Properties •  Commutative: a * b = b * a

–  Conceptually no difference between filter and image !

•  Associative: a * (b * c) = (a * b) * c –  Often apply several filters one after another: (((a * b1) * b2) * b3) –  This is equivalent to applying one filter: a * (b1 * b2 * b3)

•  Distributes over addition: a * (b + c) = (a * b) + (a * c)

•  Scalars factor out: ka * b = a * kb = k (a * b)

•  Identity: unit impulse e = [0, 0, 1, 0, 0] ⇒ a * e = a !

89!

slide by Svetlana Lazebnik!

•  Weight contributions of neighboring pixels by distance from center pixel!

!

•  Constant factor at front makes volume sum to 1!•  Convolve each row of image with 1D kernel to produce new image;

then convolve each column of new image with same 1D kernel to yield output image!

!!

0.003 0.013 0.022 0.013 0.003 0.013 0.059 0.097 0.059 0.013 0.022 0.097 0.159 0.097 0.022 0.013 0.059 0.097 0.059 0.013 0.003 0.013 0.022 0.013 0.003

5 x 5, σ = 1!

Gaussian Filtering!

90!

slide by |Christopher Rasmussen!

Smoothing with a Gaussian

•  Smoothing with a box actually doesn’t compare at all well with a defocused lens!

•  Most obvious difference is that a single point of light viewed in a defocused lens looks like a fuzzy blob; but the averaging process would give a little square!

•  Gaussian is isotropic !(i.e., rotationally symmetric)!

•  A Gaussian gives a good model of a fuzzy blob!

•  It closely models many physical processes (the sum of many small effects)!

91!

slide by David Forsyth!

What does Blurring take away?!

original!92!

slide by Derek Hoiem!

What does Blurring take away? !

smoothed (5x5 Gaussian)!93!

slide by Derek Hoiem!

Smoothing with Gaussian Filter !

94!

slide by Derek Hoiem!

Smoothing with Box Filter!

95!

slide by Derek Hoiem!

input

96!

slide by Syvlain Paris!

box average

97!

slide by Syvlain Paris!

Gaussian blur

98!

slide by Syvlain Paris!

Gaussian Filters •  What parameters matter here?!•  Standard deviation (σ) of Gaussian:

determines extent of smoothing!!

σ = 2 with !30 x 30 kernel !

σ = 5 with !30 x 30 kernel!

99!

slide by Derek Hoiem! 100!

slide by Derek Hoiem!

Smoothing with a Gaussian

for sigma=1:3:10 h = fspecial('gaussian�, hsize, sigma); out = imfilter(im, h); imshow(out); pause;

end

•  Parameter σ is the scale / width / spread of the Gaussian kernel, and controls the amount of smoothing !

101!

slide by Derek Hoiem!

Small σ !

102!

slide by David Forsyth!

Medium σ !

103!

slide by David Forsyth!

Large σ !

104!

slide by David Forsyth!

Gaussian Filters

= 30 pixels!= 1 pixel! = 5 pixels! = 10 pixels!

105!

slide by Derek Hoiem!

Spatial resolution and color

R!

G!

B!original!

106!

slide by Derek Hoiem!

Blurring the G component

R!

G!

B!

original! processed!

107!

slide by Derek Hoiem!

Blurring the R component

original! processed!

R!

G!

B!

108!

slide by Derek Hoiem!

Blurring the B component

original!

R!

G!

B!processed!

109!

slide by Derek Hoiem!

L"

a"

b"

•  A rotation of the color coordinates into directions that are more perceptually meaningful: !!

L: luminance, !a: red-green, !b: blue-yellow!

Lab Color Component

110!

slide by Derek Hoiem!

L!

a!

b!original!

Blurring L

111!

slide by Derek Hoiem!

processed! original!

L!

a!

b!

Blurring a

112!

slide by Derek Hoiem!

processed!

original!

L!

a!

b!processed!

Blurring b

113!

slide by Derek Hoiem!

Cascading Gaussian Filters •  Removes high-frequency components from the

image (low-pass filter)!•  Convolution of two Gaussians is another Gaussian !

–  Convolving two times with Gaussian kernel of size σ is same as convolving once with kernel of size !

*" ="

114!

slide by Kristen Graum

an!

Gaussian Filters •  What parameters matter here?!•  Size of kernel or mask !

σ = 5 with !10 x 10 kernel !

σ = 5 with !30 x 30 kernel !

115!

slide by Derek Hoiem!

How big should the filter be?!•  Gaussian function has infinite �support� but need a finite-size

kernel!•  Values at edges should be near 0 !•  ∼98.8% of area under Gaussian in mask of size 5σ x 5σ!•  In practice, use mask of size 2k+1 x 2k+1 where k ≈ 3σ !•  Multiply real values of Gaussian by a scale factor (= min real

value) to obtain integer weights!•  Normalize output by dividing by sum of all weights!

Practical Matters

116!

slide by Derek Hoiem!

Gaussian Filter

1 2 1

2 4 2

1 2 1

1/16

3 x 3 approximation of a Gaussian:!

117!

slide by Derek Hoiem!

Matlab Implementation >> hsize = 10; >> sigma = 5; >> h = fspecial(�gaussian��hsize, sigma); >> mesh(h); >> imagesc(h); >> outim = imfilter(im, h); % correlation >> imshow(outim);

outim!original!

hsize = width of mask!

118!

slide by Derek Hoiem!

Practical Matters •  What is the size of the output? !•  MATLAB: filter2(g, f, shape)

–  shape = �full�:%output size is sum of sizes of f and g!–  shape = �same�:%output size is same as f!–  shape = �valid�: output size is difference of sizes of f and g !

f"

g"g"

g"g"

f"

g"g"

g"g"

f"

g"g"

g"g"

full same valid

119!

slide by Svetlana Lazebnik!

Practical Matters •  What about near the edge? !

–  the filter window falls off the edge of the image !– need to extrapolate!– methods:!

•  clip filter (black)!•  wrap around!•  copy edge!•  reflect across edge!

120!

slide by Steven Marschner!

Practical Matters

– methods (MATLAB):!•  clip filter (black): imfilter(f, g, 0)•  wrap around: imfilter(f, g, �circular�)

•  copy edge: imfilter(f, g, �replicate�)

•  reflect across edge: imfilter(f, g, �symmetric�)

121!

slide by Steven Marschner!

Application: Filter Banks for Feature Detection

LM Filter Bank!

Code for filter banks: www.robots.ox.ac.uk/~vgg/research/texclass/filters.html!! 122!

slide by Derek Hoiem!

Filter Banks •  Process image with each filter and keep responses

(or squared/abs responses)!

123!

slide by Derek Hoiem!

Application: Hybrid Images Gaussian Filter

Laplacian Filter

A. Oliva, A. Torralba, P.G. Schyns, !“Hybrid Images,” SIGGRAPH 2006!

Gaussian unit impulse Laplacian of Gaussian

I1

I2

G1

(1-G2)

I1 * G1 slide by Chuck Dyer! 124!

slide by Chuck Dyer! 125!

slide by Chuck Dyer! 126!

What Makes the Mona Lisa Smile?

"The elusive quality of the Mona Lisa's smile can be explained by the fact that her smile is almost entirely in low spatial frequencies, and so is seen best by your peripheral vision“!

-- Margaret Livingstone !

The smile only becomes apparent if a viewer looks at her eyes or elsewhere on her face; the smile disappears when looking directly at her mouth!!Peripheral vision is low resolution and blurs, picking up shadows from the Mona Lisa's cheekbones, which suggests the curvature of a smile !

slide by Chuck Dyer! 127!

What Makes the Mona Lisa Smile?

128!

Non-Linear Filtering

129!

Median Filter

•  Replace pixel by the median value of its neighbors!

•  No new pixel values introduced!

•  Removes spikes: good for impulse, salt & pepper noise

•  Nonlinear filter

130!

slide by Derek Hoiem!

Median Filter Salt and pepper noise!

Median filtered!

Plots of a row of the image!Matlab: output im = medfilt2(im, [h w]);

131!

slide by Martial Hebert!

Median Filter •  Median filter is edge preserving!

132!

slide by Derek Hoiem!

133!

Next week •  Texture synthesis!•  Video textures!

134!