Image Processing and Morphing

Vision for GraphicsCSE 590SS, Winter 2001

Richard Szeliski

Image Pyramids

1/8/2001 Vision for Graphics 3

Image Pyramids

1/8/2001 Vision for Graphics 4

Pyramid Creation

“Laplacian” Pyramid• Created from Gaussian

pyramid by subtractionLl = Gl – expand(Gl+1)

filter mask

“Gaussian” Pyramid

1/8/2001 Vision for Graphics 5

Octaves in the Spatial Domain

Bandpass Images

Lowpass Images

1/8/2001 Vision for Graphics 6

Pyramids

Advantages of pyramids• Faster than Fourier transform• Avoids “ringing” artifacts

Many applications• small images faster to process• good for multiresolution processing• compression• progressive transmission

Known as “mip-maps” in graphics communityPrecursor to wavelets

• Wavelets also have these advantages

1/8/2001 Vision for Graphics 7

laplacianlevel

4

laplacianlevel

2

laplacianlevel

0

left pyramid right pyramid blended pyramid

1/8/2001 Vision for Graphics 8

Pyramid Blending

Image Warping

1/8/2001 Vision for Graphics 10

Image Warping

image filtering: change range of imageg(x) = h(f(x))

image warping: change domain of imageg(x) = f(h(x))

f

x

hf

x

f

x

hf

x

1/8/2001 Vision for Graphics 11

Image Warping

image filtering: change range of imageg(x) = h(f(x))

image warping: change domain of imageg(x) = f(h(x))

h

h

f

f g

g

1/8/2001 Vision for Graphics 12

Parametric (global) warping

Examples of parametric warps:

translation rotation aspect

affineperspective

cylindrical

1/8/2001 Vision for Graphics 13

2D coordinate transformations

translation: x’ = x + t x = (x,y)

rotation: x’ = R x + t

similarity: x’ = s R x + t

affine: x’ = A x + t

perspective: x’ H x x = (x,y,1)(x is a homogeneous coordinate)

These all form a nested group (closed w/ inv.)

1/8/2001 Vision for Graphics 14

Image Warping

Given a coordinate transform x’ = h(x) and a source image f(x), how do we compute a transformed image g(x’) = f(h(x))?

f(x) g(x’)x x’

h(x)

1/8/2001 Vision for Graphics 15

Forward Warping

Send each pixel f(x) to its corresponding location x’ = h(x) in g(x’)

f(x) g(x’)x x’

h(x)

• What if pixel lands “between” two pixels?

1/8/2001 Vision for Graphics 16

Forward Warping

Send each pixel f(x) to its corresponding location x’ = h(x) in g(x’)

f(x) g(x’)x x’

h(x)

• What if pixel lands “between” two pixels?• Answer: add “contribution” to several pixels,

normalize later (splatting)

1/8/2001 Vision for Graphics 17

Inverse Warping

Get each pixel g(x’) from its corresponding location x = h-1(x’) in f(x)

f(x) g(x’)x x’

h-1(x’)

• What if pixel comes from “between” two pixels?

1/8/2001 Vision for Graphics 18

Inverse Warping

Get each pixel g(x’) from its corresponding location x = h-1(x’) in f(x)

• What if pixel comes from “between” two pixels?• Answer: resample color value from

interpolated (prefiltered) source image

f(x) g(x’)x x’

1/8/2001 Vision for Graphics 19

Interpolation

Possible interpolation filters:• nearest neighbor• bilinear• bicubic (interpolating)• sinc / FIR

Needed to prevent “jaggies” and “texture crawl” (see demo)

1/8/2001 Vision for Graphics 20

Prefiltering

Essential for downsampling (decimation) to prevent aliasing

MIP-mapping [Williams’83]:1. build pyramid (but what decimation filter?):

– block averaging– Burt & Adelson (5-tap binomial)– 7-tap wavelet-based filter (better)

2. trilinear interpolation– bilinear within each 2 adjacent levels– linear blend between levels (determined by pixel size)

1/8/2001 Vision for Graphics 21

Prefiltering

Essential for downsampling (decimation) to prevent aliasing

Other possibilities:• summed area tables• elliptically weighted Gaussians (EWA)

[Heckbert’86]

1/8/2001 Vision for Graphics 22

Image Warping – non-parametric

Specify more detailed warp function

Examples: • splines• triangles• optical flow (per-pixel motion)

1/8/2001 Vision for Graphics 23

Image Warping – non-parametric

Move control points to specify spline warp

Image Morphing

1/8/2001 Vision for Graphics 25

Image Morphing

How can we in-between two images?1. Cross-dissolve

(all examples from [Gomes et al.’99])

1/8/2001 Vision for Graphics 26

Image Morphing

How can we in-between two images?2. Warp then cross-dissolve = morph

1/8/2001 Vision for Graphics 27

Warp specification

How can we specify the warp?1. Specify corresponding points

• interpolate to a complete warping function

• Nielson, Scattered Data Modeling, IEEE CG&A’93]

1/8/2001 Vision for Graphics 28

Warp specification

How can we specify the warp?2. Specify corresponding vectors

• interpolate to a complete warping function

1/8/2001 Vision for Graphics 29

Warp specification

How can we specify the warp?2. Specify corresponding vectors

• interpolate [Beier & Neely, SIGGRAPH’92]

1/8/2001 Vision for Graphics 30

Warp specification

How can we specify the warp?3. Specify corresponding spline control points

• interpolate to a complete warping function

1/8/2001 Vision for Graphics 31

Final Morph Result

Image Mosaics

1/8/2001 Vision for Graphics 33

Image Mosaics (stitching)

Blend together several overlapping images into one seamless mosaic (composite)

+ + … +=

1/8/2001 Vision for Graphics

Mosaics for Video CodingConvert masked images into a background sprite

for content-based coding

+ + +

=

1/8/2001 Vision for Graphics 35

f = 180 (pixels)

Cylindrical Panoramas

Map image to cylindrical or spherical coordinates• need known focal length

Image 384x300 f = 380f = 280

1/8/2001 Vision for Graphics 36

Cylindrical warping

Given focal length f and image center (xc,yc)

X

YZ

(X,Y,Z)

(sin,h,cos)

1/8/2001 Vision for Graphics 37

Radial distortion

Correct for “bending” in wide field of view lenses

1/8/2001 Vision for Graphics 38

Image Stitching

1. Align the images over each other

2. Blend the images together (demo)

1/8/2001 Vision for Graphics 39

Image Stitching Assignment

1. Take pictures on a tripod (or handheld)

2. Warp to cylindrical coordinates

3. Automatically compute pairwise alignments

4. Fix up the end-to-end alignment

5. Blend the images together

6. Crop the result and import into a viewer

1/8/2001 Vision for Graphics 40

Image registration (preview)

How do we determine alignment between images?

• One possible answer: block matching (correlation),

i.e., find minimum squared error

2),(

01 ),(),(),( yx

yxIvyuxIvuE

1/8/2001 Vision for Graphics 41

Image registration (preview)

How do we determine alignment between images?

• Another possible answer: Fourier-domain correlation

[Brown’92] or phase correlation [Kuglin & Hines’75]

• <…discuss Fourier-domain convolution formulas…>

• More on alignment/registration on Wednesday

[Anandan]

1/8/2001 Vision for Graphics 42

Full-view Panorama

++

++

++

++

1/8/2001 Vision for Graphics 43

Texture Mapped Model

Image Enhancement

1/8/2001 Vision for Graphics 45

Image Enhancement

Noise removal:• low-pass filtering

g(x) = h(x) f(x) = ih(-i) f(x+i)

• median filtering• anisotropic diffusion

– iteratively smooth with similar neighbors [Perona & Malik’90]

Sharpening:• “unsharp masking”

– remove a little bit of blurred image from original(darkroom trick)

1/8/2001 Vision for Graphics 46

Image Enhancement

Super-resolution:• hallucinate more detail from

single low-res image[Freeman & Pasztor,ICCV’99]

– “learn” mapping between levels

• combine several low-res images using known (linear) formation model [Irani’91;Mann’94]

1/8/2001 Vision for Graphics 47

Image Enhancement

Super-resolution:• [Freeman &

Pasztor,ICCV’99]– mapping between

levels depends on “training data”

1/8/2001 Vision for Graphics 48

Image Enhancement

Brightness and gamma correction• linearize input brightness

Histogram equalization• balance (“flatten”) distribution

of brightness values

1/8/2001 Vision for Graphics 49

Image Enhancement

High dynamic range photography[Debevec et al.’97; Mitsunaga & Nayar’99]• combine several different exposures together

1/8/2001 Vision for Graphics 52

BibliographyL. Williams. Pyramidal parametrics.Computer Graphics, 17(3):1--11, July 1983.

L. G. Brown. A survey of image registration techniques.Computing Surveys, 24(4):325--376, December 1992.

C. D. Kuglin and D. C. Hines. The phase correlation image alignment method.In IEEE 1975 Conference on Cybernetics and Society, pages 163--165, New York, September 1975.

J. Gomes, L. Darsa, B. Costa, and L. Velho. Warping and Morphing of Graphical Objects.

Morgan Kaufmann Publishers, San Francisco Altos, California, 1999.

G. M. Nielson. Scattered data modeling.IEEE Computer Graphics and Applications, 13(1):60--70, January 1993.

T. Beier and S. Neely. Feature-based image metamorphosis.Computer Graphics (SIGGRAPH'92), 26(2):35--42, July 1992.

1/8/2001 Vision for Graphics 53

BibliographyL. Williams. Performace driven facial animation.Computer Graphics, 24(4):235--242, 1990.

P. Perona and J. Malik. Scale space and edge detection using anisotropic diffusion.IEEE Transactions on Pattern Analysis and Machine Intelligence, 12(7):629--639, July 1990.

W. T. Freeman and E. C. Pasztor. Learning low-level vision.In Seventh International Conference on Computer Vision (ICCV'99), pages 1182--1189, Kerkyra, Greece, September 1999.

M. Irani and S. Peleg. Improving resolution by image registration.Graphical Models and Image Processing, 53(3):231--239, May 1991.

S. Mann and R. W. Picard. Virtual bellows: Constructing high-quality images from video.

In First IEEE International Conference on Image Processing (ICIP-94), volume I, pages 363--367, Austin, Texas, November 1994.

1/8/2001 Vision for Graphics 54

BibliographyPaul E. Debevec and Jitendra Malik.Recovering high dynamic range radiance maps from photographs.Proceedings of SIGGRAPH 97, pages 369--378, August 1997.ISBN 0-89791-896-7. Held in Los Angeles, California.

T. Mitsunaga and S. K. Nayar.Radiometric self calibration.In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'99), volume 1, pages 374--380, Fort Collins, June 1999.