Upload
vuongmien
View
217
Download
0
Embed Size (px)
Citation preview
Review: Image warping and morphing• Forward vs. inverse warping
f(x,y) g(x’,y’)
Forward warping
Send each pixel f(x,y) to its corresponding location (x’,y’) = T(x,y) in the second image
x x’
T(x,y)
Q: what if pixel lands “between” two pixels?
y y’
Slide credit: S. Seitz
f(x,y) g(x’,y’)
Forward warping
Send each pixel f(x,y) to its corresponding location (x’,y’) = T(x,y) in the second image
x x’
T(x,y)
Q: what if pixel lands “between” two pixels?
y y’
A: distribute color among neighboring pixels (x’,y’)– Known as “splatting”– Check out griddata in Matlab Slide credit: S. Seitz
f(x,y) g(x’,y’)xy
Inverse warping
Get each pixel g(x’,y’) from its corresponding location (x,y) = T-1(x’,y’) in the first image
x x’
Q: what if pixel comes from “between” two pixels?
y’T-1(x,y)
Slide credit: S. Seitz
f(x,y) g(x’,y’)xy
Inverse warping
Get each pixel g(x’,y’) from its corresponding location (x,y) = T-1(x’,y’) in the first image
x x’
T-1(x,y)
Q: what if pixel comes from “between” two pixels?
y’
A: Interpolate color value from neighbors– nearest neighbor, bilinear, Gaussian, bicubic– Check out interp2 in Matlab Slide credit: S. Seitz
Forward vs. inverse warpingQ: which is better?
A: usually inverse—eliminates holes• however, it requires an invertible warp function—not always possible...
Slide credit: S. Seitz
Review: Image morphing
Michael Jackson’s “Black or White” (1991)
How do we create a morphing sequence?
1. Define corresponding control points on both images2. For each intermediate image:
1. Create an intermediate shape (by interpolation)2. Warp both images towards it3. Cross-dissolve the colors between the newly warped
images
morphing
Morphing example
image #1 image #2
warp warp
Morphing example
Fun with faces
How do we average faces?
http://www2.imm.dtu.dk/~aam/datasets/datasets.html
Sir Francis Galton: Composite Portraits
Composite portraits showing "features common among men convicted of crimes of violence," by Francis Galton, 1885
Sir Francis Galton: Composite portraits
“The Jewish Type”
Sir Francis Galton: Composite portraits
“The Jewish Type”
Jason Salavon: “The Late Night Triad”
http://www.salavon.com/
Jason Salavon: “100 Special Moments”
Jason Salavon: “Every Playboy Centerfold”
Is there a better way to average faces?
Average Face
1. Warp to mean shape2. Average pixels
http://graphics.cs.cmu.edu/courses/15-463/2004_fall/www/handins/brh/final/ Slide credit: A. Efros
Representing the shape of a face
Slide credit: A. Efros
Representing the shape of a face
Vector of normalized landmark coordinates (or inter-landmark distances)
Slide credit: A. Efros
Subpopulation means
• 250 control points per face manually specified
Average male(32 faces)
Average female(64 faces)
http://www.beautycheck.de
Manipulating faces
• How can we make a face look more male/female, young/old, happy/sad, etc.?1. Obtain the two prototypes spanning the desired axis
(gender, age, expression, etc.) and find the difference vector between them
2. Add scaled versions of this vector to a given image
Prototype 1
Prototype 2Current face
D. Rowland and D. Perrett, “Manipulating Facial Appearance through Shape and Color,” IEEE CG&A, September 1995
Changing gender
Deform shape and/or color of an input face in the direction of “more female”
original shape
color both
D. Rowland and D. Perrett, “Manipulating Facial Appearance through Shape and Color,” IEEE CG&A, September 1995
Slide credit: A. Efros
Enhancing gender
more same original androgynous more opposite
D. Rowland and D. Perrett, “Manipulating Facial Appearance through Shape and Color,” IEEE CG&A, September 1995
Slide credit: A. Efros
Changing age
Face becomes “rounder” and “more textured” and “grayer” original shape
color both
D. Rowland and D. Perrett, “Manipulating Facial Appearance through Shape and Color,” IEEE CG&A, September 1995
Slide credit: A. Efros
Which face is more attractive?
http://www.beautycheck.de
Which face is more attractive?
attractive0.5(attractive + average)
Which face is more attractive?
http://www.beautycheck.de
Which face is more attractive?
0.5(adult+child) adult
Effect of skin texture
average shape + unattractive texture
average shape + average texture
Digital beautification – Video http://www.youtube.com/watch?v=iYhCn0jf46U
Data-Driven Enhancement of Facial Attractiveness
T. Leyvand, D. Cohen-Or, G. Dror, and D. Lischinski
SIGGRAPH 2008
System Overview
Feature extraction• Active shape model (Cootes et al. 1995)
• Represents distribution of shape (facial features represented by splines) and appearance (pattern of local gradients)
• Initialized from manually labeled training data• At test time, need good initial estimate of face location, scale
and orientation – use a face detector• 84 keypoints are converted into a planar graph of 234 inter-
point distances
Average and scatterfrom training set
Results of ASM search
Graph of distances
Regression of facial attractiveness model• Input: training set of distance vectors of faces rated
on a scale of 1 to 7 by human subjects (Eisenthal et al. 2006)• 92 female and 33 male photographs
• Support vector regression: learn smooth function f(v) to estimate attractiveness ranking
Beautification• Problem: given a vector v and estimated attractiveness
rating f(v), find nearby vector v’ such that f(v’) > f(v)• Method 1 (KNN): move the face vector in the direction of the
beauty-weighted average of K nearest neighbors of that face:
• Method 2 (SVR): perform gradient ascent of f starting at v• Results
• NN-based beautification worked better on male faces, but SVR-based beautification worked better on female faces
• On average, estimated beauty score was increased by 30%
,beauty score of the ith neighbor
Choosing K in KNN beautification
Comparison of KNN and SVR beautification
Additional details• Distance embedding
• Given a vector of distances v’ = {dij}, find a set of points {qi} that conform to these distances:
• Avoiding pupil distortion• Add post-process to enforce similarity transform for feature
points on the eyes
• Warping: Multilevel Free-Form Deformation (Lee et al. 1996)• Uses B-splines, avoids fold-overs
Interactive system: Video
Which face is more attractive?
Which face is more attractive?
originalbeautified
Which face is more attractive?
Which face is more attractive?
original beautified
Which face is more attractive?
Which face is more attractive?
original beautified
Which face is more attractive?
Which face is more attractive?
beautified original
Which face is more attractive?
Which face is more attractive?
original beautified
Which face is more attractive?
Which face is more attractive?
original beautified
Which face is more attractive?
Which face is more attractive?
beautified original
Results of user study• 92 face pairs (47 female, 45 male)• 68 raters, males and females aged between 25 and 40
• “For all 47 portraits of females, the beautified versions were chosen as more attractive by most raters, and in half of the cases the beautified versions were preferred by more than 80 percent of the raters. As for the male portraits, 69 percent of the beautified versions were chosen as more attractive.”
Application: Construction of facial collages
initial collage Poisson blending beautification
Face Swapping: Automatically Replacing Faces in Photographs
D. Bitouk, N. Kumar, S. Dhillon, P. Belhumeur, S. K. Nayar
SIGGRAPH 2008
Outline• Offline: create face library• Given a new query:
• Detect faces• For each face:
– Retrieve candidate replacement faces from the library– Apply recoloring and relighting– Perform face replacement– Rank replacement results
Creating a face library• Download images from the Internet and process them
using a commercial face detector• Approximately 33,000 faces found• Detector returns face pose (yaw, pitch, roll) and six fiducial points• Bin faces according to yaw and pitch angles, align to a common
coordinate system using fiducial points
Candidate selection• Detect faces in new image, estimate their pose
angles and fiducial points• Matching criteria:
• Pose: yaw and pitch angles• Resolution: inter-eye distance • Blur: histogram of gradient magnitudes in the eye region• Lighting: approximate the face by a Lambertian cylinder
under orthographic projection, estimate face albedos and coefficients for an orthonormal illumination basis
• Seam signature: ribbon of pixels within a small radius of the replacement boundary (resampled, intensity normalized)
• Retrieval• SQL query for pose, resolution, and albedo• Further prune using blur and lighting distance• Select top 50 final candidates using seam distance=> Less than 1 sec to generate list of candidate replacements
Candidate selection
• Approximate face images:
• Relit replacement:
• After relighting, match the color histograms of the two faces
Relighting and recoloring
albedo
illumination coefficient
illumination basis image
Relighting and recoloring
Face replacement: Final steps• Rank replacements by color difference along the
overlap region• Perform feathering along the overlap region
Face replacement results
Face replacement results
Face replacement results
Face de-identification
Face de-identification
More results
More results
Improving family portraits
Limitations
Limitations
User study• 50 face images, 12 subjects, 5 seconds per image
Aside: Detecting composites of faces
M.K. Johnson and H. Farid, “Exposing Digital Forgeries Through SpecularHighlights on the Eye,” 9th International Workshop on Information Hiding, 2007
More fun with faces
Example-Based Cosmetic Transfer
W.-S. Tong, C.-K. Tang, M. Brown, Y.-Q. XuPacific Graphics 2007
More fun with faces
A Conditional Random Field for Automatic Photo EditingM. Brand, P. Pletscher
CVPR 2008
More fun with faces
Expressive Expression Mapping with Ratio ImagesZ. Liu, Y. Shan, Z. Zhang
SIGGRAPH 2001
Original image Expression transfer by warping
Proposed method