Face Poser:Interactive Modeling of 3D Facial Expressions Using Model Priors

Manfred Lau1,3 Jinxiang Chai2 Ying-Qing Xu3 Heung-Yeung Shum3

1Carnegie Mellon University 2Texas A&M University 3Microsoft Research Asia

Face Poser

Inputs

Generate new facial expressions with a simple and intuitive interface

Face Poser

Inputs Output

Generate new facial expressions with a simple and intuitive interface

Why Face Poser?

Pre-defined controlsDifficult to build and use

Complex facial expressions

Applications

Films, Games Virtual Reality

Educational

Related Work

Sketched-based interfacesZeleznik et al. 96

Igarashi et al. 99Nealen et al. 05Kho and Garland 05Chang and Jenkins 06

Nealen et al. 05

Igarashi et al. 99

Related Work

Example-based modelingBlanz and Vetter 99

Pighin et al. 99Chai et al. 03Zhang et al. 04Grochow et al. 04Sumner et al. 05

Sumner et al. 05

Grochow et al. 04

OverviewDatabase

PreprocessingModel Prior

OverviewDatabase

PreprocessingModel Prior

Neutral Pose

User Constraints

Interface

OverviewDatabase

PreprocessingModel Prior

RuntimeOptimization

Neutral Pose

New Pose

User Constraints

Interface

Textured Pose

Motion capture data

Captured mesh animations of various facial expressions: anger, fear, surprise, sadness, joy, disgust, speaking, singing

All meshes translated and rotated to a standard view:

Data: PCA representation

x =

v1x

v1y

v1z

v2x

. . .

p is low-dimensional representation of x

Problem statement

Find best p satisfying user-constraints c:

Best p is:

Given a face model, how well does it match user-constraints

Likelihood of face model using knowledge of data

Point Constraints

More detailed control

User inputs:blue – 3D source vertexgreen – 2D target pixel

Can select in any camera view

Point Constraints

We optimize for best p

For each p: compute whole mesh x take selected 3D source vertex project it to 2D screen space compare to target pixel

Point Constraints

Optimization term:

Jacobian term:

Point Constraints

Inputs Solution

Point Constraints – Results

Point Constraints – Dragging interface

Stroke Constraints

Large-scale changes with minimal input

User inputs:blue – 2D source stroke (selects

3D points on mesh)green – 2D target stroke

Any curve, line, or freeform region

Stroke Constraints

2D source stroke raytrace each pixel to mesh surface to get dark gray points

These can be 3D points on mesh surface (not just original mesh vertices)

Stroke Constraints

We optimize for best p

For each p: compute whole mesh x take selected 3D points project them to 2D screen space compare to target stroke

Stroke Constraints

Optimization term:

Jacobian term:

Stroke Constraints

Inputs Solution

Stroke Constraints – Results

Stroke Constraints – Tablet interface

Stroke Constraints – Additional term

If strokes are far away from each other, energy term will reach local minimum

Need additional optimization term to minimize distance between “center” of source stroke and “center” of target stroke

Without additional term

Stroke Constraints – Additional term

Without additional term

Optimization term:

Jacobian term:

Stroke Constraints – Additional term

Without additional term With additional term

Stroke Constraints – Results

Problem statement

Find best p satisfying user-constraints c:

Best p is:

Given a face model, how well does it match user-constraints

Likelihood of face model using knowledge of data

Model Priors

There can be many solutions satisfying user constraints. Some of them are not realistic.

We add another optimization term to constrain the solution to the space defined by the motion capture data.

Without model priors term

Model Priors

Without model priors term

Learn a Mixtures of Factor Analyzers (MFA) model

Probability density function to measure naturalness of facial expression

MFA has been applied to high-dimensional nonlinear data

Model Priors

Without model priors term

Optimization term:

Jacobian term:

Model Priors – Result

Model Priors – Result

increasing weight on Model Prior term

Model Priors – Result

Computation time

Standard PC hardware (Pentium IV 2 GHz)

Point constraintstakes 0.18 seconds for 10 pointstime increases linearly with number of points

Stroke constraintstakes 0.4 seconds for source stroke of ~900 pixels (about

size of eyebrow)time increases linearly with number of pixelsfaster if using intermediate spline representation

Cross validation

New face expression samples for testingUse new samples to get target constraintsGenerate solution and compare with test sample

Cross validation

Ground truth Interpolation Optimization

Comparison with other techniques

Opt-blend: FaceIK [Zhang et al. 04]PCA: Morphable model [Blanz and Vetter 99; Blanz et al. 03]LWR: Locally weighted regression

3D errors

Comparison with other techniques

Ground truth, Optimization with PCA, Optimization with MFA

Application: Trajectory Keyframing

Green points – given 2D target pixelsBlue points and mesh – solution

Application: Trajectory Keyframing

Ground truth Result

Application: Trajectory Keyframing

Ground truth Result

Application: Trajectory Keyframing

Ground truth Result

Summary: Face Poser

Inputs Output

Users can learn to use our system within minutes and can create new facial expressions within seconds

Limitation

Global control changing mouth

also changes eyes this is natural, but

difficult to control sometimes

Local control changing mouth

without changing eyes

but this might lead to “fake smiles”

Extensions / Future work

We have added different types of constraints within the same optimization framework

More general: model face as separate regions, generate each region separately, and blend them back together