1

Capturing Facial Details by Space-time Shape-from-shading

Yung-Sheng Lo*, I-Chen Lin*, Wen-Xing Zhang*, Wen-Chih Tai†, Shian-Jun Chiou†

CAIG Lab, Dept. of CS, National Chiao Tung University*

Chunghwa Picture Tubes, LTD.†

2

Outline Introduction Acquisition of facial motion Space-time shape-from-shading Experiment and results Conclusion

3

Introduction Performance-driven method is one of the most

straightforward method for facial animation.

Expression details, e.g. wrinkles, dimples, are key factors but difficult for motion capture.

Original captured images Deformation without details

4

Introduction (cont.) Physics-based simulation and blend shape methods try

to mimic the details.

But, the synthesized details are not the exact expressions.

Muscle-based [E. Sifakis et al. 2005] Blend shape [Z. Deng et al. 2006]

5

Introduction (cont.) Our goal is to enhance existing motion capture tech.

and capture facial details.

With the captured images and directional lighting, our optimization-based shape-from-shading (SFS) can estimate details from shading in video.

With facial detailsCaptured video

6

The proposed method Combines the benefits of motion

capture and shape-from-shading. Motion capture and stereo reconstruction

accurate on feature points and general geometry.

Unreliable corresponding matching at textureless regions

Shape-from-shading don’t need detailed point correspondence for

textureless regions. Estimate relative undulation. Sensitive to noise.

Motion capture + Space-time shape-from-shading.

7

The proposed method

8

Approximate geometry by Mocap Tracking by block matching and stereo reconstruction.

Deforming a generic face model by radial-basis functions (RBF).

9

Facial details by SFS Estimating time-varying details by iterative

approximating shape V and reflectance R.

Input image

T PNum

t

Num

ptptp ISynRVO

1 1

2),(

Space-time constraints Only SFS is not enough. For more reliable detailed motions, we proposed

using space-time constraints.

10

Highly sensitive to noise After applying our spatial constraints

Spatial constraints Mostly continuous surface

High spatial coherence

11

j

tjj

tpCStp zw

zkCS 2)1(

∆ For J Є Neighbor(p)∆ Neighbor(p) denotes the 8-neighbor pixel set ∆ Wj is an adaptive weight ∆ Kcs is the weight for spatial constraints.

Reduce the noise

noiseZtp

Ztj

Ztp

Ztj

noise

ZtpZtj

Still flicker According to biomechanics properties:

A human facial surface should gradually transit between expressions.

Temporal constraints

12T0T1

flicker

T2

]3,3[ i ,)1( 2)( wherez

wzkCT

ipittpCTtp

T2T0

A video image sequence

Space-time shape-from-shading

Finally, our objective function becomes

13

])[(1 1

2tptp

Num

t

Num

ptptp CTCSISynO

T p

spatial constraints

+ temporal constraints

+shading constraints

=Space-time shape-from-shading

Performance issue if applied our optimization to the whole face.

DOF is too large Assigned some small windows.

We preferred areas with more wrinkles and creases.

14D.O.F=N*M (pixels) *i(frames)

N

M

Video image sequence

Fi

… F2F1

F3

Experiment Illumination-controlled (single light source) Two video streams. (HDV, 1280*720 ,30 fps)

We pasted 25 to 30 markers on human face.

15

{C1}

{C2}

Facial detailed results and Comparison

16

Result of synthesis

Generic model: 6078 vertices 6315 polygons

17deformation (RBF)subdivision per-pixel normal mapping

Result of animation

18

Conclusion & Future work We propose capturing detailed motion by

conventional Mocap and advanced shape-from-shading. Doesn’t need additional devices, paint pigments, or

restrict the wrinkle shape.

With spatial and temporal constraints, our optimal shape-from-shading is more reliable. Reflectance parameters are also estimated.

19

Conclusion & Future work In addition to Phong model, we will extend the

concept to other reflectance models. E.g Cook-Torrance BRDF model, BSSRDF, etc)

Currently, SFS is only applied to designated segments. An more efficient SFS for the whole face will make our animation more realistic.

20

Thank for your attention!

21

forehead details

between eyebrows

smile