37
Realistic Modeling of Animatable Faces in MPEG-4 Marco Fratarcangeli and Marco Schaerf University of Rome “La Sapienza”

Realistic Modeling of Animatable Faces in MPEG-4 Marco Fratarcangeli and Marco Schaerf University of Rome “La Sapienza”

Embed Size (px)

Citation preview

Page 1: Realistic Modeling of Animatable Faces in MPEG-4 Marco Fratarcangeli and Marco Schaerf University of Rome “La Sapienza”

Realistic Modeling of Animatable Faces in MPEG-4

Marco Fratarcangeli and Marco SchaerfUniversity of Rome “La Sapienza”

Page 2: Realistic Modeling of Animatable Faces in MPEG-4 Marco Fratarcangeli and Marco Schaerf University of Rome “La Sapienza”

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

2

Outline

Definition of the Problem Introducing MPEG-4 FBA Facial Structure Overview Numerical Simulation Results and Conclusions

Page 3: Realistic Modeling of Animatable Faces in MPEG-4 Marco Fratarcangeli and Marco Schaerf University of Rome “La Sapienza”

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

3

Definition of the Problem

MPEG-4 FBA standard mainly focuses on networking capabilities for sending/ receiving animations.

It allows employing model-based coding of human-like faces in a systematic manner.

It does not provide any solution on how to achieve the movements of the face/body.

Page 4: Realistic Modeling of Animatable Faces in MPEG-4 Marco Fratarcangeli and Marco Schaerf University of Rome “La Sapienza”

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

4

Definition of the Problem

We use some already developed techniques to build a face model conforming to the anatomical structure of the human head.

Our contribution is in devising a fast and automatic building process using MPEG-4.

The resulting virtual talking head is used to perform general facial animation coming from an encoded MPEG-4 FAB stream.

Page 5: Realistic Modeling of Animatable Faces in MPEG-4 Marco Fratarcangeli and Marco Schaerf University of Rome “La Sapienza”

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

5

Definition of the Problem

MPEG-4 FBA Encoded Stream

MPEG-4 Player

Animatable Face Model

Modeling Process

3D Mesh + Info MPEG-4 FBA

Output

Page 6: Realistic Modeling of Animatable Faces in MPEG-4 Marco Fratarcangeli and Marco Schaerf University of Rome “La Sapienza”

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

6

Outline

Definition of the Problem Introducing MPEG-4 FBA Facial Structure Overview Numerical Simulation Results and Conclusions

Page 7: Realistic Modeling of Animatable Faces in MPEG-4 Marco Fratarcangeli and Marco Schaerf University of Rome “La Sapienza”

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

7

Introducing MPEG-4 FBA

MPEG-4Standard dealing with efficient

transmission of multimedia on the net

FBAEfficient coding of shape and

animation of human-like faces and bodies

FDPControl points used to define

the shape of a proprietary face model

FAPSet of animation parameters

suitable to define facial movements

Page 8: Realistic Modeling of Animatable Faces in MPEG-4 Marco Fratarcangeli and Marco Schaerf University of Rome “La Sapienza”

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

8

MPEG-4 Facial Definition Pts

Page 9: Realistic Modeling of Animatable Faces in MPEG-4 Marco Fratarcangeli and Marco Schaerf University of Rome “La Sapienza”

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

9

MPEG-4 Facial Animation Pts

68 low-level FAPs used to express basic actions that the face can perform

20 high-level FAPs used to express more complex movements like expressions and visemes

Page 10: Realistic Modeling of Animatable Faces in MPEG-4 Marco Fratarcangeli and Marco Schaerf University of Rome “La Sapienza”

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

10

Outline

Definition of the Problem Introducing MPEG-4 FBA Facial Structure Overview Numerical Simulation Results and Conclusions

Page 11: Realistic Modeling of Animatable Faces in MPEG-4 Marco Fratarcangeli and Marco Schaerf University of Rome “La Sapienza”

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

11

Facial Structure OverviewFrom a VRML model

to a mass-spring structure

Skin structure{Masses [or nodes] from VRML vertices

Springs from VRML topologyActing on the skin model through muscle actions, movement is achieved

Page 12: Realistic Modeling of Animatable Faces in MPEG-4 Marco Fratarcangeli and Marco Schaerf University of Rome “La Sapienza”

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

12

Skin is composed by 3 layers

Skin – Prism Element

subdivided in little triangular prisms

Page 13: Realistic Modeling of Animatable Faces in MPEG-4 Marco Fratarcangeli and Marco Schaerf University of Rome “La Sapienza”

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

13

Skin – Prism Element

Page 14: Realistic Modeling of Animatable Faces in MPEG-4 Marco Fratarcangeli and Marco Schaerf University of Rome “La Sapienza”

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

14

From FDPs

Muscles – Mapping

muscles are mapped automatically

Green points are where the muscle is attached to the skull

Red points are where the muscle is inserted in the skin

Two kinds of muscle: linear and sphincter

Page 15: Realistic Modeling of Animatable Faces in MPEG-4 Marco Fratarcangeli and Marco Schaerf University of Rome “La Sapienza”

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

15

Muscle Mapping - Examples

Page 16: Realistic Modeling of Animatable Faces in MPEG-4 Marco Fratarcangeli and Marco Schaerf University of Rome “La Sapienza”

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

16

Outline

Definition of the Problem Introducing MPEG-4 FBA Facial Structure Overview Numerical Simulation Results and Conclusions

Page 17: Realistic Modeling of Animatable Faces in MPEG-4 Marco Fratarcangeli and Marco Schaerf University of Rome “La Sapienza”

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

17

Semi-Implicit Integrations

A ordinary differential equation system governs the movement of the spring-mass network.

Explicit methods are simple but unstable if applied to stiff systems [e.g. Euler steps].

Implicit methods are precise and stable but computationally expensive [e.g. Backward Euler].

Semi-Implicit Integration = implicit integration approximated with some explicit steps.

Page 18: Realistic Modeling of Animatable Faces in MPEG-4 Marco Fratarcangeli and Marco Schaerf University of Rome “La Sapienza”

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

18

Skin nodes can be displaced by muscular force and by the elastic wave coming from the muscle movement.

Adaptive Displacement

Page 19: Realistic Modeling of Animatable Faces in MPEG-4 Marco Fratarcangeli and Marco Schaerf University of Rome “La Sapienza”

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

19

Outline

Definition of the Problem Introducing MPEG-4 FBA Facial Structure Overview Numerical Simulation Results and Conclusions

Page 20: Realistic Modeling of Animatable Faces in MPEG-4 Marco Fratarcangeli and Marco Schaerf University of Rome “La Sapienza”

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

20

Achieving Facial Animation

MPEG-4 FBAdata stream

MPEG-4 player(PC, PDA, mobile phones)

Animatable Face Model

anAtomic

Audio

FDPs

Static Model

Output

One action set for each MPEG-4

FAP

Page 21: Realistic Modeling of Animatable Faces in MPEG-4 Marco Fratarcangeli and Marco Schaerf University of Rome “La Sapienza”

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

21

Page 22: Realistic Modeling of Animatable Faces in MPEG-4 Marco Fratarcangeli and Marco Schaerf University of Rome “La Sapienza”

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

22

Conclusions - Pros

This method generates realistic facial animation encoded in a MPEG-4 FAP stream of general face meshes using only a standard PC.

It is automatic and fast. It can be used to animate talking

heads in MPEG-4 compliant applications.

Page 23: Realistic Modeling of Animatable Faces in MPEG-4 Marco Fratarcangeli and Marco Schaerf University of Rome “La Sapienza”

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

23

Conclusions - Cons

Animation becomes inaccurate if the polygonal resolution of the face mesh is not enough fine (too few springs!) and/or the mesh topology is not regular (e.g. not simmetric).

Solution: Motion Cloning

Page 24: Realistic Modeling of Animatable Faces in MPEG-4 Marco Fratarcangeli and Marco Schaerf University of Rome “La Sapienza”

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

24

Main References Y. Lee, D. Terzopoulos and K. Waters. ”Constructing

Physics-Based Facial Models of Individuals”

K. Waters. ”A Muscle Model for Animating Three-

Dimensional Facial Expressions”.

D. Baraff and A. Witkin. ”Large Steps in Cloth

Simulation”.

I.S. Pandzic and R. Forchheimer, editors. ”MPEG-4 Facial

Animation – The Standard, Implementation and

Applications”.

Page 25: Realistic Modeling of Animatable Faces in MPEG-4 Marco Fratarcangeli and Marco Schaerf University of Rome “La Sapienza”

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

25

Acknowledgements

Image Coding Group, Department of Electrical Engineering, Linköping University, Sweden

Visage Technologies AB [www.visagetechnologies.com]

Page 26: Realistic Modeling of Animatable Faces in MPEG-4 Marco Fratarcangeli and Marco Schaerf University of Rome “La Sapienza”

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

26

Page 27: Realistic Modeling of Animatable Faces in MPEG-4 Marco Fratarcangeli and Marco Schaerf University of Rome “La Sapienza”

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

27

Skull

iiinet nnF )(

skullF 0

0 iinet nFwhe

notherwise

inetF is the net force on node i

inis the nodal normal of node i

Page 28: Realistic Modeling of Animatable Faces in MPEG-4 Marco Fratarcangeli and Marco Schaerf University of Rome “La Sapienza”

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

28

Skull - Example

Page 29: Realistic Modeling of Animatable Faces in MPEG-4 Marco Fratarcangeli and Marco Schaerf University of Rome “La Sapienza”

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

29

Skull – Lower Jaw

Page 30: Realistic Modeling of Animatable Faces in MPEG-4 Marco Fratarcangeli and Marco Schaerf University of Rome “La Sapienza”

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

30

Muscle Mapping - Examples

Page 31: Realistic Modeling of Animatable Faces in MPEG-4 Marco Fratarcangeli and Marco Schaerf University of Rome “La Sapienza”

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

31

Linear Muscles

Page 32: Realistic Modeling of Animatable Faces in MPEG-4 Marco Fratarcangeli and Marco Schaerf University of Rome “La Sapienza”

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

32

Linear Muscles

Page 33: Realistic Modeling of Animatable Faces in MPEG-4 Marco Fratarcangeli and Marco Schaerf University of Rome “La Sapienza”

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

33

Sphincter Muscles

Page 34: Realistic Modeling of Animatable Faces in MPEG-4 Marco Fratarcangeli and Marco Schaerf University of Rome “La Sapienza”

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

34

Semi-Implicit Integration

txFtxF,KtxFdt

tdxD

dt

txdM musskullela

2

2

Starting from the differential system governing the net:

Equations for one mass mi:

tvdt

tdxi

i

tDvKtxFtxFdt

tdvm iielaimus

ii ,

Page 35: Realistic Modeling of Animatable Faces in MPEG-4 Marco Fratarcangeli and Marco Schaerf University of Rome “La Sapienza”

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

35

Semi-Implicit IntegrationImplicit method (Lagrangian), it is obtained:

i

ni

niela

nimusn

ini m

DvKxFxFtvv

1111 ,

11 ni

ni

ni tvxx Non linear functions

very expensive extract xin+1 for each time step

xin+1 is approximated using an Euler step (explicit method):

ni

ni

ni tvxx 1

Trade-off between stability of the implicit and speed of the explicit

Page 36: Realistic Modeling of Animatable Faces in MPEG-4 Marco Fratarcangeli and Marco Schaerf University of Rome “La Sapienza”

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

36

Adaptive Displacement

Page 37: Realistic Modeling of Animatable Faces in MPEG-4 Marco Fratarcangeli and Marco Schaerf University of Rome “La Sapienza”

09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4

37