André Gagalowicz

Projet MIRAGESINRIA - Rocquencourt - Domaine de Voluceau

78153 Le Chesnay CedexE-Mail : Andre.Gagalowicz@inria.fr

3D model-based human facemodeling

IIII--II -- INTRODUCTIONINTRODUCTION

IIII--IIII FIRST STEP : DEFORMATION OF THEFIRST STEP : DEFORMATION OF THEGENERIC MODEL WITH THE AID OF FACEGENERIC MODEL WITH THE AID OF FACEFEATURE POINTSFEATURE POINTS

IIII--IIIIII SECOND STEP: USE OF LIMBS TO FINALIZESECOND STEP: USE OF LIMBS TO FINALIZETHE 3D FACE RECONSTRUCTIONTHE 3D FACE RECONSTRUCTION

IIII--IVIV RESULTSRESULTS

No RealNo Real--timetime

ContextContext

QualityQuality (model must(model must bebe preciseprecise))

Goal: FaceGoal: Face TrackingTracking in ain a videovideo sequencesequence

Application: PostApplication: Post--productionproduction

Face pose estimation forFace pose estimation for virtualvirtual realityrealityLipsLips andand emotionemotion trackingtrackingMustMust bebe quick,quick, regularregular andand reusablereusable

RotoscopyRotoscopy

IIII--II -- INTRODUCTIONINTRODUCTION

PreviousPrevious approachesapproachesHow toHow to obtainobtain aa goodgood 3D model of face3D model of face??

3D DEVICES3D DEVICES(control on the 3D model production REQUIRED(control on the 3D model production REQUIRED))

1.1. Laser captureLaser capture devicesdevices

2.2. StructuredStructured lightlight

PrevioPrevioUUss ApproachesApproaches

How toHow to obtainobtain aa goodgood 3D model of face3D model of face??FromFrom IMAGESIMAGES

1.1. GraphicsGraphics designersdesigners

2.2. ShapeShape fromfrom shadingshading

PreviousPrevious ApproachesApproachesHow toHow to obtainobtain aa goodgood 3D model of face3D model of face??

FromFrom IMAGESIMAGES

1.1. GraphicsGraphics designersdesigners

2.2. ShapeShape fromfrom shadingshading

3.3. GenericGeneric face modificationface modification

ModelingModeling isis performedperformed inin twotwo stepssteps::

••FIRST STEP: DEFORMATION OF THE GENERICFIRST STEP: DEFORMATION OF THE GENERICMODEL WITH THE AID OF FACE FEATURE POINTSMODEL WITH THE AID OF FACE FEATURE POINTS

•• SECOND STEP: USE OF LIMBS TO FINALIZE THESECOND STEP: USE OF LIMBS TO FINALIZE THE3D FACE RECONSTRUCTION3D FACE RECONSTRUCTION

CooperativeCooperativeCalibration /Calibration /

ReconstructionReconstruction

Calibration ofthe camerasusing feature

points

3Dreconstructionof face feature

points

Deformationof the whole

3D meshusing RBF

RadialRadialBasisBasis

FunctionFunction

DataData

GenericGeneric

faceface

ModelRefinment

usinglimbs

FinalizeFinalize thethedeformationdeformation

SpecificSpecific

faceface

ResultsResults

SecondSecond stepstep

GLOBAL SCHEMEGLOBAL SCHEME

IIII--IIII --11-- ChoiceChoice ofof characteristiccharacteristic pointspoints

IIII--IIII --22-- Interactive calibration of the camerasInteractive calibration of the cameras

IIII--IIII --33-- AssigningAssigning interactivelyinteractively thethe characteristiccharacteristic points ofpoints ofthethe genericgeneric model to the face imagesmodel to the face images

IIII--IIII FIRST STEP : DEFORMATION OF THE GENERIC MODELFIRST STEP : DEFORMATION OF THE GENERIC MODELWITH THE AID OF CHARACTERISTIC POINTSWITH THE AID OF CHARACTERISTIC POINTS

IIII--IIII ––44 RealisationRealisation of the modelof the model deformationdeformation

IIII--IIII –– 11 ChoiceChoice ofof characteristiccharacteristic pointspoints

a prioria priori KnowledgeKnowledge

GenericGeneric modelmodel3434 FeatureFeature pointspoints

IIII--IIII –– 11 ChoiceChoice ofof characteristiccharacteristic pointspoints

IIII--IIII –– 22 -- Interactive Calibration of the camerasInteractive Calibration of the cameras

IIII--IIII -- 44-- RealisationRealisation of the modelof the model deformationdeformation

IIII--IIII -- 33-- AssigningAssigning interactivelyinteractively thethe characteristiccharacteristicpoints of thepoints of the genericgeneric model to the face imagesmodel to the face images

IIII FIRST STEP : DEFORMATION OF THE GENERIC MODELFIRST STEP : DEFORMATION OF THE GENERIC MODELWITH THE AID OF CHARACTERISTIC POINTSWITH THE AID OF CHARACTERISTIC POINTS

IIII--IIII –– 22 -- Interactive Calibration of the camerasInteractive Calibration of the cameras

IIII--IIII -- 11 ChoiceChoice ofof characteristiccharacteristic pointspoints

IIII--IIII -- 22 -- Interactive Calibration of the camerasInteractive Calibration of the cameras

IIII--IIII -- 33-- AssigningAssigning interactivelyinteractively thethe characteristiccharacteristicpoints of thepoints of the genericgeneric model to the face imagesmodel to the face images

IIII--IIII -- 44-- RealisationRealisation of the modelof the modeldeformationdeformation

IIII--IIII --33-- AssigningAssigning interactivelyinteractively thethe characteristiccharacteristic points of thepoints of thegenericgeneric model to the face imagesmodel to the face images

IIII--IIII -- 22 -- Interactive Calibration of the camerasInteractive Calibration of the cameras

IIII--IIII -- 33-- AssigningAssigning interactivelyinteractively thethe characteristiccharacteristicpoints of thepoints of the genericgeneric model to the face imagesmodel to the face images

IIII--IIII -- 44-- RealisationRealisation of the modelof the model deformationdeformation

IIII--IIII -- 11 ChoiceChoice ofof characteristiccharacteristic pointspoints

a)a) CooperativeCooperative Calibration/3D Reconstruction ofCalibration/3D Reconstruction of featurefeature pointspointsb) Extrapolation ofb) Extrapolation of deformationsdeformations to theto the completecomplete faceface meshmesh

a)a) CooperativeCooperative Calibration/3D Reconstruction ofCalibration/3D Reconstruction of featurefeature pointspoints

Calibration ofCalibration ofthe camerasthe camerasusingusing featurefeature

pointspoints

3D3Dreconstructionreconstruction

ofof featurefeaturepointspoints

1

24

3

51

24

3

5

6

CalibrationCalibrationCalibrationCalibration

Calibration ofCalibration of eacheach cameracamera

needsneeds::thethe intrinsicintrinsic parametersparameters (focal(focal lengthlength, pixel aspect ratio, pixel aspect ratio ……))

atat leastleast 55 3D points3D pointsthethe associatedassociated projectionsprojections

IterativeIterative::

MinimizesMinimizes the distancethe distance betweenbetween thethe estimatedestimated projection ofprojection ofeacheach 3D point and3D point and eacheach associatedassociated real projection in imagereal projection in image

1

2

4

3

5

6

3434 featurefeature pointspoints

a)a) CooperativeCooperative Calibration / 3D Reconstruction ofCalibration / 3D Reconstruction of featurefeature pointspoints

methodmethod::POSITPOSIT úû

ùêë

é10TR

3D Reconstruction3D Reconstruction3D Reconstruction

Camera 1Camera 1Camera 2Camera 2

3D Reconstruction by3D Reconstruction by stereostereo CalibrationCalibration isis notnot perfectperfect

ReconstructedReconstructed point: Middle point onpoint: Middle point onthe segmentthe segment givinggiving the minimumthe minimum

distancedistance betweenbetween thethe twotwo lineslines ofof sightsight

a)a) CooperativeCooperative Calibration / 3D Reconstruction ofCalibration / 3D Reconstruction of featurefeature pointspoints

Calibration / Reconstruction

Average error for all cameras during calibration /reconstruction

00,10,20,30,40,50,60,70,80,9

1

1 2 3 4 5 6 7 8 9 10 11

Iteration number

Err

or v

alue

11

22

33 11

2244

3311

2244

33

CaseCase wherewhere a pointa point isis seenseen onlyonly onon oneone image: Useimage: Use AdaptiveAdaptive symmetrysymmetry

555555

3D Reconstruction3D Reconstruction3D Reconstruction

a)a) CooperativeCooperative Calibration/3D Reconstruction ofCalibration/3D Reconstruction of featurefeature pointspoints

HereHere point 4point 4 isis seenseen onlyonly on theon the leftleft imageimage

PP11

Proj1(P1)

Proj1(P2)

Proj2(P1)

Camera 1

Image 1

V1R1

Camera 2

Image 2

PP22

a)a) CooperativeCooperative Calibration/3D Reconstruction ofCalibration/3D Reconstruction of featurefeature pointspoints

AdaptiveAdaptive symmetrysymmetry (cont.)(cont.)

R1

Proj1(P2)

Camera 1

Camera 2

Image 1

PP11

PP22

R2

R’2

Image 2

a)a) CooperativeCooperative Calibration/3D Reconstruction ofCalibration/3D Reconstruction of featurefeature pointspoints

AdaptiveAdaptive symmetrysymmetry (cont.)(cont.)

GivenGiven a set ofa set ofdeformationdeformation vectorsvectors

BasicBasic ideaidea::

11 ComputeCompute thethe deformationdeformation atat featurefeature points (points (donedone inin a )a )))

22 InterpolateInterpolate thethe deformationsdeformations on theon the wholewhole meshmesh

USEUSE Radial BasisRadial Basis FunctionsFunctions (RBF)(RBF)

b) Extrapolation ofb) Extrapolation of deformationsdeformations to theto the completecomplete faceface meshmesh

å=

-=n

ii DiPAP

1))(.()f( s

For nFor n deformationdeformation vectorsvectors DD1..n1..n of points Pof points P1..n1..n ,,choosechoose aa deformationdeformation functionfunction f(P) as :f(P) as :

WhereWhere ss isis thethe Radial BasisRadial Basis FunctionFunction

TypicallyTypically:: rr =)(s

, for local, for local effecteffect dependingdepending on Con C)/()(2 Crer -=s

, for, for smoothsmooth resultresult

WeWe have:have: ••a set of na set of n unknowsunknows AA1..n1..n

••nn constraintsconstraints:: ii DP =)f( For i = 1 .. nFor i = 1 .. n

b) Extrapolation ofb) Extrapolation of deformationsdeformations to theto the completecomplete faceface meshmesh

Calibration / Reconstruction - Problems

• Anti-parallel positions of the cameras

C1

I1

C2

I2

R

C1

I1

C2

I2

R

Calibration / Reconstruction - Problems

• Solution:– Angle between C1, C2

– Symmetry plane

3D reconstruction without using symmetry plane 3D reconstruction with using symmetry plane

ModelModel ResultingResulting fromfrom thethe deformationdeformation basedbased onon featurefeature points (inpoints (in GREENGREEN)) isis poorpoor!!

IdeaIdea::

deformdeform thethe currentcurrent modelmodel usingusing face silhouette (face silhouette (limbslimbs) in) in REDRED

IIII--IIIIII SECOND STEPSECOND STEP–– USE OF LIMBS TO FINALIZEUSE OF LIMBS TO FINALIZETHE 3D FACE RECONSTRUCTIONTHE 3D FACE RECONSTRUCTION

ForFor eacheach cameracamera

ComputeCompute the modelthe model limblimbandand projectproject itit in image planein image plane

IIII--IIIIII SECOND STEPSECOND STEP–– USE OF LIMBS TO FINALIZEUSE OF LIMBS TO FINALIZETHE 3D FACE RECONSTRUCTIONTHE 3D FACE RECONSTRUCTION

Automatic 3D model silhouette extraction

• Silhouette construction • Special processing ofintersections

• Enhancements:– Processing of only those edges which lie between the visible and

invisible facets– Computation of the intersections only for the currently chosen edge

• Result: 60 sec→ 1.2 sec of computation time per view

Initial vertexProjections ofvisible facets on theimage plane

ForFor eacheach cameracamera

ComputeCompute thethe limblimb andand projectproject ititin image planein image plane

ConstructConstruct by hand the realby hand the realcontour in image planecontour in image plane

Captions:Captions:

Control Point of the BControl Point of the Béézierzier curvecurve

NextNext tangent,tangent, evolvingevolving alongalong thethe curvecurve

PrecedingPreceding tangent,tangent, evolvingevolving alongalong thethe curvecurve

IIIIII SECOND STEPSECOND STEP–– USE OF LIMBS TO FINALIZEUSE OF LIMBS TO FINALIZETHE 3D FACE RECONSTRUCTIONTHE 3D FACE RECONSTRUCTION

ForFor eacheach cameracamera

ComputeCompute thethe limblimb andandprojectproject itit in image planein image plane

ConstructConstruct the realthe realcontour in image planecontour in image plane

ComputeCompute the 2Dthe 2D deformationsdeformations transformingtransforming thetheprojectedprojected modelmodel limblimb to the real contourto the real contour

IIII--IIIIII SECOND STEPSECOND STEP–– USE OF LIMBS TO FINALIZEUSE OF LIMBS TO FINALIZETHE 3D FACE RECONSTRUCTIONTHE 3D FACE RECONSTRUCTION

22

2

11

1

77

7

44

4

88

8

55

5

99

9

66

6

33

3

Use local maxima (Use local maxima (improvesimproves computation)computation)andand propagatepropagate thethe deformationsdeformations fromfrom thethemaximamaxima alongalong thethe curvecurve usingusing 1D RBF1D RBF

PP’’88

RR’’88

PP’’99RR’’99

2D2D DeformationDeformation vectorsvectors

How toHow to obtainobtain the 2Dthe 2D DeformationsDeformations ??

InferInfer the 3Dthe 3D deformationsdeformations fromfrom the 2Dthe 2D onesones

ComputeCompute the 2Dthe 2D deformationsdeformations transformingtransforming thetheprojectedprojected limblimb intointo the real contourthe real contour

ForFor eacheach cameracamera

ComputeCompute thethe limblimb andandprojectproject itit in image planein image plane

ConstructConstruct the realthe realcontour in image planecontour in image plane

IIIIII SECOND STEPSECOND STEP–– USE OF LIMBS TO FINALIZEUSE OF LIMBS TO FINALIZETHE 3D FACE RECONSTRUCTIONTHE 3D FACE RECONSTRUCTION

2D2D àà 3D3D DeformationsDeformations2D2D àà 3D3D DeformationsDeformations

InferInfer the 3Dthe 3D deformationsdeformations fromfrom the 2Dthe 2D onesones

CalculateCalculate the 2Dthe 2D deformationsdeformations transformingtransforming thetheprojectedprojected limblimb to the real contourto the real contour

ForFor eacheach cameracamera

CalculateCalculate thethe limblimb andandprojectproject itit in image planein image plane

DefineDefine the realthe realcontour in image planecontour in image plane

AddAdd thisthis set of 3Dset of 3D deformationsdeformations incrementallyincrementally totothethe previousprevious set ofset of featurefeature pointspoints

IIII--IIIIII SECOND STEPSECOND STEP–– USE OF LIMBS TO FINALIZEUSE OF LIMBS TO FINALIZETHE 3D FACE RECONSTRUCTIONTHE 3D FACE RECONSTRUCTION

IIII--IVIV RESULTSRESULTS

IIII--IVIV RESULTSRESULTS

LambertianLambertiansynthesissynthesis

OriginalOriginal

IIII--IVIV RESULTSRESULTS

VALIDATIONVALIDATION

TheThe precisionprecision of the 3D reconstruction of facesof the 3D reconstruction of faces fromfrom 44imagesimages whenwhen thethe geometrygeometry of facesof faces waswas knownknown beforehandbeforehandwaswas 2%2%

((similarsimilar to theto the precisionprecision of aof a conventionalconventional laser system)laser system)

TIMETIME

The timeThe time requiredrequired toto constructconstruct a 3D modela 3D model fromfrom scratchscratch

lieslies betweenbetween 20 min to20 min to ½½ hourhour!!