View
215
Download
1
Category
Tags:
Preview:
Citation preview
3D Models for Face Image and Video Processing Gbor SzirtesELTE Dept of Information SystemsNeural Information Processing GroupLrincz-lab
SSIP 2002 Budapest, Hungary
ContentShort IntroductionMotivationsPile of concepts (framework?)Future applicationsOngoing projects and immature ideasWhat next?
SSIP 2002 Budapest, Hungary
A few words about our groupSince 1999Within the Information Systems Department, ELTE5 PhD students and ~20 grad studentsMainly biologically motivated projectsRL, ICA, machine learning, facial expressions, dynamical systems, image processing
SSIP 2002 Budapest, Hungary
MotivationsBeing in quest of the Holy Grail: intelligenceOne working example: our brainEvolutionary concepts, need for adaptationPerception and Action
SSIP 2002 Budapest, Hungary
Conceptual framework I.SYSTEMENVIRONMENTNoisy, stochastic, evolvingPerceptionAction?
SSIP 2002 Budapest, Hungary
Conceptual framework II.Central hypothesis
Internal representation (encoded signals from the environment and the systems state)Reconstruction
SSIP 2002 Budapest, Hungary
Perception Active Not simply feed-forward Feed-back modulated and controlled Modular Component based Adaptive, plastic
SSIP 2002 Budapest, Hungary
Perception II.Active: it is not a passive signal detection process. We need to `foresee` and anticipate the expected changes (prediction).Influenced by higher order modulation (e.g. FOA, focus of attention, conscious and unconscious perception)
SSIP 2002 Budapest, Hungary
Perception III.Several stages of processingNot purely hierarchical (feed-back)Distributed, parallel ways, strong interplayModularity
SSIP 2002 Budapest, Hungary
Perception IV.Components: meaningful (?) building blocks
SSIP 2002 Budapest, Hungary
Perception V.This is what we have seen before?
SSIP 2002 Budapest, Hungary
PartsDrawings of 4 year old healthy children
SSIP 2002 Budapest, Hungary
and the wholeDrawing of a 3 and a half year old child with autism
SSIP 2002 Budapest, Hungary
Beyond the theoryRecognition of faces and facial expressionsTwofold goals: Understand perception Help develop applications forHuman-Computer InteractionPsychiatric analysis and treatment
SSIP 2002 Budapest, Hungary
DatabaseIn collaboration with the Psychiatric Clinic of SOTE (Simon-lab)
SSIP 2002 Budapest, Hungary
A few examples of segmented imagesHappinessDisgust
SSIP 2002 Budapest, Hungary
The architectureM1M2M3M*M**RL containerFACESACTION?
SSIP 2002 Budapest, Hungary
Modules for recognition of facesFinding heads: Skin detectionTracking: particle filteringSegmentation3D model based transformationIdentification, recognition or analysis(back-transformation)
SSIP 2002 Budapest, Hungary
Module 1Face location (fitting)Many heuristics are possibleOne particular choiceskin-detector
SSIP 2002 Budapest, Hungary
Skin detectorrgbSkin color cluster learned by MLP
SSIP 2002 Budapest, Hungary
Module 2+Particle filteringCONDENSATION (Conditional Density Propagation )(Isard and Blake, 1998)
Segmentation Tracking
SSIP 2002 Budapest, Hungary
SegmentationImage basedFeature basedproceduresTwo approaches: approximating contours with splines or snakes (too many degrees of freedom) Template basedA simple template
SSIP 2002 Budapest, Hungary
Segmentation II.More sophisticated manually tuned templateArbitrary spine directions(with positive-negative weights)
SSIP 2002 Budapest, Hungary
Segmentation III.Many concurrent candidates
SSIP 2002 Budapest, Hungary
Segmentation IV.Head-shoulder template for better fitting
SSIP 2002 Budapest, Hungary
Particle filtering in action!Initialization made by hand
SSIP 2002 Budapest, Hungary
Well, there is no perfect methodSometimes even the best choice is far from the face to be tracked
SSIP 2002 Budapest, Hungary
Tracking of fast motion against a cluttered backgroundFrom http://www.robots.ox.ac.uk/~misard/condensation.html
SSIP 2002 Budapest, Hungary
CONDENSATIONKeywords:general,multi-modaldensities,sampling,Discrete-continuousMarkovian
SSIP 2002 Budapest, Hungary
Module 3 (off the stream)Facial expression (display) recognition
SSIP 2002 Budapest, Hungary
Facial expression recognitionHMM on segmented image sequencessurpriseReconstruction errorHMM winner: surpriseHMM emission
SSIP 2002 Budapest, Hungary
Module 43D face modelExtension of the CANDIDE (Rydfalk,1987) modelCompatible with FACS(Ekman and Friesen, 1977)Candide 3 (developed for MPEG4 standard)
SSIP 2002 Budapest, Hungary
How to use the model?Target (synthetic) faceSearching
SSIP 2002 Budapest, Hungary
Such a big space!Reconstruction error based optimization problemToo many local minimaGlobal optimum finding procedure: STAGE (Boyan, 1998)
SSIP 2002 Budapest, Hungary
STAGE Algorithm for finding the global optimum Function approximator learns an evaluation function that predicts the outcome of a local search Experience: it is able to explore the global structureLet us find the minimum of F(x)=(|x|-10)cos(2x)
SSIP 2002 Budapest, Hungary
STAGE IIIt can be combined with any local search method (hillclimbing,WALKSAT,)It works on both the objective and the evaluation function at two stagesSmart restart by a better predictionReal-valued (compared to GA)Easy to implement
SSIP 2002 Budapest, Hungary
SSIP 2002 Budapest, Hungary
What we have got so far?A few working modulesWorking RL architecturesWorking combination in an other problem domain: Internet searchand research is focused on how to link all of our concepts.
SSIP 2002 Budapest, Hungary
What next?Many avenuesOngoing projects with psychiatrists: trajectory analysis with cliplets, transient expressions, depression quantificationDistance learningHuman Computer InteractionVirtual reality
SSIP 2002 Budapest, Hungary
Infos about our research activity
http://people.inf.elte.hu/lorincz/
SSIP 2002 Budapest, Hungary
Thanks for your attention (and patience)!
SSIP 2002 Budapest, Hungary
Recommended