Upload
hera
View
42
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Recognition, Analysis and Synthesis of Gesture Expressivity. George Caridakis IVML-ICCS. Overview. Corpus Image processing module Gesture Recognition Expressivity Analysis Expressivity Synthesis Applications. Overview. Corpus mint-IVML. 7 subjects 7 gesture classes - PowerPoint PPT Presentation
Citation preview
Recognition, Analysis and Synthesis of Gesture
Expressivity
George CaridakisIVML-ICCS
humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis
Overview Corpus Image processing module Gesture Recognition Expressivity Analysis Expressivity Synthesis Applications
humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis
Overview
Video sequences varying in activation-evaluation
Head and hand tracking
Hand Coordinates/Arm configuration
Synthetic expressive gesture reconstruction
Gesture expressivity
features extractionHMM classifier
humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis
Corpus mint-IVML
• 7 subjects
• 7 gesture classes
• 20 gesture variations (3 quadrants)
• 20’ minutes – 30000 frames
humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis
Corpus EmoTV
humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis
Corpus GEMEP (on going…)
humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis
Head detection Detect candidate facial areas Validate using skin probability Conclude on number of persons
humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis
Hand Detection•Skin probability
•Thresholding & Morphology Operations
•Distance Transform
•Frame difference
humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis
Tracking Scoring system based on:
• Skin region size• Distance wrt the previous position • Optical flow alignment • Spatial constraints
Thresholding scores Periodical re-initialization
humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis
Head & Hand Tracking
humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis
HMM parameters for gestures States are head and hands coordinates
• XL-XR XH-XR XH-XL YL-YR YH-YR YH-YL 6 output states Bakis left-to-right models Continuous output distribution
• 3 Gaussian mixtures Arbitrary training initial estimation of
transition probabilities
humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis
Recognition via HMM (Why HMMs?) Stochastic models fit the nature of the
gestures Fast convergence due to effective training
algorithms Sufficient modeling of the temporal aspect
of gestures Continuous HMMs suitable for gesture-
level classification
humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis
HMM overview
λ2=(A 2, B 2, π )
λ1=(A 1, B 1, π )
λM =(A M , B M , π )
Select Maxim un
Feature VectorSequence υ*= argm ax[P (O |λ υ)]
υ
P(O | λ1 )
P(O | λ2 )
P(O | λ M )
.
.
.
.
.
.
humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis
Recognition via HMM
0 10 20 30 40 50 60 70 80 90-300
-250
-200
-150
-100
-50
0
50Xf - Xlh
Xf - XrhXlh - Xrh
0 10 20 30 40 50 60 70 80 90-400
-300
-200
-100
0
100
200Yf - Ylh
Yf - Yrh
Ylh - Yrh
humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis
Results
Gesture Attention Clapping ExplainOne
hand Oh my
god Wave Go away
Unclassified
Attention 90.9 0 0 0 0 0 9.1 0
Clapping 0 75 25 0 0 0 0 0
Explain 0 0 60 0 0 0 0 40
One hand 0 0 0 83.3 0 16.7 0 0
Oh my god 0 0 8.3 0 83.3 0 08.3
Wave 25 0 0 0 0 62.5 12.5 0Go away 25 0 0 0 0 0 75 0
humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis
Expressivity features analysis Overall activation Spatial extent Temporal Fluidity Power/Energy Repetitivity
humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis
Overall activation Considered as the quantity of movement
during a conversational turn Computed as the sum of the motion
vectors’ norm
n
i
ilirOA0
|)(||)(|
humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis
Spatial extent Modeled by expanding or condensing
the entire space in front of the agent that is used for gesturing
Calculated as the maximum Euclidean distance of the position of the two hands
The average spatial extent is also calculated for normalization reasons
max(| ( ( ) ( )) |)SE d r i l i
humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis
Temporal The temporal parameter of the gesture
determines the speed of the arm movement of a gesture’s meaning carrying stroke phase and also signifies the duration of movements (e.g., quick versus sustained actions)
OATEduration
humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis
Fluidity Differentiates smooth/graceful from
sudden/jerky ones. This concept seeks to capture the continuity between movements, the arms’ trajectory paths as well as the acceleration and deceleration of the limbs
To extract this feature from the input image sequences we calculate the sum of the variance of the norms of the motion vectors
0
var(| ( ) |) var(| ( ) |)n
i
FL r i l i
humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis
Power/Energy The power is actually identical with the
first derivative of the motion vectors calculated in the first steps
| ( ) | | ( ) |d r x d l xPOdx dx
humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis
Results of expressivity analysis
EF variation Overall Activation
Spatial Extent
Temporal Fluidity Power/Energy
humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis
Expressive synthesis A system that
mimics user’s behaviour through the analysis of facial and gesture signals and expressivity
humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis
Synthesis Greta Platform BAP calculation
• Plane assumption• Inverse kinematics• Manual adaptation
Expressivity features variations implemented in Greta’s BAP interpolation
humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis
Synthesis Results