Static Facial Expression in Tough Conditions: Data, Evaluation Protocol and Benchmark
http://cs.anu.edu.au/fewhttp://cs.anu.edu.au/few
Abhinav Dhall
Roland Goecke
Simon Lucey
Tom Gedeon
1
Content
� Why?
� Database creation
� Comparison
� Protocols� Protocols
2
Motivation
� Facial Expressions are the facial changes in response to a persons Internal
Emotional States, Intentions, or Social Communications. Automatic Facial
Expressions Analysis is the science of making computer understand “empathy”
– the art of understanding others emotions.
� Real world application• Affective Computing
• Human Computer Interaction• Human Computer Interaction
• Intelligent Environments
• Emotion and Paralinguistic Communication
• Lie Detection
• Medical Conditions
� Depression, Autism, Pain Analysis
3
Real world images, Source Flickr CC license
Motivation
� Facial Expressions are the facial changes in response to a persons Internal
Emotional States, Intentions, or Social Communications. Automatic Facial
Expressions Analysis is the science of making computer understand “empathy”
– the art of understanding others emotions.
� Real world application• Affective Computing
• Human Computer Interaction• Human Computer Interaction
• Intelligent Environments
• Emotion and Paralinguistic Communication
• Lie Detection
• Medical Conditions
� Depression, Autism, Pain Analysis
� Can be categorized into• Video based analysis: Movies, YouTube!
• Image based analysis: Flickr, Facebook
� Geometric/Appearance
4
Real world images, Source Flickr CC license
Motivation
� Computer vision is a data driven research!
� A lot of interesting datasets available:
• Belfast [1]
• CMU PIE [2]
• FEEDTUM [3]
• JAFFE [4]
• MultiPIE [5]
• MMI [6]
• CK+ [7]
• Semaine [8] ……
5
Images from JAFFE and MultiPIE
Motivation
� Computer vision is a data driven research!
� A lot of interesting datasets available:
• Belfast [1]
• CMU PIE [2]
• FEEDTUM [3]
• JAFFE [4]
• MultiPIE [5]
• MMI [6]
• CK+ [7]
• Semaine [8] ……
But majority of them are recorded in “lab” control led environment!
6
Images from JAFFE and MultiPIE
Motivation - 2
� Labeled Faces In The Wild (Huang et al.)
7
Motivation - 2
� Labeled Faces In The Wild (Huang et al.)
� Hollywood action database (Laptev et al.)
8
Motivation - 2
� Labeled Faces In The Wild (Huang et al.)
� Hollywood action database (Laptev et al.)
� There is a void for similar databases in the facial expression research community.
9
Motivation - 2
� Labeled Faces In The Wild (Huang et al.)
� Hollywood action database (Laptev et al.)
� There is a void for similar databases in the facial expression research community.
� We present two databases extracted from movies:
• Acted Facial Expressions In The Wild (AFEW) - Temporal
• Static Facial Expressions In The Wild (SFEW) - Static
� Use Subtitle for Deaf and Hearing impaired (SDH) subtitles
10
Database Creation
11
56631 subtitles -> 7025 expressive subtitles -> 957 clips
AFEW creation steps
Database Statistics
� Expression - Theme expression of the scene
� StartTime - This denotes the start timestamp of the clip in the movie DVD
� Length - Duration of the clip in milliseconds
� Person - This contains various attributes describing the actor / character in the scene
• Pose - This denotes the head pose based on the labeler’s observation
• AgeOfCharacter – Age of character in the movie
12
• AgeOfCharacter – Age of character in the movie
• NameOfActor - Real name of the actor
• AgeOfActor – Age of the actor extracted the information from www.imdb.com
• ExpressionOfPerson – Expression category of the character
• Gender - Gender of the actor
� Number of Clips – 957
� Total number of samples – 1259
� Expression category – Angry, Disgust, Fear, Happy, Neutral, Sad and Surprise
� Length – 300-5400 ms
SFEW
� Static Facial Expressions In The Wild
� Frames extracted from AFEW
� 95 Subjects
� Expression: Angry, Disgust, Fear, Happy, Neutral, Sad and Surprise
� PPI protocol part of BEFIT challenge
13
Samples from SFEW
Comparison with other databases
� Comparing
• SFEW with JAFFE and MultiPIE databases
• AFEW with CK+ database
DB Const.
Method
Environ. Age
Range
Illum. Occlus. Search. Subject
Details
Multiple
Subject
14
Method Range Details Subject
AFEW Assisted Real 1-70 CTN Yes Yes Yes Yes
CK+ Manual Lab 18-50 C No No No No
SFEW Manual Real 1-70 CTN Yes No No No
JAFFE Manual Lab ? C No No No No
M-PIE Manual Lab 27.9
(avg)
C No No No No
Database comparison. Here CTN – Close To Natural, C - Controlled
Comparison with other databases
• Face detect
• Descriptors
• Local Binary Pattern (LBP) [8]
• Pyramid of Histogram Of Gradient (PHOG) [9]
• Local Phase Quantization (LPQ) [8]
• LBP-Three Orthogonal Planes (LBP-TOP) [8]
• Support Vector Machine (SVM)
15
Classification accuracy comparison for LBP, PHOG, LPQ and LBP-TOP for CK+ and AFEW
• Support Vector Machine (SVM)
Comparison with other databases
16
Classification accuracy comparison for PHOG and LPQ SFEW, MultiPIE and JAFFE
Protocols
Protocol SFEW AFEW
Training & Testing sets have:
Strictly Person Specific (SPS) same single subject same single subject
Partial Person Independent
(PPI)
seen & unseen subjects seen & unseen subjects
(PPI)
Strictly Person Independent
(SPI)
unseen subjects (SFEW
challenge @ BEFIT challenge)
unseen subjects
Cross Database Set Test sets only Test sets only
17
Different protocol for SFEW and AFEW
The results should be reported as correct classification accuracy, precision and recall .
Conclusion
� We have proposed two new “in the wild” datasets
� A semi-automatic approach based on subtitle parsing based
recommender is used for database collection
� The database has been annotated with dense attributes about the
subjects
18
subjects
� The only labeled multi people expression database
� Multiple protocols for experiments based on subject dependency
� The datasets and related information will be updated at
http://cs.anu.edu.au/few
References
1) E. Douglas-Cowie, R. Cowie, and M. Schröder. A new emotion database: Considerations, sources and scope.
In Speech Comm., 2000.
2) Terence Sim, Simon Baker, and Maan Bsat. The CMU Pose, Illumination, and Expression Database. IEEE
Transactions on Pattern Analysis and Machine Intelligence, 25(12):1615–1618, December 2003.
3) Frank Wallhoff. Facial expressions and emotion database, 2006. http://www.mmk.ei.tum.de/
waf/fgnet/feedtum.html.
4) M. J. Lyons, S. Akamatsu, M. Kamachi, and J. Gyoba. Coding facial expressions with gabor wavelets. In
Proceedings of the IEEE International Conference on Automatic Face Gesture Recognition and Workshops,
FG’98, 1998
5) Ralph Gross, Iain Matthews, Jeffrey F. Cohn, Takeo Kanade, and Simon Baker. Multi-PIE. In Proceedings of 5) Ralph Gross, Iain Matthews, Jeffrey F. Cohn, Takeo Kanade, and Simon Baker. Multi-PIE. In Proceedings of
the Eighth IEEE International Conference on Automatic Face and Gesture Recognition, FG’2008, pages 1–8,
2008.
6) Maja Pantic, Michel Franc¸ois Valstar, Ron Rademaker, and Ludo Maat. Web-based database for facial
expression analysis. In Proceedings of the IEEE International Conference on Multimedia and Expo, ICME’05,
pages 317–321, 2005.
7) G. McKeown, M. F. Valstar, R. Cowie, and M. Pantic, “The semaine corpus of emotionally coloured character
interactions,” In Proceedings of the IEEE International Conference on Multimedia and Expo, ICME’10.
8) D. Huang, C. Shan, M. Ardabilian, Y. Wang, and L. Chen, “Local binary patterns and its application to facial
image analysis: A survey,” IEEE Transactions on Systems, Man, and Cybernetics, Part C: applications and
Reviews, pp. 1 –17, 2011.
9) A. Bosch, A. Zisserman, and X. Munoz, “Representing Shape with a Spatial Pyramid Kernel,” in Proceedings of
the ACM International Conference on Image and Video Retrieval, ser. CIVR ’07, 2007, pp. 401–408.
19
Thank You
20