View
9
Download
0
Category
Preview:
Citation preview
Social Signal ProcessingUnderstanding Nonverbal Behaviour
in Social Interactions
Alessandro VinciarelliUniversity of Glasgow - Idiap Research Institute
Institute of Neuroscience and Psychologyhttp://www.dcs.gla.ac.uk/vincia
vincia@dcs.gla.ac.uk
Outline
•Social Signal Processing
•The Conflict Example
•Conclusions
Outline
•Social Signal Processing
•The Conflict Example
•Conclusions
Nonverbal Communication (I)
3
Vinciarelli, Pantic and Bourlard, “Social Signal Processing: Survey of an Emerging Domain”, Journal of Image and Vision Computing, 27(12):1743-1759, 2009
Nonverbal Communication (I)
3
Height
ForwardPosture Forward
Posture
GesturesInterpersonalDistance
NonverbalBehavioral
Cues
VocalBehavior
MutualGaze
Vinciarelli, Pantic and Bourlard, “Social Signal Processing: Survey of an Emerging Domain”, Journal of Image and Vision Computing, 27(12):1743-1759, 2009
Nonverbal Communication (I)
3
SocialSignals
Height
ForwardPosture Forward
Posture
GesturesInterpersonalDistance
NonverbalBehavioral
Cues
VocalBehavior
MutualGaze
Vinciarelli, Pantic and Bourlard, “Social Signal Processing: Survey of an Emerging Domain”, Journal of Image and Vision Computing, 27(12):1743-1759, 2009
Nonverbal Communication (II)
5
Richmond and McCroskey, “Nonverbal Behaviors in Interpersonal Relations”, Allyn and Bacon, 1995
Nonverbal Communication (II)
5
AttractivenessClothes, etc.
ExpressionsGaze, etc.
Vocalisations,Prosody, etc.
Self-touching,Orientation
Distances, Seating, etc.
Cues
Richmond and McCroskey, “Nonverbal Behaviors in Interpersonal Relations”, Allyn and Bacon, 1995
Nonverbal Communication (II)
5
AttractivenessClothes, etc.
ExpressionsGaze, etc.
Vocalisations,Prosody, etc.
Self-touching,Orientation
Distances, Seating, etc.
CuesPhysical
Appearance
FaceBehavior
VoiceBehavior
Gestures andPostures
Space andEnvironment
Codes
Richmond and McCroskey, “Nonverbal Behaviors in Interpersonal Relations”, Allyn and Bacon, 1995
Nonverbal Communication (II)
5
AttractivenessClothes, etc.
ExpressionsGaze, etc.
Vocalisations,Prosody, etc.
Self-touching,Orientation
Distances, Seating, etc.
CuesPhysical
Appearance
FaceBehavior
VoiceBehavior
Gestures andPostures
Space andEnvironment
Codes
FormingImpressions
ExpressionEmotion
RelationalMessages
DeceptionManagement
Power Persuasion
Functions
Richmond and McCroskey, “Nonverbal Behaviors in Interpersonal Relations”, Allyn and Bacon, 1995
Nonverbal Communication (II)
5
AttractivenessClothes, etc.
ExpressionsGaze, etc.
Vocalisations,Prosody, etc.
Self-touching,Orientation
Distances, Seating, etc.
CuesPhysical
Appearance
FaceBehavior
VoiceBehavior
Gestures andPostures
Space andEnvironment
Codes
FormingImpressions
ExpressionEmotion
RelationalMessages
DeceptionManagement
Power Persuasion
Functions
Richmond and McCroskey, “Nonverbal Behaviors in Interpersonal Relations”, Allyn and Bacon, 1995
SSP: Synthesis
6
AttractivenessClothes, etc.
ExpressionsGaze, etc.
Vocalisations,Prosody, etc.
Self-touching,Orientation
Distances, Seating, etc.
CuesPhysical
Appearance
FaceBehavior
VoiceBehavior
Gestures andPostures
Space andEnvironment
Codes
FormingImpressions
ExpressionEmotion
RelationalMessages
DeceptionManagement
Power Persuasion
Functions
Vinciarelli, Pantic, Heylen, Pelachaud, Poggi, D’Errico, Schroeder, “Bridiging the Gap Between Social Animal and Unsocial Machine: A Survey of SSP”, IEEE
Transactions on Affective Computing, 3(1):69-87,2012
SSP: Analysis
7
AttractivenessClothes, etc.
ExpressionsGaze, etc.
Vocalisations,Prosody, etc.
Self-touching,Orientation
Distances, Seating, etc.
CuesPhysical
Appearance
FaceBehavior
VoiceBehavior
Gestures andPostures
Space andEnvironment
Codes
FormingImpressions
ExpressionEmotion
RelationalMessages
DeceptionManagement
Power Persuasion
Functions
Vinciarelli, Pantic, Heylen, Pelachaud, Poggi, D’Errico, Schroeder, “Bridiging the Gap Between Social Animal and Unsocial Machine: A Survey of SSP”, IEEE
Transactions on Affective Computing, 3(1):69-87,2012
Outline
•Social Signal Processing
•The Conflict Example
•Conclusions
9
Conflict
“[Conflict is a] mode of interaction [where]the attainment of the goal by one partyprecludes its attainment by the others.”
Judd, “Cognitive Effects of Attitude Conflict Resolution”, Journal of Conflict Resolution, 22(3):483-498, 1978.
10
Conflict
“[Conflict is a] mode of interaction [where]the attainment of the goal by one partyprecludes its attainment by the others.”
Judd, “Cognitive Effects of Attitude Conflict Resolution”, Journal of Conflict Resolution, 22(3):483-498, 1978.
11
Conflict
“[Conflict is a] mode of interaction [where]the attainment of the goal by one partyprecludes its attainment by the others.”
Judd, “Cognitive Effects of Attitude Conflict Resolution”, Journal of Conflict Resolution, 22(3):483-498, 1978.
Political Debates
Vinciarelli, Dielmann, Favre, Salamin, “Canal9: a database of political debates for analysis of social interactions ”, Proc. of Social Signal Processing Workshop, 2009
12
Political Debates
Vinciarelli, Dielmann, Favre, Salamin, “Canal9: a database of political debates for analysis of social interactions ”, Proc. of Social Signal Processing Workshop, 2009
12
SSPNet Conflict Corpus
Vinciarelli et al., “Collecting Data for Socially Intelligent Surveillance and Monitoring Approaches: The Case of Conflict in Competitive Conversations”, Proc. of IEEE Intl. Symposium on Communications, Control and Signal Processing, pp. 1-4, 2012.
13
Source Canal9Number of Clips 1430Clip Length 30 sec.Total Length 11 h 55 mSubjects 135Subjects per Clip At least 2Assessors 10/clip (MTurk)Questionnaire Items 15Total Items 214,500
Measuring Conflict Perception
14
The atmosphere is relaxedPeople wait for their turn before speakingOne or more people talk fastOne or more people fidgetPeople argueOne or more people raise their voiceOne or more people shake their heads and nodPeople show mutual respectPeople interrupt one anotherOne or more people gesture with their handsOne or more people are aggressiveThe ambience is tenseOne or more people compete to talkPeople are actively engagedOne or more people frown
Measuring Conflict Perception
15
The atmosphere is relaxedPeople wait for their turn before speakingOne or more people talk fastOne or more people fidgetPeople argueOne or more people raise their voiceOne or more people shake their heads and nodPeople show mutual respectPeople interrupt one anotherOne or more people gesture with their handsOne or more people are aggressiveThe ambience is tenseOne or more people compete to talkPeople are actively engagedOne or more people frown
Measuring Conflict Perception
15
The atmosphere is relaxedPeople wait for their turn before speakingOne or more people talk fastOne or more people fidgetPeople argueOne or more people raise their voiceOne or more people shake their heads and nodPeople show mutual respectPeople interrupt one anotherOne or more people gesture with their handsOne or more people are aggressiveThe ambience is tenseOne or more people compete to talkPeople are actively engagedOne or more people frown
I
Measuring Conflict Perception
15
The atmosphere is relaxedPeople wait for their turn before speakingOne or more people talk fastOne or more people fidgetPeople argueOne or more people raise their voiceOne or more people shake their heads and nodPeople show mutual respectPeople interrupt one anotherOne or more people gesture with their handsOne or more people are aggressiveThe ambience is tenseOne or more people compete to talkPeople are actively engagedOne or more people frown
I
P
16
Inference vs Physical
-10
-7.5
-5
-2.5
0
2.5
5
7.5
10
-16 -11.2 -6.4 -1.6 3.2 8
Infe
renc
e Sc
ore
Physical Score
r=0.95
Vinciarelli et al., “Collecting Data for Socially Intelligent Surveillance and Monitoring Approaches: The Case of Conflict in Competitive Conversations”, Proc. of IEEE Intl. Symposium on Communications, Control and Signal Processing, pp. 1-4, 2012.
17
Examples (Low)
Vinciarelli et al., “Collecting Data for Socially Intelligent Surveillance and Monitoring Approaches: The Case of Conflict in Competitive Conversations”, Proc. of IEEE Intl. Symposium on Communications, Control and Signal Processing, pp. 1-4, 2012.
17
Examples (Low)
Vinciarelli et al., “Collecting Data for Socially Intelligent Surveillance and Monitoring Approaches: The Case of Conflict in Competitive Conversations”, Proc. of IEEE Intl. Symposium on Communications, Control and Signal Processing, pp. 1-4, 2012.
18
Examples (Medium)
Vinciarelli et al., “Collecting Data for Socially Intelligent Surveillance and Monitoring Approaches: The Case of Conflict in Competitive Conversations”, Proc. of IEEE Intl. Symposium on Communications, Control and Signal Processing, pp. 1-4, 2012.
18
Examples (Medium)
Vinciarelli et al., “Collecting Data for Socially Intelligent Surveillance and Monitoring Approaches: The Case of Conflict in Competitive Conversations”, Proc. of IEEE Intl. Symposium on Communications, Control and Signal Processing, pp. 1-4, 2012.
19
Examples (High)
Vinciarelli et al., “Collecting Data for Socially Intelligent Surveillance and Monitoring Approaches: The Case of Conflict in Competitive Conversations”, Proc. of IEEE Intl. Symposium on Communications, Control and Signal Processing, pp. 1-4, 2012.
19
Examples (High)
Vinciarelli et al., “Collecting Data for Socially Intelligent Surveillance and Monitoring Approaches: The Case of Conflict in Competitive Conversations”, Proc. of IEEE Intl. Symposium on Communications, Control and Signal Processing, pp. 1-4, 2012.
Speech Features
20
Speech Features
20
t
Speech Features
20
t
t
Speech Features
20
t
t
t
Speech Features
20
t
t
t
t
Speech Features
20
t
t
t
t
t
21
Results
Kim, Filippone, Valente, Vinciarelli, “Predicting the Conflict Level in Television Political Debates: an Approach Based on Crowdsourcing, Nonverbal Communication
and Gaussian Processes”, Proc. of ACM Intl. Conf. on Multimedia, 793-796, 2012.
0.7
0.73
0.76
0.79
0.82
BLR GPR GPR ARD SVR LIN SVR RBF
Manual Automatic Auto w.o.s
22
ARD Results (Automatic w.o.s.)C
oeffi
cien
t Val
ue
Features
Clip
Pitc
h
Clip
Inte
nsity
Std.
Minimum
Min
imum
Turn
Std
Inte
nsity
Min
imum
Turn
Mea
n Pi
tch
Turn
Std
Pitc
h
Turn
Mea
n In
tens
ity
Ove
rlap
ping
Inte
nsity
Ove
rlap
ping
Dur
atio
n
Spea
king
Dur
atio
n
Minimum
TotalM
inim
um
Minimum Q - 99%
Max
imum
Outline
•Social Signal Processing
•The Conflict Example
•Conclusions
24
The SSPNet Portal
More information available at:
http://www.sspnet.eu
•Around 250 hours of annotated material•More than 150 Presentation Recordings•More than 20 software packages
A.Vinciarelli, M.Pantic, “A Web Portal for Social Signal Processing”, IEEE Signal Processing Magazine, 27(4):142-144, 2010
25
Conclusions
25
Conclusions
• Nonverbal communication is a physical, machine detectable evidence of social and psychological phenomena
25
Conclusions
• Nonverbal communication is a physical, machine detectable evidence of social and psychological phenomena
• Interdisciplinary approaches including both human and computing sciences can perform better
25
Conclusions
• Nonverbal communication is a physical, machine detectable evidence of social and psychological phenomena
• Real world applications are the next frontier in terms of data, problems and methodological issues
• Interdisciplinary approaches including both human and computing sciences can perform better
26
Thank you!
Many thanks to: •Gelareh Mohammadi (Idiap Research Institute / EPFL)•Maurizio Filippone (University of Glasgow)•Antonio Origlia (University of Napoli Federico II)•Fabio Valente (idiap Research Institute)•Samuel Kim (Yonsei University)•Bjoern Schuller (TU Munich)
Recommended