Laurence DEVILLERS & Jean-Claude MARTIN

LIMSI-CNRS

FP6 IST HUMAINE Network of Excellence / Association (http://emotion-research.net)W3C Emotion Incubator Group

.

Definition (Scherer 2000) An hypothetical construct

denoting a process of an organism’s reaction to significant events

Cognitive theories Focus on emotion as an

cognitive evaluation of events

Sequential evaluation of appraisal variables

Event

Coping potential?Which kind of

ressources he has to deal with the situation?

Compatibility with external and internal standards?Is the event

compatible with norms?

Conduciveness to goals?

Is this situation conductiveness goal ?

Global pleasantness?Is this situation

pleasant or not ?

Novelty?Is the event new ?

emotion

Example of a clip from French TV news (EMOTV): someone saying that « I thank you for saying that I was innocent » => a mixture of negative and positive emotions

Naturalistic clips often feature several events eliciting the emotional behavior:

Examples of events:

-event #1: the person that is videotaped has been accused although she is innocent - event #2: she was recognized innocent - event #3: she is interviewed

In this talk, « emotion » stands for « affective state » :emotions, moods, attitudes, …

Gap between emotions observed in artificial data (acted) and those observed with « real-life spontaneaous data in SITU » (Batliner, 2000)

Previous works (Ekman, Scherer, Campbell, Cowie, Devillers, Martin) have shown that there are many complex emotion mixtures in real-life audio or audio-visual data

Difference is mainly due to the context eliciting the emotion: the events that are at the origin of the emotions of a person

Existing work Scherer ‘appraisal variables’ are generally

studied in emotion recall experiments and have been used for predicting emotions from events.Most studies consider acted data collected in-lab

Our research questions How to annotate events in real life data? Can we apply the appraisal model to perception

of more general affective states from real life data?

This study explores how to code emotional events

Are we able to code the events that triggered spontaneous expressions of real-life affective states in audiovisual data?

What are the relevant dimensions of events? Are we able to infer them from short video

clips?

We already used appraisal dimensions in emotion perception investigations (Devillers et al., 2006): Method: subjects annotate their perception of the videotaped

person’s appraisal (28 clips (Belfast Naturalistic Data + French EmoTV), 5 coders, 16 appraisals dimensions)

Results: Several dimensions can be reliably annotated: conduciveness to goals, pleasantness , relation to expectations , controllability-conseqevt

Main difficulty: relations between simulatenous appraisals of multiple events and complex emotions mixtures

Complex Emotion mixtures

Event 1

Controllability consequence

events?Which kind of

ressources he has to deal with the

situation?

Compatibility with

external and

internal beliefs?

Is the event compatible

with beliefs?

Conductiveness goals?

Is this situation conductiveness

goal ?

Global pleasantn

ess?Is this

situation pkeasant or

not ?Novelty?Is the event

new ?

Event 2

Controllability consequence

events?Which kind of

ressources he has to deal with the

situation?

Compatibility with

external and

internal beliefs?

Is the event compatible

with beliefs?

Conductiveness goals?

Is this situation conductiveness

goal ?

Global pleasantne

ss?Is this

situation pkeasant or

not ?Novelty?Is the event

new ?

We use two corpora collected at LIMSI-CNRS: (www.emotion-research.net)

EmoTV : « provocative corpora » (Abrilian et al. , 2005) Naturalistic corpus extracted from FrenchTV

news 100 clips of few minutes (< 1 hour)

EmoTaboo (zara et al., 2007) Record of a two-players game designed in

order to induce emotions 10 clips of around 50 minutes each (8 hours)

Definition of an event An event is perceived in the video clips as

triggering the affective state Majority voting between 3 annotators

28 clips from EmoTV Three steps

1.Identify and annotate the emotional events

2.Annotate the temporal dimensions of event

3.Annotate appraisal dimensions of events

Three groups of emotional events are defined in the OCC model (Ortony, Clore, & Collins, 1988) depending on what is evaluated:

- the consequences of events for oneself or for others,- the actions of others - and the perception of objects.

In our corpus, we only observed events that are being evaluated with respect to their consequence for oneself or others and actions of others.

Our events scheme permits to annotate up to 3 emotional events for a given clip. Each event is annotated according to the temporal dimensions

Affective state labels Courage Disappointment

Emotional events1. Election campaign

▪ Temporality: past▪ Duration : months and

more

2. Election results ▪ Temporality: present(D-

day)▪ Duration : minutes

Emotional labels Serenity Pride

Emotional events1. Football cup

▪ Temporality: past▪ Duration : months and

more

2. Football match ▪ Temporality: close future▪ Duration : hours

Emotional labels Anger Despair Disappointment Disgust Helplessness Worry

Emotional events1. Lawsuit (current) :

▪ Temporality: past▪ Duration : months and

more2. Someone is accusing other

people▪ Temporality: present▪ Duration : hour

3. The trial has just finished▪ Temporality: present (D-

day)▪ Duration : minutes

In-lab induced protocol EmoTaboo : adaptation of the

game TABOO It involves interaction between two players One of them has to guess a word that the other

player is describing using his own speech and gestures without uttering five forbidden words

We use strategies for eliciting emotions connected to: the course of the game the selection of the cards The instructions given to the confederate

19

Dyadic interaction Word guessing

game

10 pairs naïve subjects:

4 women, 6 men confederates (close

relations with research staff): 3 women, 4 men

4 cameras 8 hours

Examples of events used for inducing emotions the card contains the word "palimpsest" the experimenter announces a penalty the confederate proposes words with no relation at all with what is

said by the naïve player the confederate finds the mimed word/does not find the mimed

word the confederate is ironic the confederate criticizes the naïve player

Ex: the card contains the word "palimpsest" is estimated by the player as: recent unexpected (the player did not expect a word that he did not

know) incompatible with his(her) purposes to win the game.

Timescale of events is not the same as in EmoTV

EmoTABOO events are rather short term

One long-term event: the game session

In this exploratory study, we tried to show the potential of a scheme of emotional events in order to better understand the emotional states

We proposed to code the temporal aspects of the events: date and duration (short-time, long-time)

Annotation of more dataAnnotate appraisal dimensions of

eventsStudy the relations between

temporal and appraisal annotations the events and the perceived complex

emotions

Goal: designing affective computing applications Possible approach:

Collect small set of real life data Annotate (events, emotion labels, …) Inspire from these annotations to define a protocol

close to the target application to elicit emotional behaviors in-lab

Collect large corpora of induced realistic emotional behavior relevant for the application

Opens a big question: how close to the intended context does training

material need to be?

Thanks for attention

The motivation of this special issue is to report innovative work on the modelling and generation of affect in real-life speech and spoken interaction (including human-human or human-machine interaction, multi-party interaction) or in “realistic” interactions (including realistic fictions).

Editors: Editors: Nick Campbell and Laurence Nick Campbell and Laurence DevillersDevillers

28

LIMSI – TV clips : Interviews from news - FrenchCause event: Unexpected bad Results in an ElectionCurrent event:Impact of election loss on political group

Multimodal Cues:+ speech content-Tense Smile (badControl of their Facial muscles)

-Emotion blends:- Disappointmentmasked by + courage

Cognitive appraisal theory (Scherer, 2000) argues that

- an organism may possess many distributed processes for interpreting this relationship (e.g., planning, explanation, perception, etc.)

-but that appraisal maps characteristics of these disparate processes into a common set of intermediate terms (intermediate between stimuli and response, between organism and environment) called ‘appraisal variables’

(Gratch and Marsella, 2004).