Affect and Emotion-Oriented Systems

Human-Computer Interaction

“I believe that logic + emotion are a winning combination. When useful and useable meet delight

great things happen. It’s about balance.”

David Armano

Master on Software Engineering Human-Computer Interaction

Stefan Negru – www.blankdots.com

http://www.flickr.com/photos/sskennel/

What are emotions, feelings and the other affect related concepts ?



Definition: ―Affect is a feeling or emotion as distinguished from cognition, thought, or action.‖



―Emotion encompasses physiological, affective, behavioral, and cognitive components.‖

Scott Brave & Clifford Nass, 2008



―Emotion is a reaction to events deemed relevant to the needs, goals, or concerns of an individual.‖

Scott Brave & Clifford Nass, 2008



neurological structure of emotion (LeDoux, 1996)



emotion versus mood

moods are not directed at any object in particular and are experienced as more diffuse, global, and general

state

a person can be sad about something - an emotion

or generally depressed - a mood



emotion versus mood

An emotion can lead to or take part in establishing a certain mood

Sadness (& other factors) depression



emotion versus feeling

Feelings relate more to a property associated to and

object or being, than to a person’s state of mind.

Feeling last more (days/weeks/years) than emotions (seconds) or moods (hours / days)



emotion versus feeling

Feelings are often pointed towards a class of objects,

beings distinguishable through a certain property

―I love Adobe CS applications!‖

―I hate text-editing applications‖



emotion and memory

negative events, which tend to be highly arousing, are typically remembered better than

positive events



emotion and performance

keeping a user happy may, therefore,

not only affect satisfaction, but may also lead to efficiency and creativity



A Circumplex model of affect (Russell, 1980).



Russell’s circumplex remodeled (Desmet, 2002; van Gorp, 2006)



emotion dimensions

Value

pleasant versus unpleasant

Arousal (psychological)

anxiety versus boredom



Anxiety, Boredom and Flow (Csikszentmihalyi, 1990; van Gorp 2006)



Flow

Achieving the desired state and keeping the user in that state is our goal.



Humane interfaces consider emotions

“Products were once designed for the functions they performed. But when all companies can make products that perform their functions equally well, the distinctive advantage goes to those who provide pleasure and enjoyment while maintaining the power. If functions are equated with cognition, pleasure is equated with emotion; today we want products that appeal to both cognition and emotion.”

Don Norman

―The Transmedia Design Challenge: Technology that is Pleasurable and Satisfying‖,

ACM Interactions Volume 17 Issue 1, January + February 2010





http://www.nbcchicago.com/news/local-beat/Frank_lloyd-wright-fallingwater-lego.html



http://getsatisfaction.com/spotify/topics/when_will_we_have_a_blackberry_client



It’s all nice, tidy and easy to understand right ?



It’s all nice and easy to understand right ?

Wishful thinking



Where’s the problem ?



A few Problems

1. Detecting Emotions

2. Labeling Emotions

3. Processing Emotions

4. Have an unobtrusive interface



1. Detecting Emotions

Neurological responses

via electroencephalography (EEG) or magneto‐resonance imaging (MRI)

User input

Facial expression Voice



http://www.emotiv.com/

based on electroencephalography (EEG) technology



―Lie to me‖– Tv show

http://www.imdb.com/media/rm817007104/tt1235099



METT – Micro Expression Training Tool

SETT – Subtle Expression Training Tool



Asking our user how he feels.



2. Labeling Emotions

Which are basic emotions ?

Group Emotions

Mixed versus pure emotions



http://wefeelfine.org/

12 million human emotions recorded since 2005 (2009 statement)



Emotion-Oriented Systems The Humaine Handbook, Springer 2011



Basic emotions

anger, disgust, fear, happiness, sadness and surprise



Labeling emotions

Labeling is important but not for a machine

A label for the machine is a x,y coordinate on the circumplex.





Proposal of labeling emotions

FeelTagged an Arduino-based Social-Emotional Interactive System – Stefan Negru 2010



3. Processing emotions

Is it a mood/feeling/emotion ?

What category/dimension ?

Multiple and/or complex emotions

Emotion intensity, timing and context observed ?

etc.



Annotating emotions

<emotionml xmlns="http://www.w3.org/2009/10/emotionml">

<emotion>

<dimensions set="PleasantAnxiety">

<pleasant value="1.3"/> <!—- somewhat positive pleasant state -->

<anxiety value="1.9"/> 

<modality set="PersonalContext" mode="face voice" medium="visual acoustic"/>

</dimensions>

</emotion>

</emotionml>

EmotionML http://www.w3.org/TR/emotionml/



5. Have an unobtrusive interface

Providing a useful, easy to understand and adaptive interface – secondary goal

Unobtrusive interface- primary goal



emotion and attention

an interface capable of detecting, or predicting a user’s emotional or mood state could

similarly assume an affect‐regulation role, helping to guide attention away from negative

and toward more positive stimuli



emotion and attention

Disregarding the flow state

―This system failed to understand your command.‖ – blaming itself

―The command was not understood.‖ – blaming no one

―You did not speak clearly enough for your command to be understood.‖ – blaming the user





Affective based interaction

Affective aspects of interaction (Rogers, 2007)

frustration

how we can detect and reduce it?



Affective based interaction

Affective aspects of interaction (Rogers, 2007)

persuasive technologies

how we can change the user attitude & behavior

by using the actual technologies?

http://www.happypills.es/



affective interaction

example: a robot hardware platform called iCAT uses a set of logical rules to convey emotional states as it makes decisions, with the goal of improving human-robot interaction

Philips Research Laboratory 2005

http://www.newscientist.com/blog/technology/2007/07/what-if-robots-not-only-seemed.html





http://www.research.philips.com/



http://pleoworld.com/

Pleo is an animatronic dinosaur toy designed to emulate the appearance and (imagined) behavior of a week-old baby Camarasaurus.



Ethical aspects

The goal is not to control people actions but to support them in order to improve their emotional

state and to provide a better interaction based on the user’s emotional state



Dark side of emotions

Humans could have an offensive behavior

regarding the digital entities

www.agentabuse.org






example (Tatsuya Nomura, 2005):

impolite behavior of the patients,

when they discovered that the therapist is a virtual agent






experiment (Christoph Bartneck et al., 2005):

humans tend to hurt more easily robots/devices

than people



Unobtrusive versus Informative

Should users be informed that their affective states are being monitored and adapted to ?



Questions

Education

Affect and Emotion-Oriented Systems