Statistical Affect Detection in Collaborative Chat

Scientific Collaboration& Creativity Lab

Statistical Affect Detection

in Collaborative Chat

CSCW 2013: Mining Social Media Data, Feb. 23

Michael Brooks, Katie Kuksenok, Megan K. Torkildson, Daniel Perry, John J. Robinson, Taylor Jackson Scott, OnaAnicello, Ariana Zukowski, Paul Harris, Cecilia R. Aragon

Scientific Collaboration & Creativity Lab 2/27/2013 2


6:07:57 Ray cool, it worked amusement, relief

6:08:04 Matt woot excitement, joy

6:08:07 Ray awesome, I don't think he needs that long of a sleep after turning it off

acceptance, no affect

6:08:47 We enhanced eready to detect the sticking

no affect

6:08:58 Matt good job supportive, acceptance

6:09:21 seems it did well there happiness, no affect

6:09:26 Ray yeah, pretty cool huh? interest, agreement, happiness

6:09:43 Matt helps keep me from having to stopaicand restart

no affect

6:09:55 Ray indeed, that was the point agreement

June, 2007


• 30 astrophysicists

• US / France

• Daily remote operation of telescope

• Rely on chat to communicate

Nearby Supernova Factory

5

6


SNfactory Chat Logs

• Four years of logs - 449,684 messages

• Manual coding for affective expressions

– 27,344 chat messages coded

– 1-5 coders per message

– 30 affect codes

– Multiple codes allowed

Scott et al. SIGDOC 2012. Adapting Grounded Theory to Construct a Taxonomy of Affect in Collaborative Online Chat.


6:07:57 Ray cool, it worked amusement, relief

6:08:04 Matt woot excitement, joy

6:08:07 Ray awesome, I don't think he needs that long of a sleep after turning it off

acceptance, no affect

6:08:47 We enhanced eready to detect the sticking

no affect

6:08:58 Matt good job supportive, acceptance

6:09:21 seems it did well there happiness, no affect

6:09:26 Ray yeah, pretty cool huh? interest, agreement, happiness

6:09:43 Matt helps keep me from having to stopaicand restart

no affect

6:09:55 Ray indeed, that was the point agreement

June, 2007


Top 13 Affect Codes

4351

3213

1763

1623

1212

1125

975

799

541

518

464

426

369

int…

am…

co…

agr…

an…

co…

acc…

ap…

fru…

sup…

sur…

ant…

ser…

0.808

0.611

0.49

0.491

0.77

0.615

0.657

0.529

0.55

0.583

0.543

0.424

0.602

interest

amusement

considering

agreement

annoyance

confusion

acceptance

apprehension

frustration

supportive

surprise

anticipation

serenity

Times Used Reliability (Kappa)


Linguistic Inquiry and Word Count

(LIWC)

• Detects words for Positive / Negative Emotions

I wish every day could be sunnyand warm. Rain makes me angry.

Positive: 15%Negative: 8%…


11:44:08 Gabri ok that's better relief, serenity

11:44:17 Marcel GREAT ! excitement, happiness, relief, joy

11:44:17 Gabri let's start aic and see anticipation, no affect

11:44:23 Marcel yes ... no affect

11:44:31 Derek Great what? confusion

11:44:32 Gabri can you do that? interest, no affect

11:44:50 derek.. it seems that now the focus is ok no affect

11:45:04 and we can finally start observing no affect

11:45:23 Derek Oh good! happiness, relief, joy

11:45:48 I have been waiting for this moment, because I want to leave the room and get my midnight snack. ;)

amusement

11:46:54 Gabri go... amusement, no affect

11:47:02 and enjoy your snack amusement, no affect

11:47:13 Derek HEhe. amusement

11:47:18 I will bring it back here of course. amusement

June, 2005


The telescope is stuck! >:(

frustration

The telescope is stuuuuuuuuuck...

annoyance

The telescope is stuck??

confusion


• Word counts

• Emoticons

• Word sets– Swear words

– Pronouns

– Negations

– Participant names

• Characters– Capitalization

– Letter repetition

– Punctuation

• Metadata– segment duration, length, rate


• Word counts

• Emoticons

• Word sets– Swear words

– Pronouns

– Negations

– Participant names

• Characters– Capitalization

– Letter repetition

– Punctuation

• Metadata– segment duration, length, rate


Emoticons

Naomi: I think we'd better stopaic... :( sadness

Matt: today was a gym + laundry day :) amusement, happiness

Marcel: and she can't teach over an ssh-channel ;-)

amusement


Word Sets

NegationsPaul: but I wouldn't hazzard a guess apprehension

Ray: cannot talk to camera frustration, no-affect

Swear WordsRay: why the **** doesn't stop_script ******* STOP THE ******* SCRIPT

rage

Matt: ******* ******* ******* I think I broke it frustration, anger, apprehension, embarrassment


Character Features

CapitalizationMarcel: ON TARGET ! relief, joy

Paul: we must set-up adopt an EXPLODING STAR amusement, no-affect

Letter RepetitionRay: noooooooooooooooo, it must be stopped annoyance, anger, fear

Marcel: AAaah too late, they will find meeee amusement

PunctuationRick: looks like something bad happened here... apprehension

Rene: 1 month before max??!? surprise, confusion, considering


Alice: ok, so where was

the ******* SN on the

image?

Feature Value“ok” 1

“telescope” 0

“where” 1

“SN” 1

“image” 1

question marks 1

swears 1

emoticon :) 0

1st person pronouns 0

capitals 2

repetition 0

punctuation 1

length 45

…


Feature importance

Confusion???? length

# question marks"understand"

"confus_""why""what"

"nothing""wrong"

msg. length"thought"

Messages labeled Confusion

Ben: ??? - the answer is likely found in

the otsim code

Marcel: well ... I'm not so sure ...

Gary: Why do we care at all then?

Ray: ummm I mean how does it get to

the header


Feature importance

Apprehension"bad"

"something""problem"

"we""seem""too"

msg. length"not"

# 3rd sg. Pronouns# swearing

Messages labeled Apprehension

Pascal: the problem is than the

automated detection will not work ...

too much galaxy

Ray: But now bad stuff in window

Ben: pascal, we had a problem with

do_fchart

Gabriel: So something is completely

wrong


Feature importance

Amusementemoticon ";)"emoticon ":)"

laughteremoticon ";-)"

"fun"laughter length

"p"# people names

"sleep""of"

Messages labeled Amusement

Kevin: hehe

Ray: hahahaah

Stef: lol ok derek :)

Ray: He never sleeps -- you know that.

Pascal: but I think it could be interesting

for Extreeeeeeeeeeme photometry

study ;-)


Specialized Features

• Count words based on the data

• Medium-specific features

– Emoticons, punctuation…

• Context-specific features

– People names, jargon…

• Affect-specific features

– Swearing vs. emoticons


5:17:48 Marcel ok, so let's cycle the stuff

5:18:04 Rick ok…

5:18:40 Marcel damn mouse cutandpast

5:19:03 Ray off 1 right? then on 1?

5:19:32 Marcel have you telnet sdsugreen ??

5:19:58 Ray director on lbl2 looks dead

5:20:34 Marcel ok, one thind at a time. have you cycled the baytech on sdsugreen ?

5:20:36 Ray what is best way to revive it

5:20:39 baytech

5:20:40 yes

5:20:46 not sdsu

5:21:08 go ahead and do it I am not evneon this **** shift...grrr

5:21:22 Marcel ok, maybe we have to kill director and restart it mkanually

5:21:32 Ray yeah but that's tricky; all these damn arguments

5:23:53 Rick emile, I have no idea what's going on here

5:23:57 only that it is bad

September, 2006


5:17:48 Marcel ok, so let's cycle the stuff

5:18:04 Rick ok…

5:18:40 Marcel damn mouse cutandpast

5:19:03 Ray off 1 right? then on 1?

5:19:32 Marcel have you telnet sdsugreen ??




5:20:39 baytech

5:20:40 yes

5:20:46 not sdsu






September, 2006


Classifier F-measure Precision Recall Accuracy

Naïve Bayes 0.650 0.637 0.691 0.637

Logistic Reg. 0.730 0.731 0.731 0.730

SVM (SMO) 0.759 0.766 0.751 0.761

C4.5 (J48) 0.700 0.724 0.680 0.710


Support Vector Machine

• Accurate

• Fast

• Transparent

# swear words

“frustration” applies“frustration” does not apply

# “

ok”


Support Vector Machine

• Accurate

• Fast

• Transparent

# swear words

?


# “

ok”


0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0

interest

amusement

considering

agreement

annoyance

confusion

acceptance

apprehension

frustration

supportive

surprise

anticipation

serenity

Precision Recall


Interpretability

• How is the classifier making decisions?

• What features are important in the model?

# swear words# “

ok”



Feature importance

Amusementemoticon ";)"emoticon ":)"

laughteremoticon ";-)"

"fun"laughter length

"p"# people names

"sleep""of"

Messages labeled Amusement

Kevin: hehe

Ray: hahahaah

Stef: lol ok derek :)

Ray: He never sleeps -- you know that.

Pascal: but I think it could be interesting

for Extreeeeeeeeeeme photometry

study ;-)


Interpretable Classifiers

• Explain classification errors

• Suggest improvement strategies

• Discover interesting anomalies


Future Work


Sequential Modeling




5:20:39 baytech

5:20:40 yes

5:20:46 not sdsu







Interactive Visual Analysis


0

5000

10000

15000

20000

25000

30000

35000

40000

45000

Nu

mb

er

of

Tw

eets

Time (EST), 2/3/2013positive negative neutral

Affect in Twitter

kick

off

gam

e o

ver

gam

e re

sum

es

bla

cko

ut

gam

e re

sum

es

hal

ftim

e


Classify…• Positive/negative/neutral

sentiment

• Highly granular emotions

• Anything else you can label

In…• longer, formal documents (blog

posts, reviews)

• individual sentences

• instant messages

• tweets

• Anything else you can put in CSV

github.com/etcgroup/aloe

Download it, use it, & tell us what you think!

Michael [email protected]

http://depts.washington.edu/sccl

Technology

Statistical Affect Detection in Collaborative Chat