MoodScope:Sensing mood from smartphone usage patterns

Robert LikamwaLin Zhong

Yunxin LiuNicholas D. Lane

Asia

2

Earlier today…

Mood-Enhanced Apps

Socialecosystems

Media recommendation

Personalanalytics

3 of 30

Some time in the future…

Affective Computing(Mood and Emotion)

Audio/Video-based(AffectAura, EmotionSense)

Biometric-based(Skin conductivity,

Temperature, Pulse rate)Highly temporalHigh cost of deploymentHassle

Captures expressionsPower hungrySlightly invasive

4 of 30

Can your mobile phone infer your

mood?

Can your mobile phone infer your

mood?From already-

available, low-power information?*

* No audio/video sensing, no body-instrumentation

MoodScope ∈ Affective Computing

Audio/Video-based

Usage Trace-based(MoodScope)

Biometric-based

Very direct, Fine-grainedHigh cost of deployment

Captures expressionsPower hungrySlightly invasive

Passive, ContinuousHow to model mood?

7 of 30

Mood is…• … a persistent long-lasting state

o Lasts hours or dayso Emotion lasts seconds or minutes

• … a strong social signalo Drives communicationso Drives interactionso Drives activity patterns

8 of 30

happysadnervousdepressed excited

relaxed

calm

stressed bored

Circumplex model (Russell 1980)

attentive

9 of 30

Mood is…• … a persistent long-lasting state

o Lasts hours or dayso Emotion lasts seconds or minutes

• … a strong social signalo Drives communicationso Drives interactionso Drives activity patterns

10 of 32

• … a strong social signalo Drives communicationso Drives interactionso Drives activity patterns

How is the user communicating?

What apps is the user using?

f ( ) = moodusage

13 of 30

iPhone Livelab Logger• Web history• Phone call history• Sms history• Email history• Location history• App usage

Adapted From C. Shepard, A. Rahmati, C. Tossel, L. Zhong, And P. Kortum, "Livelab: Measuring Wireless Networks And Smartphone Users In The Field," In Hotmetrics, 2010.

14 of 30

iPhone Livelab Logger• Web history• Phone call history• Sms history• Email history• Location history• App usage

Runs as shellHash private dataUploads logs to our server nightly

How can we generate mood labels?15 of 30

Mood Journaling App

User-base32 users aged between 18 and 29

11 females 16 of 30

Inference

• Detect a mood pattern

• Validate with only 60 days of data

• Wide range of candidate usage data

• Low computational resources

17 of 30

Daily Mood Averages

• Separate pleasure, activeness dimension

• Take the average over a day _______________

4

Σ( )

18 of 30

Exploring Features• Communication

o SMSo Emailo Phone Calls

• To whom?o# messageso Length/Duration

Consider “Top 10” Histograms

How many phone calls were made to #1? #2? … #10?

How much time was spent on calls to #1? #2? … #10?

19 of 30

?

?

Exploring Features• Communication

o SMSo Emailo Phone Calls

• To whom?o# messageso Length/Duration

• Usage ActivityoApplicationsoWebsites visitedo Location History

• Which (app/site/location)?o# instances

20 of 30

Previous Mood• Use previous 2 pairs of mood labels

21 of 30

Data Type Histogram by: Dimensions

Email contacts# Messages 10# Characters 10

SMS contacts# Messages 10# Characters 10

Phone call contacts# Calls 10Call Duration 10

Website domains # Visits 10Location Clusters # Visits 10

Apps# App launches 10App Duration 10

Categories of Apps# App launches 12

App Duration 12Previous Pleasure and Activeness Averages

N/A 4

22 of 30

??

• Multi-Linear Regressiono Minimize Mean Squared Error

• Leave-One-Out Cross-Validation

• Sequential ForwardFeature Selection during training

23 of 30

Model Design

Sequential Feature Selection

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 310

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8Improvement of model as SFS adds more

features

Number of Features Used

Mean

Sq

uare

d E

rror (Each line is

a different user)

24 of 30

1 3 5 7 9 11 13 15 17 19 21 23 25 27 290

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8Improvement of model as SFS adds more

features

Number of Features Used

Mean

Sq

uare

d E

rror

Sample Prediction

0 10 20 30 40 50 602

3

4

Mood (Pleasure)Estimated Mood

Days

Dail

y M

ood

Avera

ge

25 of 30

Error distributions• Error2 of > 0.25 will

misclassify a mood label

93% < 0.25 error2

0.0001

0.001

0.01

0.1

1

25%ile

10%ile

Users

Sq

uare

d E

rror

26 of 30

vs. Strawman ModelsModels using full-knowledge of a user’s data with LOOCV

Model A: Assume User’s Average Mood73% Accuracy

Model B: Assume User’s Previous Mood61% Accuracy

MoodScope Training: 93% Accuracy.

27 of 30

Personalized Training

10 20 30 40 50 590%

20%

40%

60%

80%

100%

Incremental personalized model

Training Days

Mod

el

Accu

racy

28 of 30

All-user modelaccuracy

Personalized/All-userHybrid Training

10 20 30 40 50 590%

20%

40%

60%

80%

100%

Incremental personalized model

Hybrid mood model

Training Days

Mod

el

Accu

racy

29 of 30

Phone

Mood Inputs/Usage Logs

Mood and Usage History

Cloud

Mood ModelMood Model

Current Usage Model

Training

Inferred Mood

Resource-friendly Implementation

30 of 30

Inferred Mood

API

TEXAS MEDICAL CENTER

RICE UNIVERSITY

Inferred Mood

API

MoodScope:Sensing mood from smartphone usage

patterns

• Robustly (93%) detect each dimension of daily moodo On personalized modelso Starts out with 66% on generalized

models

• Validate with 32 users x 2 months worth of data

• Simple resource-friendly implementation

35

Discriminative Features

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 310

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8Improvement of model as SFS adds more

features

Number of Features Used

Mean

Sq

uare

d E

rror

Relevant features

36

Discriminative Features

Calls

Emai

lSM

SW

ebApp

s

Loca

...

Prev

....

0

20

40

60

80

100

120 Pleasure

Activeness

Nu

mb

er

of

Featu

res

37

TODO• Wider, longer-term evaluation

o How does the model change over time?

• In-use accuracy metricso Not cross-validation

• Social Factors/Impacto Study Mood-sharingo Provide assistance to psychologists