PeopleTones: a system for the detection and notification of buddy proximity on mobile phones

PeopleTones: a system for the detection and notification of buddy proximity on mobile phones

Kevin A. LiTimothy SohnSteven Huang

William G. GriswoldUniversity of California, San

Diego

ubiquitous

ample computational power

a few sensors

a few actuators

proactive context-awareness

messaging

You are driving by Safeway. Reminder: Buy steak.

location-based reminders

The slopes on Beaver Run have opened!

unobtrusive notifications

Bzzzzt!

crappy sensors crappy actuators

cheap sensors could lead to many false notifications

cheap actuators could lead to misunderstood cues

proactive notification + commodity hardware flood of meaningless notifications

two proximity states: far and near (< 2 city blocks)

when a buddy becomes near,

play her sound or vibration cue

runs on commodity hardware (Windows Smartphone)

PeopleTones

“It was so cool to see who was home by the time I got home. I could tell if Jenny was home when I passed by University. So if we were going to go eat or something I could ask her. Oh she’s home, so let’s call her and see if she wants to eat.”

focus on application goals to avoid over-engineered, impractical solution

proximity is easier than location

acceptable if notifications missed or not understood

approach

privacy-friendly proximity detection algorithm

technique for reducing sensor noise without sapping power

method for generating a language of understandable vibrotactile cues

exploratory study of buddy proximity cues

contributions

proximity detection

[LaMarca et al. 2005]

build on location?

GPS doesn’t work indoors, in urban canyons

tower-based systemsmust keep database current

require wardriving

how about tower overlap?NearMe [Krumm 2004]

iPhoneiPhone

initial data collection

used a GSM phone to record the cell towers it saw every 5 minutes

3 GSM phones, kept 1 stationary

gather data at a variety of distances (0 - 1.2 miles)

initial measurements

a and b are the sets of cell towers seen by each phone

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.2 0.4 0.6 0.8 1 1.2

Ratio

of C

ell T

ower

s

Distance (miles)

Suburbs 1Suburbs 2Downtown

initial measurements

a and b are the sets of cell towers seen by each phone

evaluatingproximity algorithm

can our overlap-ratio algorithm detect proximity accurately enough to support nice-to-know information?

requirements

cannot be annoying

when the system detects a buddy is near,

they should really be near

OK to not detect every time

if a buddy is nearby and stationary, we’ll have multiple chances

the datasetused the dataset collected by wardriving seattle

[Chen et. al., 2006]

coverage

Suburb

Downtown

precision

100% precision every report is valid

recall

100% recall every near incident is detected

metric: precision and recall

how do we extract the relevant data?

only care about when two phones are near or far from each other

why not pull out each set of data by different distance thresholds?

turns out mobile phone tower readings fluctuate over time (e.g., due to load balancing)

we can crosscut the dataset to approximate precision and recall for different scenarios

nearby

extract pairs of readings taken within 90s

569,264 pairs from Suburb

379,285 pairs from Downtown

calculate precision and recall for different ratio threshold values

nearby precision

100% precision every report was valid

100% recall every near incident was detected

suburb downtown

0.75

0.8

0.85

0.9

0.95

1

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Prec

ision

Distance (km)

0.3

0.4

0.5

nearby precision

suburb downtown

100% precision every report was valid

100% recall every near incident was detected

0.75

0.8

0.85

0.9

0.95

1

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Prec

ision

Distance (km)

0.3

0.4

0.5

0.80.820.840.860.88

0.90.920.940.960.98

1

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Prec

ision

Distance (km)

0.3

0.4

0.5

nearby precision

suburb downtown

100% precision every report was valid

100% recall every near incident was detected

nearby recall

Ratio Recall(Downtown)

Recall (Suburb)

0.1km 1.0 km 0.1 km 1.0 km

0.3 0.74 0.66 0.67 0.66

0.4 0.50 0.41 0.42 0.40

0.5 0.39 0.30 0.30 0.29

100% precision every report was valid

100% recall every near incident was detected

reducingsensor noise

initialapproaches

wait for 2 consecutive-same-readings– Too many false positives

wait for 3 consecutive-same-readings– Too much delay

2-bit-filter(“eventually 3 more”)

filterevaluation

for noise filtering, interested in transitions from far to near and vice-versa

extract seattle wardrive readings at 30s intervals

try three algorithms on this subset, baseline is single report

filterevaluation

Filter False Positive Reduction

1-same (baseline) 0%

2-same-filter 53.8%

3-same-filter 80.9%

2-bit-filter 84.9%

adaptive sampling rate

sampling once every 20s kills the phone in less than a day

increasing sampling rate to once per 90s helps but introduces a worst-case delay of 270s

sample at 90s when in steady state, 20s when transitioning

buddy cues

mapping musicto vibrations

problem

we want to convert music to vibrations…

…but the phone’s vibrator only turns on and off

…at single frequency, single amplitude

pulse width modulation

electric motors do this to save power

in the case of vibrotactile motors this also decreases its rotational frequency

perceived as different vibration levels

can produce 10 levels of 20ms pulses

capturing the essence of music

overview of approach

just using beat doesn’t always work

mapping lyrics doesn’t work well

basic idea: convey the current energy level of the music

remove noise

isolate 6.6kHz to 17.6kHz components using 8th order Butterworth Filter

use amplitude threshold, to keep only components greater than the average

takerunning sum

take running sum of absolute value, generate 1 value every 20ms

this keeps length consistent

exaggeratefeatures

compose output from previous step with power function:

Axn , x is sample,

A and n are constants, 10<=A<15, 1<=n<=2

Beethoven’s 5th Symphony

matching vibration sequence

Michael Jackson – Smooth Criminal

matching vibration sequence

examples(requires imagination)

so far…

would the techniques we used for proximity detection, sensor noise filtering and vibrotactile cues work in the wild?

can peripheral cues be deployed on mobile phones despite poor sensors and actuators?

(what experiences can such a system enable?)

field study

two proximity states, far and near (< 2 blocks)

when a buddy is near, play their song

if phone is in vibrate mode, play a matching vibrotactile sequence

PeopleTones

participants

3 groups of friends, 2 weeks

could you tell who it was?

could you tell who it was?

0% 20% 40% 60% 80% 100%

My Choice

Your Choice

Nature

Yes

Ignored

No

0% 20% 40% 60% 80% 100%

My Choice

Your Choice

Nature

Acted on or Nice to KnowDidn’t Notice or not that usefulAnnoyed

user response to the cue

designing peripheral cues for the wild

higher comprehension rate when users select their own cues

obtrusiveness of music cues was not a concern

mapping music to vibration was most successful for people who knew the songs well

semantic association is key to learnability

userexperience

“One time at the library, I wanted to eat with someone and so I went outside to call someone. The phone vibrated. I just called the person to meet up.”

lessons

cues were sensible, but not socially obtrusive

proximity algorithm worked well in the wild

emphasizing elimination of false positives was effective in combination with 2-bit counter

dwelling/lingering led to successful recall

whole can be greaterthan its parts

despite crappy sensors and actuators, mobile phones can achieve adequate context awareness and notification

with careful system-level design these can be brought together into a useful proactive context-aware application like PeopleTones

PeopleTones: a system for the detection and notification of buddy proximity on mobile phones

Kevin A. LiTimothy SohnSteven Huang

William G. GriswoldUniversity of California, San

Diego

EXTRA SLIDES

measuring vibrations

0 5 10 15 20 25 30 35 40 45 50

Time (ms)

Vo

lts

(V

)0

1

.5

3.0

4.5

6.

0

0 5 10 15 20 25 30 35 40 45 50

Time (ms)

Vo

lts

(V

)0

1

.5

3.0

4.5

6.

0

0 5 10 15 20 25 30 35 40 45 50

Time (ms)

Vo

lts

(V

)0

1

.5

3.0

4.5

6.

0

0 5 10 15 20 25 30 35 40 45 50

Time (ms)

Vo

lts

(V

)0

1

.5

3.0

4.5

6.

0

Duty Cycle = 100%

Duty Cycle = 60%

Duty Cycle = 18%

Duty Cycle = 37%

zzz

z

z zvibrotactilesignals

far apart

unfortunately, there are few points in dataset that are both far apart and proximate in time

expected atypical results

used the entire dataset

55,181,015 pairs from Suburb

36,769,390 pairs from Downtown

far apart precision

suburb downtown

100% precision every report was valid

100% recall every near incident was detected

far apart precision

0.75

0.8

0.85

0.9

0.95

1

0.1 0.3 0.5 0.7 0.9 1.1 1.3 1.5 1.7 1.9 2.1 2.3

Prec

ision

Distance (km)

0.3

0.4

0.5

suburb downtown

100% precision every report was valid

100% recall every near incident was detected

0.88

0.9

0.92

0.94

0.96

0.98

1

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Prec

ision

Distance (km)

0.3

0.4

0.5

far apart precision

0.75

0.8

0.85

0.9

0.95

1

0.1 0.3 0.5 0.7 0.9 1.1 1.3 1.5 1.7 1.9 2.1 2.3

Prec

ision

Distance (km)

0.3

0.4

0.5

suburb downtown

100% precision every report was valid

100% recall every near incident was detected

far apart recall

Ratio

Recall(Downtown)

Recall (Suburb)

0.1km 1.0km 0.1km 2.4km

0.3 0.12 0.08 0.09 0.06

0.4 0.05 0.04 0.03 0.02

0.5 0.03 0.02 0.02 0.01

100% precision every report was valid

100% recall every near incident was detected

“It was so cool to see who was home by the time I got home. I could tell if Jenny was home when I passed by University. So if we were going to go eat or something I could ask her. Oh she’s home, so let’s call her and see if she wants to eat.”

“One time at the library, I wanted to eat with someone and so I went outside to call someone. The phone vibrated. I just called the person to meet up.”

generating vibrations

same location

Extract pairs of readings taken within 5s

28,625 pairs from Suburb

19,087 pairs from Downtown

GPS confirmed 99.9% within 100m of each other

Ratio Recall

(Downtown)Recall

(Suburb)

0.1 0.96 0.96

0.2 0.84 0.85

0.3 0.83 0.83

0.4 0.57 0.58

0.5 0.44 0.44

same location recall

100% precision => every report was valid

100% recall => every near incident was detected

You are driving by Safeway. Reminder: Buy steak.

eyes-freenotifications

[LaMarca et al. 2005]

build on location?GPS doesn’t work indoors, in urban

canyons

tower-based systemsmust keep database currenttower adaptation (e.g., load balancing)

location sensingon iphone

“It was so cool to see who was home by the time I got home. I could tell if Jenny was home when I passed by University. So if we were going to go eat or something I could ask her. Oh she’s home, so let’s call her and see if she wants to eat.”

2-bit-filter