Feature Extractors for Integration of Cameras and Sensors during

End-User Programming of Assistive Monitoring Systems

Alex EdgcombFrank Vahid

University of California, RiversideDepartment of Computer Science

1 of 16

?Motion sensor

Sensors and actuators in MNFL [1] for end-user programming

Alex Edgcomb, UC Riverside 2 of 16

“Person at door”

LED lights in house

“Person at door”

Outdoor motion sensor

Doorbell

• Assistive monitoring• User customizability essential [2][3]

[1] Edgcomb, A. and F. Vahid. MNFL: The Monitoring and Notification Flow Language for Assistive Monitoring. Proceedings 2nd ACM International Health Informatics Symposium, 2012. Miami, Florida.[2] Philips, B. and H. Zhao. Predictors of Assistive Technology Abandonment. Assistive Technology, Vol. 5.1, 1993, pp. 36-45.[3] Riemer-Reiss, M. Assistive Technology Discontinuance. Technology and Persons with Disabilities Conference, 2000.

Porch light

LED lights in house

Expanding the previous example

Alex Edgcomb, UC Riverside 3 of 16

“Person at door”

“Person at door”

Outdoor motion sensor

Doorbell

Light sensor

Webcams are cheap

4 of 16Alex Edgcomb, UC Riverside

Webcams can do more than sensors

Fall down at home

In room for extended time

Can do same as some sensors

Motion sensorLight sensor

5 of 16Alex Edgcomb, UC Riverside

Identify personat front door

Problem: Integration of webcams and sensors

6 of 16

Homesite

Commercial approach:

Alex Edgcomb, UC Riverside

?

Outdoor motion sensor

Solution: Feature extractor

7 of 16

92Integer stream

output

0

100

Alex Edgcomb, UC Riverside

Extract some feature

Video stream input

Identify person at door in MNFL

Alex Edgcomb, UC Riverside 8 of 16

Outdoor motion sensor

Person in room for extended period of time in MNFL

9 of 16Video’s YouTube link

Alex Edgcomb, UC Riverside

Many feature extractors are possible

10 of 16Alex Edgcomb, UC Riverside

Are feature extractors usable by lay people? Two usability trials.

• 51 participants• Trials required as 1st lab assignment• Non-engineering/non-science students at UCR

1 2 3 4 50

10

20

30

40

50

60

70

80

90

100

Programming experience(1- little/none; 5- a lot)

Perc

enta

ge o

f par

ticip

ants

11 of 16Alex Edgcomb, UC Riverside

Participant reference materials

• One-minute video showing how to spawn and connect blocks.

• Overview picture

12 of 16Alex Edgcomb, UC Riverside

Example challenge problem

13 of 16Alex Edgcomb, UC Riverside

actual participant solution

Trial 1: Increasingly challenging feature extractor problems

25 participants14 of 16

10 9 7-8 5-6 3-4 1-2 00%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Fall detector (easy) Visitor detector (medium)

Leave at night detector (hard)

Rubric score

Perc

ent o

f sub

ject

s

Alex Edgcomb, UC Riverside

Trial 2: Feature extractor vs logic block

26 participants15 of 16Alex Edgcomb, UC Riverside

10 9 7-8 5-6 3-4 1-2 00%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Visitor detector (feature)

Emergency button (logic)

Rubric score

Perc

enta

ge o

f sub

ject

s

Conclusions

• Feature extractors– Elegant integration of cameras and sensors– Quickly learnable by lay people

• Future work– Develop additional feature extractor blocks– Trade-off analysis between privacy,

communication, and computation16 o f 16Alex Edgcomb, UC Riverside