An Exploration with Online Complex Activity Recognition

An Exploration with OnlineComplex Activity Recognitionusing Cellphone Accelerometer

Zhixian Yana

Samsung Research, [email protected]

Dipanjan ChakrabortyIBM Research, [email protected]

Sumit MittalIBM Research, [email protected]

Archan MisraSingapore [email protected]

Karl AbererEPFL, [email protected]

aThis work was performed while the author was at EPFL.

Permission to make digital or hard copies of part or all of this work for personalor classroom use is granted without fee provided that copies are not made ordistributed for profit or commercial advantage and that copies bear this noticeand the full citation on the first page. Copyrights for third-party componentsof this work must be honored. For all other uses, contact the Owner/Author.Copyright is held by the owner/author(s).UbiComp’13 , Sep 08–12 2013, Zurich, SwitzerlandACM 978-1-4503-2215-7/13/09.

http://dx.doi.org/10.1145/2494091.2494156

AbstractWe investigate the problem of online detection of complexactivities (such as cooking, lunch, work at desk), i.e.,recognizing them while the activities are being performedusing parts of the sensor data. In contrast to prior work,where complex activity recognition is performed offlinewith the observation of the activity available for its entireduration and utilizing deeply-instrumented environments,we focus on online activity detection using onlyaccelerometer data from a single body-worn smartphonedevice. We present window based algorithms for onlinedetection that effectively perform different tradeoffsbetween classification accuracy and detection latency. Wepresent results of our exploration using alongitudinally-extensive and clearly-annotated cellphoneaccelerometer data trace that captures the true-lifecomplex activity behavior of five subjects.

Author Keywordsactivity recognition, complex activities, accelerometer,online detection, detection accuracy, detection delay

ACM Classification KeywordsI.2.1 [Applications and Expert Systems]

General TermsAlgorithms, Experimentation

Poster, Demo, & Video Presentations UbiComp’13 Adjunct, September 8–12, 2013, Zurich, Switzerland

199

IntroductionComplex activities are referred to as Activities of DailyLiving (ADL) that can be detected using sensor datacollected from a consumer-grade personal mobile devicelike smartphone. Examples of complex activity are“cooking at home”, “watching TV at home”, “lunch inoffice”. Prior work on such complex activity detectionoperates offline [2, 4], which means the classification isperformed by analyzing the entire stream of sensor dataafter the completion of an activity episode, based onfeatures extracted over the entire activity duration.Additionally, it utilizes deeply-instrumented settings [1, 3]such as the use of multiple body-worn accelerometersensors or RFID tagging of household objects. Our workdiffers from such works from three critical aspects: (1)online or in-situ complex activity detection, i.e., the abilityto classify ADLs rapidly and early, while they are beingperformed ; (2) require no special instrumentation ofeither the individual or the environment, and insteadinvestigate the extent to which such online classificationcan be performed solely using accelerometer data from acommodity smartphone; (3) perform the algorithms undernatural conditions of daily living, i.e., in the wild.

A Uniform Model

Acti

Act1

ActN

Act2

Figure 1: Uniform window model

Acti

Act1

ActN

Act2

M1 M2 MK …

Figure 2: Multiple window model

Figure 3: Continuous ACTdetection – three activityboundaries detected based onthis strategy. The three activities(act1,act2,act3) are detected asACTj , ACTi, and ACTj ,respectively.

Our motivation is to investigate the research challenges ofonline continuous complex activity detection in completelynaturalized real-life data, with a single smartphoneaccelerometer. The detailed contributions are:

1. We propose two different window-based learningapproaches for early detection of complex activities.By investigating discriminatory features of differentcomplex activities, we study their across-time effectsboth on classification accuracy and detection delayfor online detection.

2. To perform continuous activity recognition in astreaming fashion, we present a simple, but

demonstrably effective, demarcation algorithm thatidentifies the points where an individual transitionfrom one complex activity to another.

3. We evaluate our approach using a real-lifenaturalized setting dataset that is significant bothfor its longitudinal trace and for the availability ofground-truth of corresponding complex activities.

Online Continuous ApproachOur approach is designing window-based model to studyearly detection of complex activities in real-time usingcellphone accelerometer; in addition, we investigate thecontinuity of accelerometer streams for detecting activitytransitions (e.g., from ‘cook’ to ‘eat’).

Window based Early Detection StrategiesWe design window based strategies to make early, onlinedetection of activities from accelerometer streams. Beforeproceeding to describe our algorithms, it is first importantto establish the set of classificatory features, defined overeach frame of observed accelerometer data. A frame is asmall window of accelerometer data, e.g., 5 secs or 10secs. Like [4], we transform the raw accelerometer streaminto locomotive features - each frame indicates a motionstatus, such as sit, stand and walk, in order to get a richset of feature inputs. We design two window models.

• Uniform Modeling – The first model is building asingle classifier model M, which is trained by usingthe entire duration of the complex activity instances(see Fig. 1). For each instance of a specific complexactivity Acti, we calculate a set of featurescomputed over the entire duration of the activityinstance. All activities are used to train a uniformmodel that will be used for online testing withpartial ongoing sensor streams.


200

• Multiple-Window based Models – Complex activitiestypically have pronounced differentiation duringcertain sub-intervals. Intuitively, classification could

Figure 4: Detection accuracyuniform model (δconf=0.9)

Figure 5: Detection delay withuniform model (δconf=0.9)

0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95 120

30

40

50

60

70

80

confidence threshold

accu

racy

(%)

user1 user2 user3 user4 user5

Figure 6: Accuracy vs.Confidence threshold (uniformmodel, ∆W = 50S)

be accurate if we had explicit and separate modelsthat were trained on the corresponding time intervalof the activities. Accordingly, we investigate the useof ‘multiple-window’ based approach, where eachtime window is associated with a different classifiermodel that is tuned to the specific features exhibitedby the complex activities within that window. Webuild K separate models (i.e., M1 to MK in Fig.2), corresponding to the time-window partitions ofan ACT . The online testing is incrementally usingthe K models that denote the time-instants thatpartition the incoming online sensor data stream.

Online Demarcation StrategiesAfter early detection of activity label, we need to identifythe activity duration, i.e., (tsi, tei) from a continuousstream containing a sequence of complex activities. Fig. 3provides a sketch of our online activity boundarysegmentation algorithm. For simplicity, we show thereal-time prediction probabilities of two activity labels,i.e., p(ACTi) and p(ACTj). Since the activities aresequential, the end time of the current activity alsoindicates the start time of the next one.1 Consider a newactivity has started at tstart and is predicted as ACTi attdetect . As the stream continues, we calculate thecomplete prediction vector of all possible activitiespredVec = 〈· · · , p(ACTi), · · · , p(ACTj), · · · 〉at each time tcurr . We can declare that tcurr is the tendof ACTi if the following conditions hold:

1Although concurrent activities can be also supported using theprobabilistic model, this is not the focus of this paper.

• Persistent Gradient Monotonicity - δtrend: Theprobability of the currently predicted activity, i.e.,p(ACTi), shows negative slope, and in themeanwhile, at least another activity probability [i.e.,p(ACTj), j 6= i] shows positive slope, for a timeduration larger than δtrend.

• Exceed Minimal Duration - timeOut(ACTi): Foreach activity type, a timeout value is set to theminimum duration of all instances of that ACT .This means that we cannot declare ‘ACTi hasended’ until at least the corresponding timeout(ACTi) has elapsed. Hence, a transition can bedeclared only if the current time instant tcurrsatisfies the minimal duration property, i.e.,tcurr-tstart ≥ timeOut(ACTi).

Activity Label RefinementSo far, we have computed activity labels and activityboundaries only from sensor observations at currentstatus. Once the end time of the current activity ACTihas been predicted, we exploit an activity state transitionmatrix between different activities, i.e., p(ACTi|ACTj),to re-compute the predicted label for ACTi. In effect, thisis a refinement of the predicted label from the earlydetection strategies, with knowledge of the activityevolution till its (predicted) boundary and knowledge ofthe transition probabilities. The transition matrixp(ACTi|ACTj) is computed from the available ground

truth data. We recompute a new

p(ACT(t)i ) after the

boundary of ACTi being predicted, combining the supportof the current observations in time state t and the statetransition from the previous activity at time state t-1 .

p(ACT(t)i ) = w1 ∗ p(ACT (t)

i ) + w2 ∗ p(ACT (t)i |ACT

(t−1))

= w1 ∗ p(ACT (t)i ) + w2 ∗

N∑j=1

{p(ACT (t−1)j )× p(ACTi|ACTj)}


201

Exploration Results

Figure 7: Online demarcationperformance

Our experiments are based on previous real-life‘in-the-wild’ accelerometer data collected from 5 userscarrying a Nokia N95 phone, loaded with an applicationthat sampled the accelerometer at 30Hz, continuously asthe users were immersed in their daily activities at homeor in office. The data details can be found in [4].

Window-based Early Detection – We experimented bothuniform and multiple-window based models, for testinghow models built with partial observation data. Weidentified the accuracy increases with increasing frameincrements (∆W ), while an overly large value of ∆W cancause more errors in transition detection, especially if theincremented buffer straddles the boundary of twoactivities and includes a bigger chunk of the next activity.Fig. 4-6 shows the early detection performance for all 5subjects. We found that most complex activities can bedetected within ≈5-25% of their overall activity length. Inessence, many of the long running activities (30-100 mins)could be detected within 4-20 minutes of their onset. Anotable aspect was that almost all activities need lessthan 50% of their data, in order to be classified with thesame accuracy as they would be with complete data.

Demarcation detection and label refinement – Fig. 7shows the detailed continuous detection performance offive users. We fixed the monotonicity tolerance δtrend to10 for the state transition algorithm and timeOut(ACTi)was computed from activity logs. We consider twometrics, Timeslice and Offset . The Timeslice accuracy isthe percentage of the stream that has been correctlypredicted by the algorithm; and the Offset is the timedifference between the predicted activity transition andreal transition. Fig. 7 shows better performance of usinglabel refinement compared to the inference only using

current status sensor observations. The results areindicative of the overall performance that theaccelerometer alone as a sensor can achieve in realsettings. Our results give cues on which activities areconfusing in real-life settings. One can investigatealternate sensors on the phones (e.g. microphone) forsuch confusing activities.

ConclusionThis paper explored a challenging problem of continuousonline detection of complex activities in the wild usingonly a single accelerometer sensor embedded in thesmartphone. We designed both uniform and multiplewindow-based models to study early detection of complexactivities in real-time, and demarcation method to detectactivity transitions. We showed exploration results ofdetection accuracy and detection delay, and also analyzedthe tradeoffs between the two metrics.

References[1] Gu, T., Wu, Z., Tao, X., Pung, H. K., and Lu, J.

epsicar: An emerging patterns based approach tosequential, interleaved and concurrent activityrecognition. In PerCom (2009), 1–9.

[2] Huynh, T., Fritz, M., and Schiele, B. Discovery ofactivity patterns using topic models. In Ubicomp(2008), 10–19.

[3] Wu, T., Chiang, Y., and Hsu, Y.-j. Continuousrecognition of daily activities from multipleheterogeneous sensors. In AAAI Human BehaviorModeling (2009), 80–85.

[4] Yan, Z., Chakraborty, D., Misra, A., Jeung, H., andAberer, K. Sammple: Detecting semantic indooractivities in practical settings using locomotivesignatures. In ISWC (2012), 37–40.


202

Documents

An Exploration with Online Complex Activity Recognition