Activity Recognition and Monitoring using Smart Phones

Activity Recognition and Monitoring using Wearable Sensors and Smart PhonesOutlineActivity recognition applicationsUnder the hood of activity recognitionExisting activity recognition systemsFurther design considerationsActivity Recognition (AR)AR identifies the activity a user performsRunning, walking, sitting Provides important context in addition to locationsDedicated sensors or smart phones

End-user ApplicationsFitness trackingDistance traveledIntensity and duration of activityCalories burnedHealth monitoringAllows long-term monitoring and diagnosis using continuously generated data, e.g., Parkinson diseaseChanges in behavior patterns can be tellingPositive feedback to ratify behaviors, e.g., reducing hyperactivity via feedback actigraphFall detection

End-user ApplicationsContext-aware behaviorCustomized device behavior, e.g., Playing different kinds of music based on the activity levelChanging display fonts based on moving speedManage device resource based on user activities, e.g., reduce GPS sampling interval when users are stationaryHome and work automation

Third-party ApplicationsTargeted advertisingInferring interest categories, e.g., a person visits Chinese restaurants a lot (but not working there)Adapting to present context, e.g., when and how to display ads based on user activitiesCorporate management and accountingMandatory AR, e.g., monitoring whereabout and activities of hospital staffsVoluntary AR, e.g., car insurance tied to driving behaviorApplications for Crowds and GroupsEnhancing traditional social networks, e.g., uploading activity information such as joggingDiscovery friends based on common activities in close proximityTag places based on activities or detect changes

Basic AR System Diagram

Attributes and SensorsEnvironmental attributesTemperature, humidity, audio level Providing contextual informationAccelerationTriaxial accelerometers> 90% accuracy for ambulatory activitiesEating, tooth brushing, and working on a computer more difficult to distinguish, and is dependent on the location of the sensorLocationPhysiological signals: vital signsFeature Extraction Acceleration

Environment variables

Vital signalsStructural features better capture the trendE.g., Coefficients of fitting polynomial

ClassificationSupervised classificationSemi-supervised classification

Supervised Online AR SystemsOnline classification of activities

Supervised Offline AR SystemsGathered data analyzed offlineApplications: calorie burned over a day

System Issues on Implementing AR in Smart PhonesMultiple sensors on a single platform have different characteristics/requirementsAccelerometer sensitive to orientation but incurs little computation costsAcoustic sensor robust to positions but has high computation cost to processGPS has high energy cost for continuous sensingModular design allowing incorporation of new signal processing algorithmsFlexible programming model in building new applicationsJigsaw A continuous sensing engine

Code in the air (CITA)Tasking framework: developers write task scripts and compile to server and mobile codesActivity layer: high level abstraction allowing activity composition such as isBikingPush service: communicates between devices & server

Activity CompositionSupport AND, OR, NOTEvent A WITHIN xx secEvent A for xx secEvent A next B

Ex: Alice wants her phone to be silent if she is in meeting room with her colleague Bob or AlexIf Alice in the meeting roomBob is the meeting room and Alex is in the meeting roomChallenges and OpportunitiesOpen problems:Individual characteristics (age, gender, height, weight) affects the accuracy of ARConcurrent/overlapping movementsComposite activities: playing tennis

Interesting directions: Collective activity recognitionPrediction of future activities

ReferenceJ. Lockhart, T. Pulickal, and G. Weiss, Applications of Mobile Activity RecognitionOscar D. Lara and Miguel A. Labrador, A Survey on Human Activity Recognition using Wearable SensorsHong Lu,Jun Yang Zhigang Liu Nicholas D. Lane, Tanzeem Choudhury,Andrew T. Campbell, The Jigsaw Continuous Sensing Engine for Mobile Phone ApplicationsLenin Ravindranath, Arvind Thiagarajan, Hari Balakrishnan, and Samuel Madden, Code In The Air: Simplifying Sensing and Coordination Tasks on Smartphones