Upload
gates
View
28
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Bite detection and differentiation using templates of wrist motion. Department of Electrical and Computer Engineering. MS Defense Exam Soheila Eskandari Committee members: Dr. Adam Hoover (chair ) Dr. John N. Gowdy Dr. Eric R. Muth December 5 th , 2013. Outline. - PowerPoint PPT Presentation
Citation preview
Bite detection and differentiation using templates of wrist motion
MS Defense Exam Soheila Eskandari
Committee members:Dr. Adam Hoover (chair)
Dr. John N. Gowdy Dr. Eric R. Muth
December 5th, 2013
Department of Electrical and Computer Engineering
2
Outline
◦Motivation and Background◦Methods◦Results◦Conclusions
3
Motivation
• One third of U.S. adults were overweight and another one third were obese in 2003-2004 (reported by NHANES)
• Cost associated with obesity was $117 billion in the US in 2000
Obesity treatments
Weight maintenance goal is to achieve: EI=EEThe problem is with the tools people use to measure EI
Mobile Health technologies• Mobile monitoring of the human electrocardiogram
(ECG)• Heart rate,• Breathing frequency,• Blood pressure variations,• Breathing amplitude.• Detection of different sleep phases
6
Wrist motion tracking Dong et al. [7,8] developed a wrist-worn device to
track wrist motion and measure the number of bites taken during a meal. Additional research showed that bites, automatically counted using this method, correlated with self-reported caloric intake at the meal level at 0.5.
Amft [1] developed a wrist-worn device with the primary objective of detecting drinking activities, the container used, and the fluid level.
Junker and Amft [1,2] presented a recognition system that used five inertial sensors located on the wrists, upper arms, and upper torso. Their research describes motion gestures based on the particular utensil used, establishing four gestures (cutlery, drink, spoon, hands).
7
Bite detection based on threshold method by wrist motion tracking
T1 and T2 : The roll velocities T3 : Time interval between the
first and second events of roll motion
T4 : Time interval between the end of one bite and beginning of the next bite
Tested on total of 276 subjects 22,383 bites True detection rate of 76% with
a positive predictive value of 87%
Adjusting the second timing threshold (T4): True detection rate of 82% and a positive predictive value of 82%
Threshold algorithm:
Let EVENT = 0Loop Let V_t = measured roll vel. at time t if V_t > T1 and EVENT = 0 EVENT = 1 Let s=t if V_t<T2 and T-s>T3 and EVENT = 1 Bite detected Let s=t EVENT = 2 if EVENT = 2 and T-s>T4 EVENT = 0
8
Template matching Determine similarity between templates and an
unknown signal Similarity by sum of the cross correlation coefficient:
and the value of absolute difference:
9
MethodsData collectionBite templatesBite differentiationBite detection
10
Data collectionData recorded in a cafeteria environment at
Clemson University (NIH grant 1R41DK091141-A1).
Cafeteria info: 800 guests, provides a wide range of foods and beverages, utensils, and containers.
Total data collected: 276 subjects (131 males and 145 females, ages from 18 to 75 years old, BMI from 17.4 to 46.2 , ethnically diverse)
11
Data collection tools:
Ground truth Total of 22,383 bites
Bite TemplatesDetermine the overall pattern and variability pattern of wrist motion of a biteCreated by :Using both the accelerometers and gyroscopes data• Averaging the motion data across all the bites in the
22,383 total ground truth bites• Over a six second window centered on the bite time• Templates of food and drink bites• Four different types of food bites: bites taken with a fork, bites taken with a spoon, food bites eaten using one hand food bites eaten using both hands
14
Bite differentiationRecognizing different types of bites using
template matching against the typical motion pattern
? ? ?
Algorithm:
• Minimum scoring template identifies the most closely matching bite
Bite detection
Detect the bites from other activities during a meal by template matching based on just roll motionSteps: Sum of absolute difference between a bite template and
the wrist motion data at every time step Detecting local minima Best template matched at the local minima position
Detected bite
Detected bite
Detected bite
Ground truth bites
Computer detected bites
18
ResultsBite templatesBite differentiationBite detection
19
Total bite templates
20
17,166 ground truth food bites3,185 bites drink bites
Food bites (17,166 bites) Drink bites (3,185 bites)
21
Food bites larger average motion in the Z and roll axesDrink bites larger average motion in the X and yaw axes
Food bites (17,166 bites) Drink bites (3,185 bites)
22
Drink bites longer (slower) motion than food bites in the yaw axis. Roll motion for drink bites is opposite to food bites, with negative roll preceding positive roll.
Food bites (17,166 bites) Drink bites (3,185 bites)
23Food bites (17,166 bites) Drink bites (3,185 bites)
Food bites opposite average motion with drink bites in roll axes
24Fork (8,764 bites)
Spoon (1,986 bites) Single hand (9,241 bites)
Both hand (2,441 bites)
Ax
Ay
Az
Yaw
Pitch
Roll
25
Bite differentiation
Ground truth
Computer detected
Food (Ax,Ay,Az,Yaw,Pitch,Roll)
Drink(Ax,Ay,Az,Yaw,Pitch,Roll)
Food 75%,72%,68%,72%,43%,64%
25%,28%,32%,27%,57%,36%
Drink 13%,10%,12%,40%,19%,5.6%
87%,90%,88%,60%,81%,94%
Accuracy 81%,81%,78%,66%,62%,79%Ground truth
Computer detected
Food
Drink
Food 70% 30%
Drink 5% 95%
Accuracy 83%
Bite differentiation of food and drink bites using all 6 motion axes.
Accelerometer and gyroscope motions confusion table for food & drink bites recognition.
26
Confusion matrices for the five types of bites according to utensil, for each axis Overall accuracy for recognizing for the 4 different types
of utensils :19-48% and Drink: 80%
Confusion Accelerometer motion axes.
Ground truth(Ax,Ay,Az)
Computer detected (Ax,Ay,Az)% Fork Spoon Drink Both
handSingle hand
Fork 23,20,21
49,56,51
4,5,6 13,9,10 12,10,1.4
Spoon 19,14,18
20,64,60
4.3,5,5 14,8.5,9 8,9,9
Drink 1,1,1 5,4.5,6 81,84,82
6,6,6.5 7,5,5
Both hand
8,6,7 30,36,31
18,28,37
28,18,16
17,12,10
Single hand
15,11,10
21,27,28
20,28,35
19,15,11
25,19,17
Accuracy 42,41,39 %
27
Confusion gyroscope motion axes.
Ground truth(Yaw,Pitch,Roll)
Computer detected (Yaw,Pitch,Roll)% Fork Spoon Drink Both
handSingle hand
Fork 42,14,49
31,33,17
14,25,8
6,13,12 7,14,14
Spoon 40,9,37
38,40,20
11,25,13
5,11,14 7,15,16
Drink 28,3,1.5
10,10,1.6
51,48,71
10,27,18
1.5,12,8
Both hand
41,6,4 17,19,6
28,31,40
9,31,32 6,13,19
Single hand
36,8,35
29,27,12
20,31,18
7,16,18 10,19,18
Accuracy 30, 31, 38
28
Confusion combining all 6 motion axes.
Ground truth(Ax,Ay,Az,Yaw,Pitch,Roll)
Computer detected (Ax,Ay,Az,Yaw,Pitch,Roll)% Fork Spoon Drink Both
handSingle hand
Fork 48 27 3 15 6
Spoon 30 38 6 21 5
Drink 1.5 1.5 80 10 8
Both hand
5 5 28 46 16
Single hand
33 13 14 22 19
Accuracy 46
29
Bite detectionTested on 22,383 total bitesDetection rate: 48%Positive predictive value: 75%No higher performance for different axes
anddifferent combinations of axes
30
Conclusions Food and drink bites appear to have different wrist motion patterns Different types of utensils for food bites also appear to have
different wrist motion patterns, however, they are not consistent enough to enable differentiation via template matching
Original threshold-based algorithm: 77% true detections, 86% PPVTemplate matching algorithm: 46% true detection, 75% PPVTemplate matching is too rigid for detecting bites; there is too much variability in appearance; interestingly, it yielded the close PPV in the threshold-based algorithm suggesting it might be useful for suppressing false positives
31
References[1] O. Amft, H. Junker, and G. Troster, \Detection of eating and drinking arm gestures using inertial body-worn sensors," in Proceedings of the ninth IEEE International Symposium on Wearable Computers, 2005, pp. 160-163.[2] O. Amft and G. Troster, \On-body sensing solutions for automatic dietary monitoring,"IEEE Pervasive Computing, vol. 8, no. 2, pp. 62-70, 2009.[3] G. Billington, Epstein, \Overweight, obesity, and health risk." Arch Intern Med,vol. 160, pp. 898,904- 2000.[4] M. Boninsegna and M. Rossi, \Similarity measures in computer vision," PatternRecognition Letters, vol. 15, no. 12, pp. 1255-1260, 1994.[5] C. Champagne, G. Bray, A. Kurtz, J. Monteiro, E. Tucker, J. Volaufova, andJ. Delany, \Energy intake and energy expenditure: a controlled study comparingdietitians and non-dietitians," Journal of the American Dietetic Association, vol.102, no. 10, pp. 1428-1432, 2002.[6] C. Ching, M. Jenu, and M. Husain, \Fitness monitor system," in Proceeding ofConference on Convergent Technologies for Asia-Pacic Region, vol. 4, 2003, pp.1399-1403.
32
[7] Y. Dong, \Tracking wrist motion to detect and measure the eating intake of free-living humans," Ph.D. dissertation, Electrical and Computer Engineering Department, Clemson University, 2012.[8] Y. Dong, J. Scisco, M. Wilson, E. Muth, and A. Hoover, \Detecting periods of eating during free living by tracking wrist motion," IEEE of Biomedical Health Informatics, 2013.[9] K. Flegal, M. Carroll, B. Kit, and C. Ogden, \Prevalence of obesity and trends in the distribution of body mass index among us adults, 1999-2010," The journal of the American Medical Association, vol. 307, no. 5, pp. 491-497, 2012.[10] J. Foreyt and W. Poston II, Overview and the future of obesity treatment. Springer, 1999.[11] C. Gagnadre, M. Billon, and S. Thuillier, \Fibre optic sensor for physiologicalparameters," Electronics Letters, vol. 34, no. 21, pp. 1991-1993, 1998.[12] C. Harland, T. Clark, and R. Prance, \High resolution ambulatory electrocardiographic monitoring using wrist-mounted electric potential sensors," Measurement Science and Technology, vol. 14, no. 7, pp. 923-928, 2003.[13] B. Howard and S. Howard, \Lightglove: Wrist-worn virtual typing and pointing,“ in Proceedings of IEEE Fifth International Symposium on Wearable Computers,2001, pp. 172-173.
33
[14] Z. Huang, \An assessment of the accuracy of an automated bite counting method in a cafeteria setting," Master's thesis, Electrical and Computer Engineering Department, Clemson University, 2013.[15] H. Junker, O. Amft, P. Lukowicz, and G. Troster, \Gesture spotting with bodyworn �inertial sensors to detect user activities," pattern Recognition, vol. 41, no. 6, pp. 2010-2024, 2008.[16] E. Kelly, Obesity: Health and Medical Issues Today. Greenwood Publishing Group, 2006.[17] S. Kumar, W. Nilsen, M. Pavel, and M. Srivastava, \Mobile health: Revolutionizing healthcare through transdisciplinary research." IEEE Computer, vol. 46, no. 1, pp. 28-35.[18] J. Lementec and P. Bajcsy, \Recognition of arm gestures using multiple orientation sensors: gesture classication," in Proceedings of the 7th IEEE International Conference on Intelligent Transportation Systems, 2004, pp. 965-970.[19] A. Mikhail, C. Frederic, and D. Philippe, \An algorithm for estimating all matches between two strings," INRIA, Tech. Rep., 2001.[20] C. Ogden, M. Carroll, L. Curtin, M. McDowell, C. Tabak, and K. Flegal, \Prevalence of overweight and obesity in the united states, 1999-2004," The journal of the American Medical Association, vol. 295, no. 13, pp. 1549-1555, 2006.[21] G. Ogris, T. Stiefmeier, H. Junker, P. Lukowicz, and G. Troster, \Using ultrasonic hand tracking to augment motion analysis based recognition of manipulative gestures," in proceedings of the Ninth IEEE International Symposium on Wearable Computers, 2005, pp. 152-159.
34
[22] W. H. Organization, \Overweight and obesity," http://www.who.int/ mediacentre/factsheets/fs311/en/index.html, 2008.[23] J. Salley, \Accuracy of a bite-count based calorie estimate compared to human estimates with and without calorie information available," Master's thesis, Psychology Department, Clemson University, 2013.[24] E. Sazonov and S. Schuckers, \The energetics of obesity: A review: Monitoring energy intake and energy expenditure in humans," IEEE Engineering in Medicine and Biology Magazine, vol. 29, no. 1, pp. 31-35, 2010.[25] D. Schmidt, R. Dannenberg, A. Smailagic, D. Siewiorek, and B. Biigge, \Learning an orchestra conductor's technique using a wearable sensor platform," in Proceeding of the 11th IEEE International Symposium on Wearable Computers, 2007, pp. 113-114.[26] A. Smailagic, D. Siewiorek, U. Maurer, A. Rowe, and K. Tang, \Ewatch: Context sensitive system design case study," in Proceedings of the IEEE Computer Society Annual Symposium on VLSI, 2005, pp. 98-103.[27] STMelectronics. (2013) Mems inertial sensor 3-axis linear accelerometer. http: //www.st.com/web/catalog/sense power/FM89/SC444/PF207281. [28] STMelectronics. (2013) Mems motion sensor 2-axis pitch and roll gyroscope. http://www.st.com/web/catalog/sense power/FM89/SC1288/PF248621.
35
[29] STMelectronics. (2013) Mems motion sensor 2-axis pitch and yaw gyroscope. http://www.st.com/web/catalog/sense power/FM89/SC1288/PF248616.[30] C. Sugimoto, H. Ariesanto, H. Hosaka, K. Sasaki, N. Yamauchi, and K. Itao, \Development of a wrist-worn calorie monitoring system using bluetooth," Mi-crosystem technologies, vol. 11, no. 8-10, pp. 1028-1033, 2005.[31] L. Terre, W. Poston II, and J. Foreyt, 12 Overview and the Future of Obesity Treatment. Springer, 2005.[32] N.Wellman and B. Friedberg, \Causes and consequences of adult obesity: health, social and economic impacts in the united states," Asia Pacic journal of clinical nutrition, vol. 11(s8), pp. S705-S709, 2002.
36
Thank you!
Questions?