View
2
Download
0
Category
Preview:
Citation preview
Koichi Okamoto, Keiji Yanai
The Univ. of Electro-CommunicationsTokyo, Japan
• Food recording habit is helpful for diet
• However to record
foods, people
have to
－take photos
－input food names
－estimate calories
Time-consuming !
Give up recording soon !!
Food recording app by image recognition ◦ It requires taking meal photos before eating
inappropriate for sharing large platter, hot pot, or BBQ
A novel food recording system Is applicable even for food sharing situation
Recognize eating action during a meal
Record all eaten foods, calories and amounts
smartphone Monitoring eating action
Eating meat !
Mouth detection Chopsticks detection
Eating region detection
Food item
classification
Update total calories
If they come close
1. Detect a face
2. Detect a mouth from the face region
(We used face and mouse detectors in OpenCV.)
Without face detection With face detection
1. Detect moving areas by background subtraction
2. Detect lines from the moving areas by Hough transform
The entire image Only moving area
If mouth and chopstick regions come
close to each other “eating”
Eating region containing a food item
Fusion of two Image features1. Bag-of-features with ORB
2. HSV color histogram
Classifier : Linear SVM
→fast χ2feature map (Vedaldi al et. PAMI12)
Target meal: ‘Yakiniku Grill’◦ Japanese-style BBQ: grill thin-sliced meat & vege.
Target food items for classifiction:5 kinds of typical items in Yakiniku Meat, Rice, Pumpkin, Bell pepper, Carrot
Sharing a grill outdoor Yakiniku alone
1. Evaluation of food classification accuracy
2. User study on system usability
Classification accuracy : 74.8%#Training images : 450 #Test images : 50
Comparison of two systems:◦ Baseline system:
manual recording system by touching food item buttons
◦ Proposed system eating action recognition
5-step evaluation ◦ 5 (better) .... 3 (soso) .... 1 (bad)
Two questions for five subjects
Screen of the
baseline system
Two questions for five subjecs:
1.How easy to take eating record ?
Baseline system: 2.0
Proposed system: 4.8
2.How easy to see calorie intake during meal ?
Baseline system: 3.0Proposed system: 3.4
Better than the manual baseline system
A novel food recording systemRecognize eating action during a meal
Record all eaten foods, calories and amounts
Is appropriate for food sharing situation such as “Yakiniku”
Accuracy: 74.8%
User study: Effective,but need to improve UI
• Add other types of meals:
• Improve classification accuracy:→ Fisher Vector
• Estimate the food volume
Yakiniku alone with !
Mapo doufu◦ Tofu, Minced meat…
Shrimps in chili sauce◦ Shrimps, Onion…
sweet‐and‐sour pork◦ Pork, Carrot, Pineapple…
Eat some food items in one bite
→Pre-defined fixed calories is average calories
Mouth detection accuracy is high
Chopsticks detection accuracy is not high◦ There are lines in background
◦ Detected Chopsticks are longer than the actual
False Eating region
Fork and knife are straight◦ Can be same method as chopsticks detection
If eat sandwiches or onigiri◦ We have to detect hand
Left : real foodRight : food sample
Mixture of real foods and food samples for training image set◦ food samples are not good for training
image set
Food sample Real Food
Koichi Okamoto and Keiji YanaiThe University of Electro-Communications, Tokyo
• Recording food habit is helpful for dietary
• However,
－take photos
－input food names
－estimate calorie
－etc…
Time-consuming !
Quit recording soon !!
Food recording app by image recognition ◦ It requires taking meal photos before eating
inappropriate for sharing large platter, hot pot, and BBQ
Recognize eating action during meal
Automatic recording all the eaten foodfood item category, calorie, amounts…
smartphone Monitoring eating action
Eating meat !Eating meat !
Very new type of mobile food recording system
Capture frame
Mouth detection Chopstick detection
segmentation
Classification of food item
Record eating action
Android App : (In the current implemention, the target is a “Yakiniku” meal.)
Two kinds of evaluations
1. Classification accuracy (5-fold CV)
2. Simple user study (5 subjects)
→ 74.4 %
→Comparison with the baseline systemwhich has no recognition in 5-steps
baseline GrillCam
usability 2.0 4.8
Much better than the no-recognition baseline
You can experience a new type of meal with !
Recommended