Miguel Jiménez, Prashanti Angara, Harshit Jain, Kirti Agarwal, Roshni Jain, Hausi Müller, Ulrike Stege, Joanna Ng

IBM Canada LabWeek - CASTLE 2017University of Victoria

Miguel Jiménez, Prashanti Angara, Harshit Jain, Kirti Agarwal, Roshni Jain, Hausi Müller, Ulrike Stege, Joanna Ng

IBM Canada LabWeek - CASTLE 2017University of Victoria

Cognitive IoT Recipe MavenDigital Expertise in the Kitchen

Cognitive IoT Recipe MavenDigital Expertise in the Kitchen

Picture from http://foodindustryexecutive.com/2016/04/the-internet-of-things-and-the-future-of-food/

Motivation● Food wastage is currently 40% in North America [1]● Obesity rate is increasing globally [2]● Limited context integration in existing applications

○ Nutrient information and dietary goals○ Effective use of expiry dates○ Knowledge of users’ taste, preferences, allergies and diseases

● Limited knowledge and time to cook healthy● Sparse and duplicate data across multiple devices and applications

[1] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3228640/[2] https://www.hsph.harvard.edu/obesity-prevention-source/obesity-trends/obesity-rates-worldwide/

2

Outline1. Overview of the IoT Recipe Maven2. Key stakeholders3. Maven components

a. CAPRecipes: context-aware personalized recipes recommenderb. Foodie Fooderson: a conversational agent for the smart kitchenc. Smidge: Smart Refrigerator and Groceryd. SmartGrocer: a profit-aware store path optimizer

4. Challenges5. Conclusion

3

Key Stakeholders

Grocerystores

FridgeMakers

Users

❖ Smidge❖ SmartGrocer❖ CAPRecipes❖ Foodie

4

Cognitive IoT Recipe Maven

Context Sphere

Web app FoodieMobile app

SmartGrocer CAPRecipes

SpoonacularGrocery Store

Smidge Kitchen data

Personal data

Heart monitor

diabetes monitor

Weighing scale

Fridge camera

Smartermats

Food data

5[1] https://spoonacular.com/food-api

CAPRecipes: Context-Aware Recipe Recommendation

6

Collaborative filtering

Content-based filtering

Recommender

Personalized Recipes

Personal health information: allergies and diseases

Cuisine and ingredient preferences

Preferences of family and friends

Browser search history and social interactions

Time of day, location (general context)

Expiry date, quantity and missing ingredients

Context

Recipe preferences and ratings

Ratings

+Reasoning

CAPRecipes: Hybrid interfaceMobile ApplicationWeb UI

7

Foodie: A Cognitive Kitchen Assistant

A cognitive conversational assistant connected to our recipe recommendation engine that:

○ Assists people in finding a suitable recipe○ Describes step-by-step cooking instructions○ Provides nutrition information○ Helps people in creating and tracking dietary goals

I’m hungry

Would you like to make Lemon Garlic Chicken?

I don’t like garlic

How about Grilled Cheese

Pizza?

Sounds good!

Cuisine

Ingredients 8

Sugar intake

Body Weight

Foodie, I’d like to reduce my intake of sugar

[...] how much fat is in thatrecipe?

Under the Hood● #Intents

○ Determine the purpose of arbitrary user input○ Example: “I’m feeling hungry” is classified under the intent #start_cooking

● @Entities○ Keyword identification○ Example: “I want to eat a french breakfast”

■ @cuisine = french

■ @mealType = breakfast

● Dialog○ Possible flows of a conversation via nodes○ Nodes are triggered by conditions

● Context○ Mechanism for passing information between the dialog and the application

Foodie: A Cognitive Kitchen Assistant (2)

9

Smidge: Smart Refrigerator and Grocery

10

Smarter Mats

FridgeCam

Grocery Store

● Update ingredient inventory through grocery receipts

● Personalize recommendations for coupons to increase savings

● FridgeCam: Analyze contents of the fridge

● Smarter Mats: Track weights for contents in the fridge

●

Cashier

Smart Grocer: Profit-Aware Store Path Optimization

Promote products by incentivizing customers:

● Guide users through the store to purchase a list of items

● Indicate additional items in or near the customer’s path that the store wants to promote or offer discounts on.

11

List itemPromoted itemOptimal PathSuggested Detour

Entry

Smart Grocer: Profit-Aware Store Path Optimization

12

Grocery List

User’s preferences

Context Sphere

Kitchen data

Personal data

Smidge

Smart Grocer

Optimal Path

Optimal Path + Suggested Detours

Optimal Path + Extra Detours

ChallengesDiscovering agents in a shared environment to accomplish tasks collaboratively.

Effective context management

Foodie

Building a richer conversational agent is difficult. Platforms are not sophisticated enough yet.

Accurate voice recognition services for people with accents and environments with noise

Smidge

Accurately identifying products inside the fridge

13

Key TakeawaysCognitive IOT Recipe Maven integrates food-related applications and uses context to collaboratively enhance the user experience.

It includes CAPRecipes, a context-aware personalized recipes recommender, Foodie: a conversational agent for the smart kitchen, Smidge: An IoT Enabled Fridge and SmartGrocer: a profit-aware store path optimizer

There is great potential for building systems which cross context barriers and enhance user experiences.

14

Thanks

15

Miguel Jiménez, miguel@uvic.ca

Prashanti Angara, pangara@uvic.ca

Hausi Müller, hausi@uvic.ca

Joanna Ng, jwng@ca.ibm.com