CAN SUPERVISED AND UNSUPERVISED LEARNING MARRY HAPPILY?

USE CASES ON HUMAN ACTIVITY RECOGNITION

Natalia Díaz RodríguezVisiting scholar

University of California, Santa Cruznadrodri@ucsc.edu

4th August 2015, Mobilize Center, Stanford University

OUTLINE

•ABOUT ME & MY RESEARCH•RECENT PROJECTS•SUPERVISED AND UNSUPERVISED MACHINE LEARNING•Use Cases on Activity Recognition

ABOUT ME: What excites me in AI?

• Semantic computing• Interpretable, intuitive, human-readable knowledge representation

• Unsupervised and Deep Learning (e.g. Computer Vision) • Automatic and powerful statistical learning

• Can semantics and statistics blend together?• Cognitive neuroscience

• Can the brain’s learning mechanism inspire machine learning?• Quantified Self

• Actionable, personalized well-being

RECENT ACTIVITIES

•Google Anita Borg Scholar •/SYS/TUR Global•Women in Math and CS

•India collab.NIT Meghalaya:Wearables activity recognition

RECENT ACTIVITIES: India collab: IIT KharagpurVirtual tutor: India folklore dance:

Kinect touch free interaction

RECENT ACTIVITIES: SICN 2015 Fellow Summer Institute of Cognitive Neurosciences

http://sicn.cmb.ucdavis.edu/ Program http://sicn.cmb.ucdavis.edu/SI_2015_Week_1_Schedule_1_1_15.pdf http://sicn.cmb.ucdavis.edu/SI_2015_Week_2_Schedule_1_2_15.pdf

RECENT ACTIVITIES:Smart Dosing (with Nursing Science Dept., Finland)

Medication tray filling and dispensing in hospital wards

RECENT ACTIVITIES: First Person Vision for Activity RecognitionUnsupervised annotation (collab. with A. Betancourt, Netherlands)• Recording public Autographer dataset

Betancourt, A., Morerio, P., Barakova, E. I., Marcenaro, L., Rauterberg, M., & Regazzoni, C. S. (2015). A Dynamic Approach and a New Dataset for Hand-Detection in First Person Vision. In International Conference on Computer Analysis of Images and Patterns. Malta.

Currently: Visiting Scholar

HYBRID POSSIBILISTIC AND PROBABILISTIC SEMANTIC MODELLING OF UNCERTAINTY FOR SCALABLE HUMAN ACTIVITY RECOGNITION

•Prof. Lise Getoor•Probabilistic Soft Logic (PSL)

RESEARCH AGENDA

• Supervised and unsupervised learning USE CASES:1. Remote rehabilitation with Kinect2. Human activity recognition in Ambient Intelligence3. Semantic lifestyle profiling with wearables4. Conciliating probabilistic and possibilistic Activity

Recognition with PSL• Mentally stimulating future challenges

Classical data-driven Machine Learning

Knowledge-based Machine Learning

•Event calculus

•Situation calculus

•Rule-based systems

•Fuzzy logic

Knowledge-based Machine Learning

WHY Semantic Technologies & Ontologies?

•Semantic Web: well-defined meaning

•Ontology:

• In Philosophy: study of entities and their relations

• In Artificial Intelligence: “Explicit specification of a

conceptualization” [Gruber, 93]

• Web Ontology Language (OWL)

Background: Ubiquitous computing and Ambient Intelligence

•Smart Space (SS)

•Context-awareness: •Infrastructure and architectures•End-user programming frameworks for AmI

16

Background: Ambient Assisted Living

•Usage of technology to provide assistance to peoplewho needs it in their daily activities, in the less obstrusive way

•Aim: support older/disadvantaged people, independent living, safety

•Includes: methods, systems, products and services

17

Case study 1: A Kinect ontology for physicalexercise annotation and recognition

•Active Healthy Ageing project (EIT Digital)

with Philips Personal Health Labs (PHL)

•Sensor data aggregation platform

18

19

Kinect for remote rehabilitation DEMOS

•Sit-to-stand testhttps://www.youtube.com/watch?v=g8HOtFTk80c• Remote monitoring of post-surgery rehabilitation exercises on shoulder, knee, hip

https://www.youtube.com/watch?v=XL4JexDNs-Q

Ontology features

Skeleton tracking (bone joint rotations + bone orientations)

21

Exercises & Workouts Ontology

22

Kinect ontology: examples of use

23

• Example 1: Defining basic movement (Stand, BendDown, TwistRight, MoveObject, etc.)

• Example 2: Provide workout feedback (# series in time, quality comparison with medical guidelines)

Kinect ontology: examples of use

24

• Example 3: Historic analysis can monitor posture quality in time. E.g. having back less straight than 1 year ago-> notify to correct/prevent on time.

• Example 4: Notifications for office workers sitting too long/ not properly

Case Study 2: PhD (Cum laude, Finland-Spain): Semantic and fuzzy modelling for human behaviour recognition in Smart Spaces, a case study on Ambient Assisted Living

Ros et. Al. 2011

26

SPAIN: 15m of elders in 2049 (1/3 of the population) (INE)FINLAND population 65+ years: 18.14% [1]

• [1] http://www.finnbay.com/media/news/government-prepares-to-set-out-new-requirements-for-senior-caretakers/

PHD OBJECTIVES

27

•Understand Smart Spaces•Human Activity Modelling and Recognition

•Program Smart Spaces

PHD OBJECTIVES

•Understand Smart Spaces•Human Activity Modelling and Recognition

•Program Smart Spaces

28

Activity Recognition in Smart Spaces 29

[Image: http://www.businesskorea.co.kr/sites/default/files/field/image/smart%20home.jpg + The noun project]

Human Activity Recognition

30

Handling uncertainty, vagueness and imprecision

• Broken/ missing sensors

• Incomplete data, vagueness

• Different ways of performing activities• Different object usage, duration, etc.

• Behaviour change

31

Tools

32

33

[CONON Context Ontology]

Methods: Ontologies

34

JULIOANA MARIA

NATALIA

Has Brother

Has Mother

Methods: Ontologies

35

JULIOANA MARIA

NATALIA

Has Brother

Has UncleHas Mother

Methods: Ontologies

Methods: Fuzzy LogicWHY fuzzy (description) logics and fuzzy ontologies?

• Real life is not black & white• Classical (Crisp) Logic: True/False

• Fuzzy Logic: [0, 1]

• e.g. blond, tall

• For automatic reasoning about uncertain, vague or imprecise knowledge

• For natural language expressions

[Bobillo 2008 fuzzyDL: An Expressive Fuzzy Description Logic Reasoner: http://gaia.isti.cnr.it/straccia/software/fuzzyDL/intro.html] 36

37

[Image: http://www.harmonizedsystems.co.uk/]

Example: Take medication

A fuzzy ontology for activity modelling and recognition

38

Classes, Individuals, Data Properties and Object Properties

SUBJECT PREDICATE OBJECT

User performs activity Taking medicine =

(0.3 User performs sub-activity reach Cup or Medicine Box)

(0.3 User performs sub-activity move Cup or Medicine Box)

(0.1 User performs sub-activity place Cup or Medicine Box)

(0.1 User performs sub-activity open Medicine Box)

(0.1 User performs sub-activity eat Medicine Box)

(0.1 User performs sub-activity drink Cup)

39

SurveyingActivityRecognitiontechniques

40

SurveyingActivityRecognitiontechniques

Surveyingontologies for activitymodelling

41

AR Ontologies ranking: domain coverage evaluation

42

OOPS! (OntOlogy Pitfall Scanner!) evaluation

43

Case study 1: A fuzzy ontology for AR in the office/workenvironment

44

45

2-phased algorithm:

1. Sub-activities (data-driven phase)

2. High-level activities (knowledge-based phase)

Validation: CAD-120 dataset:

•10 sub-activities, 10 activities, 10 objects, 4 users46

Hybrid activity recognition with fuzzy ontologies

Cornell Activity Dataset [Koppula et al. 2013]

47

Hybrid data-driven andknowledge-basedactivity recognition

48

a) Data-driven sub-activity recognition phase

49

b) Knowledge-driven sub-activity recognition phase

50

Ontological

definitions

Ontological definitions: object interaction

51

Ontological definitions: object affordances

52

10 semanticrules

53

SUB-ACTIVITY prediction accuracy

54

55

ACTIVITY prediction accuracy

Activity recognition - comparison with state-of-the-art

56

Activity recognitiontimes (ms)

57

PHD OBJECTIVES

•Understand Smart Spaces•Human Activity Modelling and Recognition

•Program Smart Spaces

58

Deployment: Programming Smart Spaces

59

Ros et. Al. 2011

A visual language to configure the Smart Space behaviour

•TARGET USER: a) Developerb) Non-technical background

•AIM: •Rapid & easy programming of applications•Improve interoperability and usability

60

Programming environments for novice programmers

61

[Scratch] [IFTTT]

PROPOSAL: SS visual language mapping to OWL 2

62

PROPOSAL: Smart Space visual programming

63

PhD main contributions

1. A set of ontologies to model human behaviour and tackleuncertainty and vagueness inherent to real life

2. An architecture that integrates Semantic Web and Fuzzy Logicfor interpretable activity recognition

3. A hybrid knowledge-based and data-driven algorithm for real-time, robust activity recognition (84.1% prec.)

4. Design & development of a toolbox for non-expert users and rapid programming of Smart Spaces

[4 Journals -3 on Q1-, 9 conf. Papers. Google Anita Borg, Nokia and HLF scholar. University entrepreneurship award]

64

Startup technology transfer:

65

USE CASE 3: Semantic lifestyle profiling with wearables

Can we recognize lifestyle patterns automatically?1. Provide meaning to large heterogeneous data

-Interpretable, actionable insights2. Knowledge-based methods & uncertainty handling

-Behavior vs Profile recognition

Day routines and lifestyles:• Work/shop-aholics, Gym addicts• Pet/ Partner/ Kids • Retired/ Worker/ On holiday

USE CASE 4: Activity recognition with Probabilistic Soft Logic

•Can manual work be automated?•Can we improve…

•Model & rule learning •Accuracy•Scalability •Genericity

Probabilistic Soft Logic (PSL) :

Expresses collective inference problems mapping logical rules to convex functions(defining a hinge-loss Markov Random field).• FOL Predicate: relationship, property or role• Atom: (continuous) random variables• Rule: dependencies or constraints• Set: aggregatesPSL Program = Rules + Input DB

[PSL (open-source): psl.umiacs.umd.edu]

PSL advantages against fuzzy OWL

•Statistical relational learning •Adds probabilistic component to possibilistic one•Captures cyclic dependencies•Rule weight & latent variable learning

•Scalable (convex optimization) learning•Most probable explanation (MPE) inference

Current work @LINQS Lab

•Can we seamlessly blend…• Knowledge-based and data-driven mechanisms• Supervised and unsupervised learning

for a general activity recognition framework?

•Can TIMED streams be handled naturally?•Cost-sensitive, Online model learning and evolution

•Can models balance•Flexibility and reproducibility?•Accuracy VS deviation/anomaly detection?

General activity recognition framework

AIM: Automate heuristics while maintaining rich semantic granularity:

• Context-awareness • Object interaction, cardinality, recursion, rule subsumption

• Unordered/ordered (sub)sequences of sub-activity-object pairs • Min/ max pattern sequential repetition in Δt

Activity rule examples (PSL)

performsSubActivity(Object, ObjPosition, Time)

m.add rule:

performsMove(MedicineBox, P, T1) &

PerformsDrink(WaterGlass, P, Tn)) >>

PerformsTakingMedicine(Tn),

weight : 10;

Future Challenges in Activity Recognition

•Multiple human sensing•Parallel/interleaved activities

•Automatic ontology learning and evolution•Reduce manual work

•(FOL/DL) Logics support for temporal constraints73

• Unsupervised activity modelling• First camera vision AR

• Automatic dataset annotation

• Wearables sparsity and uncertainty• Scaling and real-timeness

74

Future Challenges in Activity Recognition

TAKE HOME MESSAGE

• Supervised ML: Statistical and probabilistic approaches

• Unsupervised ML: Interpretable knowledge-based techniques

Both are needed!

Domain expert knowledge and common sense knowledge: Classical ML & Deep Learning: unable to exploit it!

THANK YOU!

Brainstorming/collaboration ideas, tips, pointers are welcome!

Natalia Díaz Rodríguez nadrodri@ucsc.edu

https://research.it.abo.fi/personnel/ndiaz

COST Action on Architectures, Algorithms and Platforms for Enhanced Living

Environments: aapele.eu and Finnish Foundation for Technology Promotion

Kinect for remote rehabilitation DEMOS

•Sit-to-stand testhttps://www.youtube.com/watch?v=g8HOtFTk80c• Remote monitoring of post-surgery rehabilitation exercises on shoulder, knee, hip

https://www.youtube.com/watch?v=XL4JexDNs-Q

Comparison with existing state-of-the-art (sub-activity and activity recognition modules)

78

Human Activity Recognition

A crucial but challenging task in Ambient Intelligence and AAL. Requires:

Context-awareness and heterogeneous data sources

Training data: examples

Common-sense knowledge /domain experts

Adaptation of behavioursAlzheimer, Parkinson

79

ACTIVITY prediction accuracy (ideal situation)

(100% accurate sub-activity prediction) 80

SUB-ACTIVITY prediction: accuracy results

81

82

Fuzzy KB and rules in fuzzyDL

Smart Space Architecture: Smart-M3

83

84

Equivalent SPARQL Query

85

Each rule is converted into a SPARQL query, which can be transformed into a Smart-M3 subscription.

Crisp to fuzzy OWL query mapping to improvesemantics and usability

86

Handling uncertainty reasoning whenprogramming Smart Spaces

• Fuzzy reasoners: expressivity VS

computational requirements and

platform versatility:

• Best compromise: fuzzyDL

87

Activity recognition Algorithm

89

Ontology classes, data & object properties

90