The DemaWare Service-Oriented AAL Platform for People with Dementia

Information Technologies Institute

Thanos G. Stavropoulos* Georgios Meditskos Efstratios Kontopoulos Ioannis Kompatsiaris Centre for Research and

Technology Hellas

This work has been supported by the FP7 project Dem@Care: Dementia Ambient Care - Multi-Sensing Monitoring for Intelligent Remote Management and Decision Support (No. 288199)

Outline 1. Introduction 2. State of the art 3. DemaWare Components 4. DemaWare Architecture 5. Semantic Interpretation

Introduction • Goals

• Timely diagnosis and assessment of People with Dementia

• Support various target pilot scenarios • Labs, Homes, Nursing Homes

• DemaWare is an AAL platform to unify

• Data retrieval from heterogeneous sensors • Multi-modal analysis algorithms • Semantic knowledge storage and interpretation

Related Work • OpenAAL [8], FamiWare [9]

▫ Support knowledge management, Service composition, fusion etc.

▫ Yet lacking diverse hardware support

• Various existing middleware e.g. for smart homes ▫ AIM [12], Hydra [13], aWESoME-S [11], [14] ▫ Do not provide higher-level functions

DemaWare Components • SleepClock (Gear4) logs sleep states, time, duration

and interruptions • Wristwatch (Philips DTI2) logs moving intensity, skin

conductance and temperature • An ambient depth camera (Asus or Kinect) is used for detecting  the  user’s  location  (within  zones)  and  performed activity (Complex Activity Recognition)

• A camera closer to the user is used for activity recognition using different models (Human Activity Recognition)

DemaWare Components (2) • A wearable camera (GoPro) is used for object, room and

activity detection • Wireless microphones are used for Offline Speech

Analysis, which returns various dementia indicators • The Knowledge Base (KB) Manager stores all detected

events, measurements and activities in a semantically enhanced format

• The Semantic Interpretation (SI) module performs analysis on collected data to infer higher-level information, sensor fusion and complex event detection

DemaWare • Some components work on various, remote platforms

(OS), some online and some offline • Need for

▫ complex data transfer under common schema ▫ Uniform, platform-independent API

• DemaWare unifies all components under WSDL/SOAP, using an XML/XSD Exchange Model

• Meanwhile real-time events are streamed to the KB-service

• Various GUIs visualize data according to scenarios • GUI backend modules coordinate data collection and

processing

Semantic Interpretation • Need to integrate and process data of different

sensors and modalities ▫ e.g. contact sensors, cameras, microphones

• By combining different modalities we can infer more about the context ▫ any information that can be used to characterise the situation of

an entity (e.g. the condition of the patient) • Use of OWL ontologies and rules

▫ Ontologies provide the domain vocabulary for representing activity-related contextual information (representation layer)

▫ Rules define the structure and semantics of the complex activities (interpretation layer)

Event Ontology

Problem Ontology

Interpretation Layer • Hybrid of OWL reasoning and SPARQL rule execution

for inference of complex activities • Key inferencing tasks:

▫ Temporal reasoning: SPARQL rules to identify temporal dependencies

▫ Complex correlations: SPARQL rules to overcome OWL’s tree model property (composite activities)

▫ Assertion of new individuals: SPARQL rules to generate composite activity individuals

Example • Night sleep monitoring scenario in an ambient assisted

living environment ▫ Atomic activities: {NightSleep, OutOfBed, InBathroom}

▫ Complex Activities: {BedExit, Nocturia} x BedExit: An OutOfBed incident during a NightSleep

(classification of OutOfBed in the BedExit class) x Nocturia: An incident that involves a BedExit incident and an InBathroom incident (composition of a Nocturia incident out of a BedExit incident and an InBathroom incident)

Rule for classifying an OutOfBed into the BedExit class

CONSTRUCT { ?y a BedExit; hasClassifier ?x. } WHERE { ?x a NightSleep; hasStartTime ?st1; hasEndTime ?et1; hasActor ?p. ?y a OutOfBed; hasStartTime ?st2; hasEndTime ?et2; hasActor ?p. FILTER( :contains(?st1, ?et1, ?st2, ?et2) ) }

Rule for composing a Nocturia instance CONSTRUCT { ?new a Nocturia; hasStartTime ?st1; hasEndTime ?et1; hasActor ?p; hasSubActivities ?x; hasSubActivities ?y. } WHERE { ?x a BedExit; hasStartTime ?st1; hasEndTime ?et1; hasActor ?p. ?y a InBathroom; hasStartTime ?st2; hasEndTime ?et2; hasActor ?p. FILTER( :contains(?st1, ?et1, ?st2, ?et2) ) BIND(:newURI(?x, ?y) as ?new) FILTER NOT EXISTS {?new a [] .} }

Conclusion • The system so far enables:

▫ Both real-time and offline data collection and processing

▫ Collection and processing of multi-modal data ▫ Fusion and Semantic Interpreation of data ▫ Various assessment scenarios ▫ Data visualization

Future Work • Enrich real-time data collection

▫ Energy data for powered appliances x Detect cooking, lighting, watching tv etc.

▫ Motion from objects x Detect book reading, watering plants, taking pills etc.

▫ Wearable wristband x More acceptable 24/7 x Detect daily physical activity patterns

• Extensive pilots to infer patterns based on data • Extended AAL @Homes

Thank you

References 1. Anna Fensel, Slobodanka Tomic, Vikash Kumar, Milan Stefanovic, Sergey V Aleshin, and Dmitry O

Novikov,  ‘Sesame-s: Semantic smart home system  for  energy  efficiency’,  Informatik-Spektrum, 36(1), 46–57, (2013).

2. Richard Etter, Patricia Dockhorn Costa, and Tom Broens,  ‘A  rule-based approach towards context-aware user  notification  services’,  in  Pervasive Services, 2006 ACS/IEEE International Conference on, pp. 281–284. IEEE, (2006).

3. Markus Eisenhauer, Peter Rosengren, and Pablo Antolin,  ‘A  development platform for integrating wireless devices and sensors into ambient intelligence systems’,  in  Sensor,  Mesh  and  Ad  Hoc  Communications and Networks  Workshops,  2009.  SECON  Workshops’  09.  6th  Annual IEEE Communications Society Conference on, pp. 1–3. IEEE, (2009).

4. Nikolaos Georgantas, Valerie Issarny, Sonia Ben Mokhtar, Yerom-David Bromberg, Sebastien Bianco, Graham Thomson, Pierre-Guillaume Raverdy, Aitor Urbieta, and Roberto Speicys Cardoso,  ‘Middleware architecture for ambient intelligence in the networked home’,  in  Handbook of Ambient Intelligence and Smart Environments, 1139–1169, Springer, (2010).

5. Antonis Bikakis, Grigoris Antoniou, and Panayiotis Hasapis,  ‘Strategies for contextual reasoning with conflicts  in  ambient  intelligence’,  Knowledge  and Information Systems, 27(1), 45–84, (2011).

6. L. Klein, J. Kwak, G. Kavulya, F. Jazizadeh, B. Becerik-Gerber, P. Varakantham, and M. Tambe, “Coordinating    occupant    behavior    for building energy and comfort management using multi-agent systems,”  Autom. Constr., vol. 22, pp. 525–536, 2012.

7. Z.    Wang,    R.    Yang,    and    L.    Wang,    “Multi-agent control system with intelligent optimization for smart and energy-efficient    buildings,”    in IECON 2010-36th Annual Conference on IEEE Industrial Electronics Society, 2010, pp. 1144–1149.

8. T. G. Stavropoulos, D. Vrakas, D. Vlachava, and N. Bassiliades, “BOnSAI: a smart building ontology    for    ambient    intelligence,”    in  Proceedings of the 2nd International Conference on Web Intelligence, Mining and Semantics, 2012, p. 30.