Upload
diannepatricia
View
416
Download
0
Embed Size (px)
Citation preview
kognit.dfki.de
Kognit – Cognitive Assistants for Dementia Patients
Daniel Sonntag German
Research Center for Artificial Intelligence
AI IUI HCI
Interdisciplinary Field and Transcommunity
https://dl.dropboxusercontent.com/u/48051165/ISMAR-2015-IUI-TUTORIAL.pdf
AI as the Basis for Multimodal Interaction in IUIs
Multimodal Multisensor Interfaces
Kognit Theory Design Aspects
Topic
• The use of AI to elders with dementia
• Intelligent assistive technology
• Intelligent cognitive assistance technology
• Design of advanced assistive technology
• While many older adults will remain healthy and productive, overall this segment of the population is subject to physical and cognitive impairment at higher rates than younger people.
• Because of the demographic chance, here will be fewer young people to help older adults cope with the challenges of aging.
• Intelligent cognitive assistance technology may enable older adults to “age in place,” that is, remain living in their homes and independently for longer periods of time.
Motivation
Kognit’s win-win effectImprove quality
of life Living a self-determined independent
life.
Save enormous amounts of
money
Provide relief and
more time for caregivers
Reduce healthcare
systemcosts
Institutionalisation has an enormous
financial cost
The change in demographics is immediately clear: older adults make up an increasingly greater proportion of
the population.
The most rapid growth will occur within a subgroup of this cohort—the so-called “oldest old,” or people over the age of 80.
Compensate for the physical and sensory deficits that may accompany aging
no computer technology
- lift chair, wheel chair- ergonomic handles
- hearing aid device- cardiac pacemaker
Advanced computer-basedtechnologies for AAL
(ambient assisted living)
- SSPI - exoskeleton
- control household appliances (using, e.g.,
head gestures)
Towards cognitive enhancement
computer technology
AI technology
- SSPI (Speech)- AI companion
Assurance of, compensation for, assessment of cognitive deficits
CIND / Dementia
sensor-motor andpsychosocial issues
cognitivedecline
Goals for Kognit
Assurance and Monitoring:
ensuring safety and well-being and
reducing caregiver burden, by tracking an
elder’s behaviour, assessment of regularities and
providing up-to-date status reports to a
caregiver.
Compensation:provide guidance to
people as they carry out their daily activities, reminding them of what they need to do, how to do it and related
this to active memory training (in
AR, MR, VR and serious games)
and proactive multimodal help (in
the field of view).
Assessment: attempt to infer how
well a person is doing—what his or
her current cognitive level of functioning
is—based on continual multimodal
observation of performance of
routine activities (in MR, VR and
speech-based serious games)
Compensation Paradox
compensation
user must be made aware of planned task/activity and must be guided
user and caregiver satisfaction- usability / utility
avoid introducing inefficiency into user activities
- usability / utility
avoid making the user overly reliant on the compensation system
request confirmation about whether an activity has been completed
successfully
Sensors for Activity Monitoring
VideoCameras
GPS
BluetoothBeacons
Eye Tracker
SpeechInput
Bio-Sensors
Domain and location model
Interaction with smart objects
Activity recognition
Activity performance
SeriousGame
Cognitive Status
Task and user model
Context Models
AI Technology
• plan generation and execution monitoring
• reasoning under uncertainty
• machine learning
• natural language processing
• intelligent user interfaces
• robotics and machine vision
• collaboration with colleagues having expertise in
• sensor-network architectures
• privacy and security, and
• human-machine interaction
• failure to eat or drink regularly, pill taking
• wandering around
• ATM: don’t give your money to strangers
• avoid stress situations or recover from them
• household chores and many more …
• https://www.linkedin.com/topic/group/cognitive-systems-institute?gid=6729452
Scenario demands
Kognit Storyboard and implementation
• Memory disorder result in loss of episodic memory in particular, which accounts for our memory of specific events and experiences that can be associated with contextual information. Towards compensating such mental disorder, our goal is to provide the user with episodic memory augmentations by using AI technologies.
• Autobiographical events (times, places, associated emotions, and other con- textual who, what, when, where, why knowledge) that can be explicitly stated constitute information fragments for which a prosthetic memory organisation would be needed.
• A major question concerns the recall of only useful information along the thought process of the individual (and not to slow it down).
• For everyday memory support, we aim to develop a system that can recognise everyday visual content that the user gazes at and construct an episodic memory database entry of the event. The episodic memory database is used to save and retrieve the user’s personal episodic memory events.
kognit.dfki.de/media
Text Recognizer “aspirin”
Databases and recognition modules
Object DB
Activity DB
Episodic Memory DB
{ id: “bread”, type: “object”, image: {“sample1.png”, ...}, features: {“feature1.txt”, ...}, description: “bread is a food” } { id: “Takumi”, type: “person”, image: {“face1.png”,…}, features: {“feature1.txt”,…} description: “Mitarbeiter” } …
{ id: [UNIQUE ID], start: "2014/10/30/20:10:14", end: "2014/10/30/20:10:16", activity: "eat", object: "bread“ } { id: [UNIQUE ID], start: "2014/10/30/10:05:54", end: "2014/10/30/10:20:12", activity: “discuss", object: “Takumi“ } …
{ id: “eat”, level: 2, derived_from: {“bite”,”chew”,…} form: “have_a_meal” } { id: “have_a_meal”, level: 3, derived_from: {“eat”,”drink”,…}, form: “” } { id: “discuss”, level: 2, derived_from: {“look_at_face”,”speak”,…}, form: “meeting” } …
Kognit Cloudant Database: https://kognit-tt.cloudant.com/
Face Recognizer
Person A
BreadObject Recognizer
Object database (including faces)
Sensor Data Attention to …
Gaze
Gaze
Face Object Text
Gaze
Episodic memory event encoding model (Breakfast Scenario)
Encoded Event
Person A
Person A
Cheese
Spoon
Bread
“ingredients:…”
“take 1 pill in the morning…”
Eye tracker (with scene camera)
GPS, or other sensors Location
Living room
Interpretation of raw sensor data: e.g., object recognition, location
estimation,…
Encode the observations into an
episodic event: [Activity] -> [Object]
Activity database (created by
crowdsourcing platform:
LabelMovie)
Make a sandwich
Speak with Person A
Read Medication Instruction
Kognit Hardware Overview
Narrative Clip Pupil Labs Eye-tracker
Oculus DK 2
Anoto
SMI Eye-tracker
Space Glasses Meta Pro (3D cam)
NAO Humanoid Epson Moverio BT-200
Structure Sensor
Brother AirscouterTobii EyeX
WheelPhone
Low range 3D cam
Cybershot scene cam
Leap Motion Accu LED projector
Serious Games in VR
Conclusion• Towards constructing episodic memory event database of the user (as the basis for
compensation), we developed a method for recognition of the visual content that the user gazes at in an everyday scenario.
• Though the face recognition showed robustness, we still have to improve object recognition in natural environments.
• In future work, we will use an HMD to present the information of previous events or recognised objects to the user to further evaluate the presented technical implementation of episodic memory along the thought-process of the user.
• 2D —> 3D, deep learning, GPU