Augmented reality based affect adaptive neurocognitive rehabilitation

Augmented reality based affect adaptive neurocognitive rehabilitation

Intermediate report of Msc. Thesis project A.S.Panic

• Student Media & Knowledge Engineering (EEMCS) - Man Machine Interaction

• 5 months project work at ETH Zurich

• Financial support provided byImperial College, London, TU Delft, ETH Zurich,

RWTH Aachen, ParisTechhttp://www.idealeague.org/

http://www.idealeague.org/

In this presentation


1. Neurocognitive rehabilitation is possible to a higher extent than previously assumed

2. Therapeutic exercises can be made more motivating for the patient

3. In the near future more people will need cognitive (re)habilitation than have access to, or can be treated in the existing clinics

4. Human factors play a significant and undervalued role when designing technology

Thesis position


• Three questions drive the literature survey:

1. Which concepts are used to understand cognitive processes and neurocognitive rehabilitation?

2. How can neurocognitive rehabilitation be supported by virtual reality applications?

3. How can therapeutic games be created so that they promote a motivated learner?

Surveying the literature


Main topics for literature survey


• Lack or loss of language severely impairs cognition

• Sound and music can significantly influence emotion

• Emotion significantly influences cognition

Source: Groome & Dewart, 1999, An introduction to cognitive psychology: processes and disorders

Basic stages of cognition


• In a study at Tufts University:

1. Test subjects were blindfolded and physically moved a random distance between two houses

2. The test subject were then asked to which of the two houses he or she was closer to

The conclusion: the reported distance estimates were significantly different if the two houses were religious in nature (e.g. church, mosque) versus neutral

Source: Qui Wang at the Psychology department of Tufts University

One example of cognitive influences of emotion:


Emotion and cognition

Stages not easily summarizable

• Stages of visual perception include processing horizontal lines, vertical lines, arcs, edge detection, shape detection,

• Stages of auditory perception include pitch, timbre, rhythm, melody, localization, so…

Source: Groome & Dewart, 1999, An introduction to cognitive psychology: processes and disorders

A multi-stage model of memory


• In reality cognitive processes are not identifiable by independent and distinct stages, but composed of many (overlapping) substages

• Cognition operates at the levels of basic skills, executive functions and general intelligence (fluid and crystallized)

• A loss of cognitive skills can be caused by:– Typical aging process

– Neurodegenerative diseases (see picture)

– Traumatic brain injury

• Similar symptoms exist in young and old people, but with different names

• Mostly pathological causes


Loss of cognitive skills

Source: Fisk et al, Designing for older adults

• Cognitive therapy (CT) aims at rehabilitating any of the basic cognitive skills

• Cognitive behavioral therapy (CBT) aims at rehabilitating one or more Activities of Daily Living (ADL)

• Issues with CT and CBT: – Treatment or assessment can be dependant on

administrator (therapist)– Lack of standardized stimuli or treatment content– Therapeutic exercises can be boring

Cognitive (behavioral) therapy


Source: Buschert, 2009, Kognizionsbezogene Interventionen bei Alzheimer-Krankheit

Cognitive rehabilitation can aim to:

1. Reinforce, strengthen or reestablish previously learned skills or behavior

2. Establish patterns of compensatory cognitive activity to cope with impairment of cognitive systems

3. Establish environmental compensation that allow new patterns of activity

4. Enable the patient to adapt to their cognitive impairment in order to improve overall level of functioning and quality of life

Cognitive rehabilitation strategies


Source: Cicerone, 2000, 2005, Evidence based cognitive rehabilitation, recommendations for clinical practice

• Since the 1980s two (theoretical) constructs became adopted when explaining how a loss of cognitive skills can be rehabilitated:

1. Brain plasticity = the changing of neurons, the organization of their networks, and their function via new experiences.

2. Cognitive reserve = the brain’s resilience to neuro pathological damage


Brain plasticity and cognitive reserve

• MRI assessments show that structural engagement with the Tetris game increases formation of ‘gray matter’

Source: Haier et al, 2009, MRI Assessment of cortical thickness and functional activity changes in adolescent girls, following three months of practice on a visual-spatial task

• Lack of conclusive evidence for other games, e.g. Dr Kawashima’s “Brain Training” for Nintendo DS.

• However many of these games are based on standard instruments of cognitive assessment.

Gaming stimulates brain plasticity


• Cognitive reserve is based on two mechanisms:1. Neural reserve2. Neural compensation

Sources: Stern, 2006, Cognitive reserve and Alzheimer’s diseaseSnowdon, 2004, Healthy aging and dementia: findings from the Nun study

Cognitive reserve


• Evidence for cognitive reserve: Nuns have been diagnosed with Alzheimer’s Disease (neurodegenerative!), but without any noticeable loss of cognitive skills

Brain plasticity, Discovery channel clip (5 min)

This video clip, from the Discovery Channel and McGraw-Hill Higher Education, details how the brain can "repair" itself. It introduces a case study of young Jody Miller, who had half of her brain removed, in an attempt to control her epileptic seizures.

Case studies and explanations


http://www.youtube.com/watch?v=TSu9HGnlMV0

http://www.ted.com/talks/michael_merzenich_on_the_elastic_brain.html

Re-wiring the brain, 2008 TED talk (20 min)

Neuroscientist Michael Merzenich looks at one of the secrets of the brain's incredible power: its ability to actively re-wire itself. He's researching ways to harness the brain's plasticity to enhance our skills and recover lost function.

http://www.youtube.com/watch?v=TSu9HGnlMV0

http://www.ted.com/talks/michael_merzenich_on_the_elastic_brain.html

Brain plasticity – a case study


http://www.ted.com/talks/jill_bolte_taylor_s_powerful_stroke_of_insight.html

stroke of insight, 2008 TED talk (20 min)

Brain researcher Jill Bolte Taylor studied her own stroke as it happened. She has become a spokesperson for stroke recovery and for the possibility of coming back from severe brain injury stronger than before.



• At any point in one’s lifetime, cognitive reserve results from a combination of exposures:– Socio-economic status (e.g. occupation)– Educational attainment– Leisure activities

• Physical activity stimulates neurogenesis

Source: Stern, 2009, Cognitive reserve

Stimulating cognitive reserve


• Allows the classification of applications that create an illusion of an alternate reality

Reality Virtual reality

Mixed Reality

Augmented reality,more real than virtual

(e.g. HMD)Non or partially-immersive VR

(e.g computer monitor)

Augmented virtuality,more virtual than real

Source: Milgram & Kishino, 1994, A taxonomy of mixed reality visual displays

The virtuality continuum


Sensorama (1957) by the ‘godfather of virtual reality’

Source: Morten Heilig, 1962, Sensorama patent

Creating a fake reality


• One to four simultaneous users

• 3D motion picture, smell, stereo sound, vibrations, wind

• The holy grail of virtual reality is to fool all the senses and create a ‘suspension of disbelief’ for the virtual environment

• When using virtual reality for neurocognitive rehabilitation, what are the:– Strengths?– Weakness?– Opportunities?– Threats?

• Assumptions:– Administration in home

environment

Source: Adapted from Rizzo & Kim, 2005, A SWOT analysis of the field of virtual reality rehabilitation and therapy

VR based cognitive rehabilitation


• Strength:– High fidelity training and assessment environments can be

created– Economy of scale – Gaming factors can be used to increase patient motivation

and adherence

• Weakness:– Interface challenge: wires, displays and peripherals– Side effects of exposure to virtual environments may not be

unavoidable– Engineering challenge: proper support for data mining,

extraction & analysis (for therapists!)


SWOT of particular interest


• Opportunities:– Emerging technologies:

• Unobtrusive and natural interaction devices, wires• Wearable computing, sensor networks

– Situated cognitive rehabilitation / telerehabilitation– Preventive training during preclinical stage increases

cognitive reserve and resilience to pathology

• Threats:– Privacy issues with telerehabilitation and medical data– Potential for lawsuits because of side-effects– Ethical challenges


SWOT of particular interest


• The goal is to create a game cycle which: – fulfills learning objectives and– creates and sustains a motivated learner

Source: Paras, 2005, Games, motivation and effective learning: an integrated model for educational game design

Motivation in instructional games


Sys

tem

fe

edba

ck

User judgement

User behavior

Learning outcome

Instructional Content

Game Characteristics

Game Cycle

debriefing

Input Process Outcome

• Affective computing: computers and programs which take the player’s affective (emotional) state into account

• Affective games: affect-adaptive games adapt their content based on the player’s emotional state. E.g. adapt gameplay difficulty, offer more or less feedback and support on progress, offer tips and suggestions on how to continue, increase or decrease number of concurrent tasks etc.

• High level design heuristics: “Emote me, assist me, challenge me”

Sources: Picard, 2000, Affective ComputingGilleade et al, 2005, Affective videogames and modes of affective gaming

Affective gaming


• Core requirements:

1. A knowledge base which (hierarchically) describes emotions, used for emotion generation or expression.

2. An affective user model (AUM) of the player, which facilitates recognition of emotion and transitions between them.

3. Modeling of game characters’ emotions.

• No computational model of cognitive processes required, a few simple rules can already have a measurable effect

Source: Hudlicka, 2009, Affective game engines: motivation & requirements

Affective game engines


1. Neurocognitive rehabilitation is possible to a higher extent than previously assumed

– This may be explained by theories of brain plasticity and cognitive reserve

2. Therapeutic exercises can be made more motivating for the patient

– This can be achieved by designing therapeutic games using criteria from affective computing and affective gaming

Thesis position (revisited)


3. In the near future more people will need cognitive (re)habilitation than have access to, or can be treated in the existing clinics

– Using telerehabilitation and commonly available (cheap) computing technology, virtual reality based therapy can be offered to a larger part of the patient population

4. Human factors play a significant and undervalued role when designing technology

– Human Computer Interaction design should target natural and embodied interaction as much as possible.

– Differences in technological aptitude should not prevent people from accessing healthcare

Thesis position (revisited)


• Principal research questions:– Does the use of affect-adaptive training influence the

rate of learning and motivation on a commonly used instrument of cognitive performance, when compared to a standard computer based training?

– Does the use of human computer interaction devices which support more natural and embodied interaction lead to a higher rate of acceptance by the targeted population?

Experimental research questions


• Research vehicle: the mental rotation task (MRT)

• Response time depends on:• Difference in rotation between object pair• Object complexity (but this diminishes with

practice!)Sources: Shepard & Metzler, 1972, Mental rotation of three dimensional objects,

Bethell-Fox & Shepard, 1988, Mental Rotation: effects of stimulus complexity and familiarity

Are these two objects the same?

The mental rotation task


Rationale for using the MRT in this experiment:

– As a widely used instrument for cognitive assessment it is particularly suitable for investigating gaming factors that influence motivation

– As a visuo-spatial task it is particularly suitable for virtual and augmented reality

– Stimulus complexity can be controlled precisely

Sources: Rizzo et al, 1998, the virtual reality mental rotation spatial skills project

Why use the mental rotation task?


Training,Assessment

IR LED glasses allow tracking of head position & gaze direction

Wii remote allows gestural interaction with virtual

objects (pointing, dragging, selecting) and tactile and

auditory output

With headtracking support, a TV screen allows virtual objects to appear in front

or behind of it

Experiment setup


• Targeted population:– Elderly people with mild cognitive impairment– Vision corrected to standard, no history of

postural instabilities, no motion sickness

• In a clinical session the participants complete:1.A training round (using 1 of 3 game modes)2.A feedback round (motivation questionnaire)3.An assessment round (of task performance)

Experiment protocol


• Dec 2009 – Mar 2010 – Project work @ ETH

• Mar 2010 – Jul 2010– Finish writing Msc. Thesis @ DUT– Thesis defense– Complete 2 remaining human factors in aerospace

courses from my honors program

Project status


Do you want to know more?Y. Stern,“Cognitive Reserve”, Neuropsychologia vol 47, 2009

A. Rizzo & G. Kim, “A SWOT analysis of the Field of Virtual Reality Rehabilitation and Therapy”, Presence vol 14, 2005

http://www.cumc.columbia.edu/dept/sergievsky/cnd/pdfs/sdarticle-1.pdf

http://vrpsych.ict.usc.edu/PDF/1Rizzo_2005_A%20swot%20analysis%20of%20the%20field%20of%20VR.pdf

http://www.imgd.wpi.edu/speakers/0910/Hudlicka_ICFDG_09_Dist.pdf

E. Hudlicka, “Affective Game Engines: Motivation and Requirements”, ICDFG proceedings, 2009

http://www.cumc.columbia.edu/dept/sergievsky/cnd/pdfs/sdarticle-1.pdf

Documents

Augmented reality based affect adaptive neurocognitive rehabilitation