The Advanced Reality Lab - WordPress.com · Brain-computer interface, virtual reality, media &...

The Advanced Reality Lab

Understanding human-machine confluence in the 21st century

Brain-computer interface, virtual reality, media & brain

ISVAR, April 2016

Brain-Computer Interface (BCI)

BCI

Application

closed

loop

system

feedback

brain signal control signal

BCI: Signals

ISVAR, April 2016

EEG NIRS

fMRI MEG

Electrical activity Blood oxygen level

BCI: Signals

ISVAR, April 2016

EEG NIRS

fMRI MEG

Electrical activity Blood oxygen level

expensive

BCI: Signals

ISVAR, April 2016

EEG NIRS

fMRI MEG

Electrical activity Blood oxygen level

Low spatial resolution

BCI: Signals

ISVAR, April 2016

EEG NIRS

fMRI MEG

Electrical activity Blood oxygen level

Low temporal resolution

BCI: Invasive

ISVAR, April 2016

ECoG Implanted electrode array (Hochberg et al, Nature, 2012)

BCI: Consumer level devices

ISVAR, April 2016

Thought-controlled VR: Paradigms

– Navigation (controlling the viewpoint)

– Controlling an avatar

– Controlling the world directly

• In place control

• Hybrid control

– Beyond control

ISVAR, April 2016

BCI based on EEG and motor imagery

Pfurtscheller and Neuper, Proceedings IEEE, 2001

ISVAR, April 2016

Graz-BCI and UCL-CAVE

ISVAR, April 2016

Pfurtscheller et al., Brain

Research., 2006,

Friedman et al., HCI J 2010,

Leeb et al. Presence, 2007

BCI-VR by a tetraplegic patient

ISVAR, April 2016

Leeb et al., Computational Intelligence and Neuroscience, 2007 Pfurtscheller et al., Int J Psychophysiology, 2007

Thought-controlled VR: Paradigms

– Navigation (controlling the viewpoint)

– Controlling an avatar

– Controlling the world directly

• In place control

• Hybrid control

– Beyond control

ISVAR, April 2016

Graz-BCI and UCL-CAVE

ISVAR, April 2016

Friedman et al., GRAPP, 2007

Real-time fMRI

ISVAR, April 2016

Cohen et al., IEEE Conf Biomedical Robotics, 2012 Cohen et al., J Neural Eng, 2014

robot - Video

1Video 2Video

3 classes: Left hand, right hand, and legs

ISVAR, April 2016

time

% signal change

Whole brain machine learning

ISVAR, April 2016

Classification Results

ISVAR, April 2016

Amp - 95.8%, ctrl - 97.2 % (no sig difference)

Cue-based performance (4 classes)

Cue FC

A gallery visualization of the left vs right contrast using a p<0.05 for the cue-based and free choice tasks, for all subjects from the amputees and the control group. For 'PW' the inflated brain is a mapping of the flat brain below. The red arrows represent the dominant hand and intact hand for the control and amputees groups, respectively.

Thought-controlled VR: Paradigms

– Navigation (controlling the viewpoint)

– Controlling an avatar

– Controlling the world directly

• In place control

• Hybrid control

– Beyond control

ISVAR, April 2016

Steady-state visually evoked potential (SSVEP)-based BCI

ISVAR, April 2016

ISVAR, April 2016

“Unconscious” BCI

0

10

20

30

40

50

60

70

80

90

Experiment Sham

Mean of minimum error rate

Experiment Sham

Minimum error rates between groups at optimal

accuracy time stamp (11-12 seconds, 6 seconds post

stimuli onset), t (19) = 2.225, p < 0.05

Thought-controlled VR: Paradigms

– Navigation (controlling the viewpoint)

– Controlling an avatar

– Controlling the world directly

• In place control

• Hybrid control

– Beyond control

ISVAR, April 2016

Hybrid in-place SSVEP-based BCI

ISVAR, April 2016

Thought-controlled VR: Paradigms

– Navigation (controlling the viewpoint)

– Controlling an avatar

– Controlling the world directly

• In place control

• Hybrid control

– Beyond control

ISVAR, April 2016

Neurofeedback

ISVAR, April 2016

New Paradigms: Beyond Control

• Passive BCI

• Affective Computing

• Next-generation Neurofeedback

ISVAR, April 2016

Thank you

Doron Friedman

http://avl.idc.ac.il

ISVAR, April 2016