Toward a QoL improvement of ALS patients: Development of the Full-body P300-based Tactile...

Takumi Kodama, Shoji Makino and Tomasz M. Rutkowski*Life Science Center of TARA, University of Tsukuba

1

*The University of Tokyo, Tokyo, Japan

Toward a QoL improvement of ALS patients: Development of the Full-Body P300-based Tactile

Brain-Computer Interface

Toward a QoL improvement of ALS patients: Development of the Full-Body P300-based Tactile

Brain-Computer Interface @2016 AEARU Young Researchers International Conference@2016 AEARU Young Researchers International Conference

1: Introduction - What’s BCI?

● Brain Computer Interface (BCI)○ Neurotechnology ○ Exploits user intention ONLY using brain waves

2

1: Introduction - ALS Patients

● Amyotrophic lateral sclerosis (ALS) patients○ Have difficulty to move their muscle by themselves○ BCI could be a communicating tool for ALS patients

3

http://www.businessinsider.com/an-eye-tracking-interface-helps-als-patients-use-computers-2015-9

Dr. Hawkins

● Tactile (Touch) P300-based BCI paradigm○ Predict user’s intentions with finding P300 responses○ P300 responses are evoked by touch stimuli

1: Introduction - Research Approach

41, Stimulate touch sensories 2, Classify brain response

AB

A

B

3, Predict user intention

92.0% 43.3%

A B

TargetNon-Target

P300 brainwave response

● Development of a new tactile BCI paradigm● Propose new communication option toward ALS

patients for improving their Quality of Life

1: Introduction - Research Purpose

5

● Full-body Tactile P300-based BCI (fbBCI) [1]○ Applied six vibrotactile stimulus patterns to user’s back○ User can use fbBCI with their body laying down

2: Method - Our Approach

6[1] Kodama T, Shimizu K, Rutkowski TM. Full Body Spatial Tactile BCI for Direct Brain-robot Control. In: Proceedings of the Sixth International Brain-Computer Interface Meeting. Asilomar Conference Center, Pacific Grove, CA USA: Verlag der Technischen Universitaet Graz; 2016. p. 68.

● Full-body Tactile P300-based BCI (fbBCI)

2: Method - Demonstration

7https://www.youtube.com/watch?v=sn6OEBBKsPQ

2: Method - fbBCI specification

8

Subject

Robot

Brain waves

Commands

2: Method - Signal Acquisition

9

● Event related potential (ERP) interval○ captures 800 ms long after vibrotactile stimulus onsets○ will be converted to feature vectors with their potentials

In fbBCI (default settings): fs = 512 [Hz] ERPinterval = 800 [ms] = 0.8 [sec] Vlength = ceil(512・0.8) = 410

Vlength

VCh○○

p[0]

…

p[Vlength - 1]

Vlength = ceil( fs・ERPinterval)where fs [Hz] , ERPinterval [sec]

Ch○○

2: Method - Signal Processing

10

● Bandpass Filtering & DownsamplingCh○○

Before:

After:

fs = 512 [Hz]

fs’ = 128 [Hz]

● Concatenating feature vectors

2: Method - Feature Extraction

In fbBCI: fs’ = 128 [Hz] Vlength = ceil(128・0.8) = 103

11

…

Vlength

VCz …

Vlength

VPz …

Vlength

VCP6

…

…

… … ……Vex.) VlengthALL = Vlength・8 = 103・8 = 824

VlengthALL

…

Ch1 Ch2 Ch8

● Non-linear SVM (Gaussian Kernel)

2: Method - Classification (1)

12

where γ = 1/VlengthALL (c = 1)

hyperplane

● Training the classifier

2: Method - Classification (2)

13

VT1

VT2

VlengthALL VlengthALL

VN1

VN2

Classifier (2cls)

VTmax

・

・

・

・

・

・

VNmax

VTmax = 60 / ne VNmax = 60 / ne

Random chooseas many as Tmax

}

Non-Target Target

● Evaluation with trained classifier○ P300 responses exist or not?

2: Method - Classification (3)

14

VT1

VlengthALL

・

・

VTmax = 10

Target? orNon-Target? Classifier (2cls)

Test data

2: Method - Experimental settings

15

Condition Details

Number of users (mean age) 10 (21.9 years old)

Number of trials 1 training + 5 tests

Stimulus frequency of exciters 40 Hz

EEG recording system g.USBamp active electrodes EEG system

EEG sampling rate 512 Hz

Vibration stimulus length 100 ms

Inter-stimulus Interval (ISI) 400 ~ 430 ms

● P300 responses were confirmed (> 4 μV) in every channel

3: Result - ERP (P300) responses

16

TargetNon-Target

3: Result - Classification accuracy

17

● Mean classification average: 59.83 %

4: Discussion and conclusions

18

● The effectiveness of fbBCI modality was confirmed○ Classification accuracy : 59.83 % by GSVM○ Expect to improve a QoL for ALS patients

● However, more analyses would be required○ Only 10 healthy users has tried yet○ Need higher accuracies for practical

applications

● [1] Kodama T, Shimizu K, Rutkowski TM. Full Body Spatial Tactile BCI for Direct Brain-robot Control. In: Proceedings of the Sixth International Brain-Computer Interface Meeting: BCI Past, Present, and Future. Asilomar Conference Center, Pacific Grove, CA USA: Verlag der Technischen Universitaet Graz; 2016. p. 68.

● [2] Kodama T, Shimizu K, Makino S, Rutkowski TM. Tactile Brain–computer Interface Based on Classification of P300 Responses Evoked by Spatial Vibrotactile Stimuli Delivered to the User’s Full Body. In: Asia-Pacific Signal and Information Processing Association, 2016 Annual Summit and Conference (APSIPA ASC 2016). APSIPA. Jeju, Korea: IEEE Press; 2016. p. (accepted, in press).

● [3] Kodama T, Shimizu K, Makino S, Rutkowski TM. Full–body Tactile P300–based Brain–computer Interface Accuracy Refinement. In: Proceedings of the International Conference on Bio-engineering for Smart Technologies (BioSMART 2016). Dubai, UAE: IEEE Press; 2016. p. (accepted, in press).

Publications

19

20

Thank you for listening!

● Training the classifier

2: Method - Classification (2)

21

VT1

VT2

VlengthALL VlengthALL

VN1

VN2

VTmax

・

・

・

・

・

・

VNmax

・

・

・

VTmax = 60 / ne VNmax = 300 / ne

Classifier (2cls)

Non-Target Target

● How to train the P300-based BCI classifier?○ Each stimulus pattern were given 10 times randomly○ Altogether 360 (60×6) times for a training classifier

2: Method - Training phase

22

ω1 : Target

Classifier (2cls)

Target 1

1

2

345

6

1

6

5

4

3

2

ω2 : Non-Target

× 10

× 10

× 10

× 10

× 10

× 10Session: 1/6

● How to train the P300-based BCI classifier?○ Each stimulus pattern were given 10 times randomly○ Altogether 360 (60×6) times for a training classifier

2: Method - Training phase

23

ω1 : Target

Classifier (2cls)

Target 2

1

2

345

6

1 × 102 × 10

Session: 2/6

6

5

4

3

2

ω2 : Non-Target× 20

× 20

× 20

× 20

× 10

1 × 10

● How to train the P300-based BCI classifier?○ Each stimulus pattern were given 10 times randomly○ Altogether 360 (60×6) times for a training classifier

2: Method - Training phase

24

ω1 : Target

Classifier (2cls)

Target 3

1

2

345

6 ω2 : Non-Target

Session: 3/6

1 × 102 × 10

6

5

4

3

2

× 30

× 30

× 30

× 20

× 20

1 × 20

3 × 10

● How to train the P300-based BCI classifier?○ Each stimulus pattern were given 10 times randomly○ Altogether 360 (60×6) times for a training classifier

2: Method - Training phase

25

ω1 : Target

Classifier (2cls)

Target 4

1

2

345

6 ω2 : Non-Target

Session: 4/6

1 × 102 × 10

6

5

4

3

2

× 40

× 40

× 30

× 30

× 30

1 × 30

3 × 104 × 10

● How to train the P300-based BCI classifier?○ Each stimulus pattern were given 10 times randomly○ Altogether 360 (60×6) times for a training classifier

2: Method - Training phase

26

ω1 : Target

Classifier (2cls)

Target 5

1

2

345

6 ω2 : Non-Target

Session: 5/6

1 × 102 × 10

6

5

4

3

2

× 50

× 40

× 40

× 40

× 40

1 × 40

3 × 104 × 105 × 10

● How to train the P300-based BCI classifier?○ Each stimulus pattern were given 10 times randomly○ Altogether 360 (60×6) times for a training classifier

2: Method - Training phase

27

ω1 : Target

Classifier (2cls)

Target 6

1

2

345

6 ω2 : Non-Target

Session: 6/6

1 × 102 × 10

6

5

4

3

2

× 50

× 50

× 50

× 50

× 50

1 × 50

3 × 104 × 105 × 106 × 10

60 300

2: Method - Evaluation phase

● How to predict user’s intention with trained classifier?○ Correct example

28

ω1 : Target

Classifier (2cls)

1 × 10

72.6 %

Target 1

Session: 1/6

ω1 : Target

Classifier (2cls)

2 × 10

24.4 %ω1 : Target

Classifier (2cls)

3 × 10

56.3 %ω1 : Target

Classifier (2cls)

4 × 10

44.1 %ω1 : Target

Classifier (2cls)

5 × 10

62.9 %ω1 : Target

Classifier (2cls)

6 × 10

39.8 %

1

2

345

6

2: Method - Evaluation phase

29

ω1 : Target

Classifier (2cls)

1 × 10

35.1 %

Target 6

Session: 6/6

ω1 : Target

Classifier (2cls)

2 × 10

48.1 %ω1 : Target

Classifier (2cls)

3 × 10

69.2 %ω1 : Target

Classifier (2cls)

4 × 10

54.3 %ω1 : Target

Classifier (2cls)

5 × 10

50.9 %ω1 : Target

Classifier (2cls)

6 × 10

64.3 %

1

2

345

6

● How to predict user’s intention with trained classifier?○ Wrong example

2: Method - Evaluation phase

Target 11/6

5

Target 2

Target 3

3

5

● Calculate stimulus pattern classification accuracy○ How many sessions could the user classify targets?

Target 4

Target 5

Target 6

2

4

Result

1

Session

2/6

3/6

4/6

5/6

6/6

1 Trial

Classification accuracy rate:

4/6 = 0.667 ⇒ 66.7 [%]

Correct

Correct

Wrong

Correct

Correct

Wrong

Target Status